Grab Post from Steem API, Analyse Post using Datum Box Machine Learning, Initial Styling

in #utopian-io7 years ago (edited)

Steem Analyser is a post analyzer that allows a user to enter a post URL and be presented with a list of results relating to the analysis of that content. It uses a collection of machine learning APIs to analyze a user's post. This release provides the foundation for further integration with multiple machine learning providers and deeper integration into the steem blockchain.

Within this release I have created the following features:

  1. Ability to grab and display post
  2. Integration into DatumBox machine learning API's to grab and display results for adult content, spam, author gender, commercial detection, educational material & sentiment.
  3. Basic Styling

A screenshot of the overall experience which has currently been created can be seen below:

post-demo.gif

What & How was this created?

The first thing that I created within this release was the ability to grab a post and display it, this can be broken down into a few different parts such as:

  1. Ajax Request to grab content
  2. Define & Update
  3. Convert Body from Markdown to HTML
  4. Display Result & CSS

Firstly I created the following function which is used to grab the users post from the steem blockchain, in order to make a request I used the getContent call which requires you to pass in both author & permalink. In order to get the author a permalink I used the following regular expressions:

  • Identify Permalink from URL: ()?([a-zA-Z0-9-]+)/?$/igm
  • Identify Author from URL: /@[a-z][a-z0-9-]+[a-z0-9]/igm

For the request mentioned above, I also need to strip the '@' symbol from the match found with the above regular expression for the author. I did this by turning it into a string and then replacing the '@' with ''.

Following this, I then defined the items I need to display within my results and used them to create the HTML in index.html.

When grabbing the body of a post from the API it is displayed in markdown, in order to display this on my page correctly I needed to convert this to HTML, I did this using Remarkable.js using the following code:

   var md = new Remarkable();
   var body = md.render(body);

The function I used to grab the posts and generate the HTML for the post can be seen here:

image.png

Following that, I created multiple integrations into a machine learning provider called Datum Box (http://www.datumbox.com/machine-learning-api/). Within this release I created an integration for the following:

  1. Sentiment
  2. Commerical Detection
  3. Spam Detection
  4. Author Gender Detection
  5. Educational Detection
  6. Adult Content Detection

The method for each of these are the same, I simply passed the body of the post grabbed in getPosts into the function and was able to generate a result which I was able to pass into the HTML and display as a result for that post.

An example of one of the functions which are used to grab and create results for these integrations can be seen in the below screenshot:

image.png

A full list of the functions to generate the results from Databox are listed below:

https://github.com/tobias-g1/steem-post-insight/blob/master/js/adult.js
https://github.com/tobias-g1/steem-post-insight/blob/master/js/commerical.js
https://github.com/tobias-g1/steem-post-insight/blob/master/js/education.js
https://github.com/tobias-g1/steem-post-insight/blob/master/js/gender.js
https://github.com/tobias-g1/steem-post-insight/blob/master/js/sentiment.js
https://github.com/tobias-g1/steem-post-insight/blob/master/js/spam.js

Note: I'm aware that I could have done these in single function if I desired, however in future releases I believe I will begin to substitute some of the above for providers who are able to provide more detailed results.

Finally, I applied some basic styling to the page, I used a gridview in order to divide the page. This is a 12 column grid view provided by bootstrap, I do not use any other bootstrap styling within the project. I introduced some of the issues on reduced width windows/mobile when making adjustments to the overflow for the post columns vs the results and will resolve in a later release where I will design and implement a new responsive style.

Commits

The following commits are related to this release:

(Ordered Newest or Oldest)

https://github.com/tobias-g1/steem-post-insight/commit/8f2acf6de2ac92b99e72efa6cdaf53ef068eb5aa

https://github.com/tobias-g1/steem-post-insight/commit/0951578c730ea6e42c6424b4bebc3091366ec20a

https://github.com/tobias-g1/steem-post-insight/commit/a954469b7a571d51bb8713b9a2b01d8b5102e395

Current Limitations

  1. Limited Validation on post URL field
  2. Errors on reduced width windows & mobiles
  3. Limited Integrations & Limitations in detail of results returned from API
  4. Spelling Errors & Badly written copy

Roadmap

  1. Mobile Ready Design
  2. Animation and Custom CSS for results sets
  3. Deeper integrations with other opensource machine learning providers for things such as keyword/entity detection/extraction, details sentiment analysis using a more relevant data set, spelling and grammar.
  4. Ability to share results on steemit
  5. General Clean Up



Posted on Utopian.io - Rewarding Open Source Contributors

Sort:  

Thank you for the contribution. It has been approved.

Great post with a lot of possibilities.

Running the index.html as suggested got me errors: Origin null is not allowed by Access-Control-Allow-Origin. So you have to start your browser with the --allow-file-access-from-files for it to work properly.

I also got some interesting results whiles testing it

  1. educational positive
  2. non-educational neutral
  3. spam negative

While the first two are ok verdict, the last one is not! Might there be a language bias?

You can contact us on Discord.
[utopian-moderator]

ezgif-4-20bd0fa670.gif
Your post is upvoted and resteem by @resteemsteemit bot. Follow @resteemsteemit bot to increase your chance of getting upvote and resteem on your future post.
ezgif-4-4829b469f6.gif

Your Post Has Been Featured on @Resteemable!
Feature any Steemit post using resteemit.com!
How It Works:
1. Take Any Steemit URL
2. Erase https://
3. Type re
Get Featured Instantly � Featured Posts are voted every 2.4hrs
Join the Curation Team Here | Vote Resteemable for Witness

Hey @tobias-g I am @utopian-io. I have just upvoted you!

Achievements

  • Seems like you contribute quite often. AMAZING!

Community-Driven Witness!

I am the first and only Steem Community-Driven Witness. Participate on Discord. Lets GROW TOGETHER!

mooncryption-utopian-witness-gif

Up-vote this comment to grow my power and help Open Source contributions like this one. Want to chat? Join me on Discord https://discord.gg/Pc8HG9x

to @utopian-io
you should be proud of and worth the thumbs up with the achievements you achieve .. I support you and when you succeed .. marry to shareian-io
you should be proud of and worth the thumbs up with the achievements you achieve .. I support you and when you succeed .. marry to share

actually it is an opportunity to enter the contest .. but unfortunately my ability is still very weak. i need real help if you are willing ... thanks

what do you mean you need help?