Even Twitter itself provides a method based on time or popularity. You can rank them based on TF-IDF similarity, text summarization, spatial and temporal factors, or you can choose the machine learning ranking method. I want to filter high-related tweets (topK) to do the Sentiment Analysis, which will avoid trivial tweets that make our results inaccurate. Ranking technology in my application is to parse tweets crawled from Twitter and then rank related tweets according to their relevance to a specific university. Let's go back to the University Ranking of my designed application.
The number of most universities’ tweets crawled is less than 2000. In this case, I extracted 462,413 tweets results totally. Then, I customized the data fields I need to use for crawling tweets into JSON format. I collected University Ranking data of USNews 2016, which includes 244 universities and their rankings. Then, I generated a query set to crawl tweets, displayed as the figure below.
Then we can use the search function to crawl these structured tweets related to university topics. This mechanism allows us to pull users’ information from the data resource. Note that APP developers need to generate twitter application accounts, so as to get the authorized access to twitter API.īy using a specific Access Token, the application made a request to the POST OAuth2 to exchange credentials so that users can get authenticated access to the REST API. The data crawled will be returned as JSON format. Twitter data can be crawled according to a specific time range, location, or other data fields. Twitter4j is imported to crawl twitter data through twitter REST API. Thus, we can utilize twitter REST APIs to get the most recent and popular tweets. The REST API identifies Twitter applications and users using OAuth. We can also use the public APIs provided by certain websites to get access to their dataset.įirst, it is well known that Twitter provides public APIs for developers to read and write tweets conveniently. There are several methods that we can crawl data from Twitter - Build a web crawler on our own by programming, or choosing an automated web crawler, like Octoparse, Import.io, etc. Take one of the most popular social media websites Twitter as an example. When we think of sentiment analysis, what comes first to our mind is where and how we can crawl oceans of data. In this article, I want to share with you how to crawl Twitter through API or with a web crawler and deal with the data for sentiment analysis.
0 kommentar(er)
