10
$\begingroup$

Robert Raguet-Schofield in the Mathematica Blog introduced a way to import tweets from Twitter into Mathematica.

I attempted to use this method. I started with this line of code:

xml = Import["https://twitter.com/statuses/user_timeline.xml?screen_name=WolframResearch"]

and received this message:

FetchURL::httperr: The request to URL https://twitter.com/statuses/user_timeline.xml?screen_name=WolframResearch was not successful. The server returned the HTTP status code 404 ("Not Found").

Then I tried this 

xml = Import["https://twitter.com/WolframResearch"]a

and this time I got some output. But when I continue the procedure to extract the tweets with this line

tweets = Cases[xml, XMLElement["text", _, {s_String}] :> s, ∞]

I get an empty set.

How can I import tweets into Mathematica in an effective way, given I am not a familiar with Apachi?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ I think the Twitter API has changed a lot since that blog post. You'll probably need to look at the API documentation and start over. $\endgroup$ Commented Jul 18, 2013 at 22:56
  • $\begingroup$ It is true that the Twitter API has changed and the old one is deprecated. They now require the user to be authenticated, which makes it a bit more of a job retrieving tweets. MMA has SocialMediaData which lets you authenticate yourself and then get some data, but not tweets. It would be interesting to know if there was a way of using the MMA authentication to get other things from the API. $\endgroup$ Commented Jul 18, 2013 at 23:11
  • $\begingroup$ Using the second XML function as above, I could import some tweets but not all. And I know there is a function in R to do so. So it shoul be possible to implement it in Mathematica also. $\endgroup$ Commented Jul 19, 2013 at 8:19
  • $\begingroup$ @Morry Screen scraping can be done elegantly in a lot of languages, but it's harder in Mathematica because there aren't any really good libraries although it is absolutely possible. If you keep working with Cases, eventually you surely find a pattern that can yield all the tweets. APIs are always preferable if one can get them to work though. $\endgroup$ Commented Jul 19, 2013 at 12:18

3 Answers 3

Reset to default
3
$\begingroup$

One first should access his developer account which is codded here as

token = HTTPClient`OAuthAuthentication[ "ConsumerKey" -> "your_api_key", "ConsumerSecret" -> "your_api_secret", "RequestEndpoint" -> "https://api.twitter.com/oauth/request_token", "AuthorizeEndpoint" -> "https://api.twitter.com/oauth/authorize", "AccessEndpoint" -> "https://api.twitter.com/oauth/access_token"]; url = "https://api.twitter.com/1.1/statuses/retweets_of_me.json"; URLFetch[url, "OAuthAuthentication" -> token]`

Then simply you should use API functions. As sample:

urlList = "https://api.twitter.com/1.1/lists/list.json?screen_name=BarackObama";

Then you should take the name of the Slug from the output and then

urlTweet = "https://api.twitter.com/1.1/lists/statuses.json?slug=ofa-legacy-conference&owner_screen_name=BarackObama&per_page=20&page=24" xml = URLFetch[urlTweet, "OAuthAuthentication" -> token];

Then you need a simple StringCases analysis to get the tweets!

Improve this answer
$\endgroup$
1
  • $\begingroup$ You figured it out. Great answer! :) $\endgroup$ Commented Jul 25, 2013 at 10:54
2
$\begingroup$

Perhaps it should be mentioned that since this question was active, (prior to version 10) Mathematica has introduced ServiceConnect which allows a user to access the twittersphere and handles the authentication issue. The documentation provides an example of importing tweets as the OP requested.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Could you include a valid example, for completeness? $\endgroup$ Commented Mar 8, 2019 at 12:17
0
$\begingroup$

Here's a quick hack that nearly answers the question:

xml = Import["http://twitter.com/WolframResearch", "XMLObject"]; tweet = Cases[xml, XMLElement[ "p", {"class" -> "js-tweet-text tweet-text"}, {tweet__, XMLElement["a", details__]}] :> {tweet, details }, Infinity]

resulting in:

{{"Perfect your parking moves with this new Wolfram Demonstration! ", {"shape" -> "rect", "class" -> "twitter-timeline-link", "dir" -> "ltr", "href" -> "http://t.co/YAa2poc8qi", "rel" -> "nofollow", "data-expanded-url" -> "http://wolfr.am/14dZd2G", "target" -> "_blank", "title" -> "http://wolfr.am/14dZd2G"}, {XMLElement[ "span", {"class" -> "invisible"}, {"http://"}], XMLElement["span", {"class" -> "tco-ellipsis"}, {}], XMLElement[ "span", {"class" -> "js-display-url"}, {"wolfr.am/14dZd2G"}], XMLElement["span", {"class" -> "invisible"}, {}], XMLElement[ "span", {"class" -> "tco-ellipsis"}, {XMLElement[ "span", {"class" -> "invisible"}, {" "}]}]}}, {"In case you missed it, here's our latest blog post on using Mathematica to do random walks of ", XMLElement[ ... and so on

Looks like there's hours of 'amusement' there if you like decoding Symbolic XML!

Improve this answer
$\endgroup$
2
  • $\begingroup$ Is it possible to push the code in a way to extend the results to get the whole tweets? I mean what you proposed is just taking the first split of the tweets, not the whole history of tweets!! $\endgroup$ Commented Jul 19, 2013 at 14:27
  • $\begingroup$ I don't think they would do that. Without the Api you may be restricted. Google brings up many non-Mathematica discussions, eg this one $\endgroup$ Commented Jul 19, 2013 at 14:39

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.