0

I am trying to store a list of objects using Tweepy's statuses_lookup api call. Each call to statuses_lookup takes in a list of IDs, and can contain up to 100 IDs.

This function below takes in a list of IDs, and I am trying to append all the metadata returned from the API call into the tweetData list. 

def lookupTweets(self, tweetIds): tweetData = [] i = 0 while i < len(tweetIds): print(i) if len(tweetIds) - i > 0: statuses = self.status_lookup(tweetIds[i + 99]) else: statuses = self.status_lookup(tweetIds[i, len(tweetIds) - i]) tweetData.append(statuses) i += 100 return tweetData

And here is the async function that makes the api call

async def status_lookup(self, tweets): return self.api.statuses_lookup(tweets)

And here is the main method:

if __name__ == "__main__": twitterEngine = TwitterEngine() tweets = twitterEngine.ingestData("democratic-candidate-timelines.txt") twitterData = twitterEngine.lookupTweets(tweets) loop = asyncio.get_event_loop() loop.run_until_complete(asyncio.wait(twitterData)) print(twitterData)

When I print the result of twitterData, I get a list of coroutine objects. The output looks something like this: [<coroutine object TwitterEngine.status_lookup at 0x105bd16d0>]. However, I want the actual metadata and not the coroutine object.

I'm new to async programming in Python, and any guidance would be greatly appreciated!

Improve this question
1
  • Does tweepy library support async Commented Apr 28, 2019 at 9:13

2 Answers 2

Reset to default
3

When I print the result of twitterData, I get a list of coroutine objects.

Calling a coroutine function will just create the coroutine object, much like invoking a generator just creates the generator object. To get actual data from a coroutine object, you need to either await it from another coroutine, or run it in the event loop. In case of status_lookup, lookupTweets should itself be an async def coroutine, and it should await the statuses:

statuses = await self.status_lookup(tweetIds[i + 99])

The same goes for status_lookup:

async def status_lookup(self, tweets): return await self.api.statuses_lookup(tweets)

The return value of the outer-most coroutine will be returned by run_until_complete:

loop = asyncio.get_event_loop() twitterData = loop.run_until_complete(twitterEngine.lookupTweets(tweets)) print(twitterData)
Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

When I set status_lookup to be async, I get the following error: TypeError: object ResultSet can't be used in 'await' expression
@VarunVu It seems that status_lookup is not a coroutine. Is tweepy asyncio aware at all?
1

Coroutine objects, the result of calling async def functions, need to be associated with futures to get access to the returned value.

There are several ways of doing this, but if you have a list of coroutine objects, you can use asyncio.gather:

twitterData = twitterEngine.lookupTweets(tweets) future = asyncio.gather(*twitterData) loop = asyncio.get_event_loop() loop.run_until_complete(future) print(future.result())
Improve this answer

6 Comments

This looks promising, but the result from printing future.result() is list of empty lists: [[], [], [], [], [], [], [], [], [], []]
@VarunVu If you add print(tweetData) just before the return in lookupTweets, does it show non-empty lists?
It shows a list of coroutine objects, like so: [<coroutine object TwitterEngine.status_lookup at 0x105bda360>, <coroutine object TwitterEngine.status_lookup at 0x105bdaa98>...]
@VarunVu Ah ok... and just to confirm, self.api.statuses_lookup(tweets) does return something that isn't empty?
Ah I made a dumb mistake with my slicing, I put a comma instead of a semicolon! Thanks!!
|

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.