Asynchronous Web Scraping, Python

Question

Im trying to run this function every 2seconds forever :

import requests from bs4 import BeautifulSoup import asyncio async def scrape(): test = [] r = requests.get(coin_desk) soup = BeautifulSoup(r.text, features='xml') title = soup.find_all('title')[2] await asyncio.sleep(2) for x in title: test.append(x) print(test)

if name == 'main':

try: loop = asyncio.new_event_loop() asyncio.set_event_loop(loop) loop.run_until_complete(scrape()) except (KeyboardInterrupt, SystemExit): pass

If i use

run_forever()

instead of

run_until_complete(scrape())

nothing gets printed out, it just runs forever and skips the function it seems.

Do you get any error messages with an exception / stack trace? Have you tried adding other prints to see which lines you actually reach before it exits? — dskrypa
– dskrypa, Commented Nov 18, 2022 at 22:03
No error messages, when i try to use run_forever() nothing gets printed out. It just runs forever and skips the function completly. — Snuskbetty
– Snuskbetty, Commented Nov 18, 2022 at 22:27
the async routine (scrape) was never placed into the runtime/evloop. run_until_complete includes the parameter to queue a routine for you — Christian
– Christian, Commented Nov 18, 2022 at 22:48

Christian · Accepted Answer · 2022-11-18 22:53:37Z

The async routine (scrape) was never placed into the event loop.

You can add it with

loop.create_task(routine)

See: What does asyncio.create_task() do?

Collectives™ on Stack Overflow

Asynchronous Web Scraping, Python

1 Answer 1

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Linked

Related