Skip to main content

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

Required fields*

10
  • 2
    It isn't thread-safe to modify shared globals without a lock. And it's especially dangerous to do things like urlsToFetch-=1. Inside the interpreter, that compiles into three separate steps to load urlsToFetch, subtract one, and store urlsToFetch. If the interpreter switches threads between the load and the store, you'll end up with thread 1 loading a 2, then thread 2 loading the same 2, then thread 2 storing a 1, then thread 1 storing a 1. Commented Apr 24, 2013 at 0:13
  • hi abarnert, thanks for your answer can you please suggest a solution for thread-safe? many thanks Commented Apr 24, 2013 at 0:17
  • You can put a threading.Lock around every access to the variable, or lots of other possibilities (use a counted semaphore instead of a plain integer, or use a barrier instead of counting explicitly, …), but you really don't need this global at all. Just join all the threads instead of daemonizing them, and it's done when you've joined them all. Commented Apr 24, 2013 at 0:20
  • In fact… daemonizing the threads like this and then not waiting on anything means your program quits, terminating all of the worker threads, before most of them can finish. On a fastish MacBook Pro with a slowish network connection, I often don't get any finished before it quits. Commented Apr 24, 2013 at 0:21
  • And all of these fiddly details that are very easy to get disastrously wrong and hard to get right are exactly why you're better off using higher-level APIs like concurrent.futures whenever possible. Commented Apr 24, 2013 at 0:23