Race condition in pydoc._start_server #116143
Description
Bug report
Bug description:
There's a race condition in pydoc._start_server - when _start_server() returns, we should get an object with a valid docserver attribute (set here). However, the function only checks that the serving attribute is truthy before returning (here).
The race is triggered if setting serving to True here happens before setting the docserver attribute here -- we observed this happening frequently in the Cinder ASAN test suite (originally observed and fixed by @jbower).
The race can be forced to happen by forcing a context switch after setting self.serving = True:
diff --git a/Lib/pydoc.py b/Lib/pydoc.py index b0193b4a851..117a1dc8369 100755 --- a/Lib/pydoc.py +++ b/Lib/pydoc.py @@ -2511,6 +2511,7 @@ def run(self): def ready(self, server): self.serving = True + time.sleep(0.1) self.host = server.host self.port = server.server_port self.url = 'http://%s:%d/' % (self.host, self.port)and running the test_pydoc.PydocServerTest.test_server test, which would fail and hang:
$ ./python.exe -m test test_pydoc -v -m 'test.test_pydoc.test_pydoc.PydocServerTest.test_server' == CPython 3.13.0a4+ (heads/main-dirty:06565090339, Feb 29 2024, 11:49:21) [Clang 15.0.0 (clang-1500.1.0.2.5)] == macOS-14.3.1-arm64-arm-64bit-Mach-O little-endian == Python build: debug == cwd: /Users/itamaro/work/pyexe/main-dbg/build/test_python_worker_66701æ == CPU count: 12 == encodings: locale=UTF-8 FS=utf-8 == resources: all test resources are disabled, use -u option to unskip tests Using random seed: 792156149 0:00:00 load avg: 5.34 Run 1 test sequentially 0:00:00 load avg: 5.34 [1/1] test_pydoc.test_pydoc test_server (test.test_pydoc.test_pydoc.PydocServerTest.test_server) ... ERROR test_server (test.test_pydoc.test_pydoc.PydocServerTest.test_server) ... ERROR Warning -- threading_cleanup() failed to clean up threads in 1.0 seconds Warning -- before: thread count=0, dangling=1 Warning -- after: thread count=1, dangling=2 Warning -- Dangling thread: <_MainThread(MainThread, started 7977835584)> Warning -- Dangling thread: <ServerThread(Thread-1, started 6150828032)> ====================================================================== ERROR: test_server (test.test_pydoc.test_pydoc.PydocServerTest.test_server) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/itamaro/work/cpython/Lib/test/test_pydoc/test_pydoc.py", line 1823, in test_server self.assertIn('localhost', serverthread.url) ^^^^^^^^^^^^^^^^ AttributeError: 'ServerThread' object has no attribute 'url' ====================================================================== ERROR: test_server (test.test_pydoc.test_pydoc.PydocServerTest.test_server) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/itamaro/work/cpython/Lib/test/test_pydoc/test_pydoc.py", line 1821, in <lambda> lambda: serverthread.stop() if serverthread.serving else None ~~~~~~~~~~~~~~~~~^^ File "/Users/itamaro/work/cpython/Lib/pydoc.py", line 2521, in stop self.docserver.quit = True ^^^^^^^^^^^^^^ AttributeError: 'ServerThread' object has no attribute 'docserver' ---------------------------------------------------------------------- Ran 1 test in 1.321s FAILED (errors=2) Warning -- threading._dangling was modified by test_pydoc.test_pydoc Warning -- Before: {<weakref at 0x10499c280; to '_MainThread' at 0x102eb0a10>} Warning -- After: {<weakref at 0x10550f3f0; to '_MainThread' at 0x102eb0a10>, <weakref at 0x10550f380; to 'ServerThread' at 0x1049b81f0>} test test_pydoc.test_pydoc failed test_pydoc.test_pydoc failed (2 errors) == Tests result: FAILURE == 1 test failed: test_pydoc.test_pydoc Total duration: 1.4 sec Total tests: run=1 (filtered) Total test files: run=1/1 (filtered) failed=1 Result: FAILURE The race can be fixed by making sure the docserver attribute is also set before returning (PR incoming).
CPython versions tested on:
3.8, 3.10, 3.12, CPython main branch
Operating systems tested on:
Linux, macOS