-
- Notifications
You must be signed in to change notification settings - Fork 33.6k
Description
Bug report
Bug description:
TL;DR
_SSLProtocolTransport.is_closing should match its inner _SelectorTransport.is_closing, indicating to the user that the transport is actually closed instead of silently logging an error.
Description
I've been using the aio-libs library aiohttp in production together with its WebSocket client implementation, and found an interesting issue that sometimes occured on certain devices (Specifically M-series macbooks).
The logs i've been seeing looks something like this:
( Indication that we are sending messages over websocket successfully ) ... [2024-05-07 09:34:13,403] WARNING asyncio.write: socket.send() raised exception. ... ( Sucessfully sent a message over websocket ) ... [2024-05-07 09:34:13,553] WARNING asyncio.write: socket.send() raised exception. ( No more indication that we're sending or recieving messages over websocket ) Digging deeper the issue occurs when the connection has been lost due to an exception when invoking socket.send, this normally will result in the Transports is_closing() function returning True.
The issue occurs when using TLS, which now uses the transport _SSLProtocolTransport which implements its own is_closing logic.
When _SocketSelectorTransport.write gets an OSError such as Broken Pipe (which is the issue i've experienced in the wild) it sets its inner transport state as closed but when a library such as aiohttp checks its transport is_closing it returns False leading to it silently assuming that it is still connected.
I've been able to recreate the flow by raising a different exception (by manually closing the socket) but the error source and flow is the same in both cases as far as i can tell.
Full example (out of the box + SSL cert generation)
import asyncio import contextlib import logging import socket import ssl import subprocess import tempfile @contextlib.contextmanager def server_ssl_context(host): with tempfile.NamedTemporaryFile() as keyfile, tempfile.NamedTemporaryFile() as certfile: subprocess.run([ 'openssl', 'req', '-new', '-newkey', 'ec', '-pkeyopt', 'ec_paramgen_curve:prime256v1', '-keyout', keyfile.name, '-nodes', '-x509', '-days', '365', '-subj', f'/CN={host}', '-out', certfile.name, ], check=True, shell=False) context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH) context.load_cert_chain(certfile.name, keyfile.name) yield context @contextlib.contextmanager def client_ssl_context(): try: context = ssl.create_default_context(ssl.Purpose.SERVER_AUTH) context.check_hostname = False context.verify_mode = ssl.CERT_NONE yield context finally: pass async def client_handle(reader, writer): ... async def main(host, port): with server_ssl_context(host) as server_context, client_ssl_context() as client_context: await asyncio.start_server(client_handle, host, port, ssl=server_context) reader, writer = await asyncio.open_connection(host, port, ssl=client_context) transport = writer._transport from asyncio.sslproto import _SSLProtocolTransport assert isinstance(transport, _SSLProtocolTransport) inner_transport = transport._ssl_protocol._transport from asyncio.selector_events import _SelectorSocketTransport assert isinstance(inner_transport, _SelectorSocketTransport) sock: socket.socket = inner_transport._sock assert isinstance(sock, socket.socket) # Simulate a broken pipe, this invokes the OS error "Bad file descriptor" # but triggers the same flow as "Broken Pipe" sock.close() # Invoke write so socket returns an OSError print('[Client] Sending x6: %r' % 'Hello, world!') # Increment _conn_lost to more than 5 to trigger the logging. # This silently fails, but the logging is triggered. for i in range(6): writer.write('Hello, world!'.encode()) await writer.drain() print(f"{inner_transport._conn_lost=}, {transport.is_closing()=}, {inner_transport.is_closing()=}") if __name__ == '__main__': logging.basicConfig(level=logging.DEBUG) asyncio.run(main('localhost', 8443), debug=True)CPython versions tested on:
3.12
Operating systems tested on:
Linux, macOS
Linked PRs
- gh-118950: Fix SSLProtocol.connection_lost not being called when OSError is thrown on asyncio.write. #118960
- [3.13] gh-118950: Fix SSLProtocol.connection_lost not being called when OSError is thrown (GH-118960) #125931
- [3.12] gh-118950: Fix SSLProtocol.connection_lost not being called when OSError is thrown (GH-118960) #125932
Metadata
Metadata
Assignees
Labels
Projects
Status