Skip to content

asyncio.sslproto._SSLProtocolTransport can experience invalid state, leading to silent failures. #118950

@cjavad

Description

@cjavad

Bug report

Bug description:

TL;DR

_SSLProtocolTransport.is_closing should match its inner _SelectorTransport.is_closing, indicating to the user that the transport is actually closed instead of silently logging an error.

Description

I've been using the aio-libs library aiohttp in production together with its WebSocket client implementation, and found an interesting issue that sometimes occured on certain devices (Specifically M-series macbooks).

The logs i've been seeing looks something like this:

( Indication that we are sending messages over websocket successfully ) ... [2024-05-07 09:34:13,403] WARNING asyncio.write: socket.send() raised exception. ... ( Sucessfully sent a message over websocket ) ... [2024-05-07 09:34:13,553] WARNING asyncio.write: socket.send() raised exception. ( No more indication that we're sending or recieving messages over websocket ) 

Digging deeper the issue occurs when the connection has been lost due to an exception when invoking socket.send, this normally will result in the Transports is_closing() function returning True.

The issue occurs when using TLS, which now uses the transport _SSLProtocolTransport which implements its own is_closing logic.

When _SocketSelectorTransport.write gets an OSError such as Broken Pipe (which is the issue i've experienced in the wild) it sets its inner transport state as closed but when a library such as aiohttp checks its transport is_closing it returns False leading to it silently assuming that it is still connected.

I've been able to recreate the flow by raising a different exception (by manually closing the socket) but the error source and flow is the same in both cases as far as i can tell.

Full example (out of the box + SSL cert generation)

import asyncio import contextlib import logging import socket import ssl import subprocess import tempfile @contextlib.contextmanager def server_ssl_context(host): with tempfile.NamedTemporaryFile() as keyfile, tempfile.NamedTemporaryFile() as certfile: subprocess.run([ 'openssl', 'req', '-new', '-newkey', 'ec', '-pkeyopt', 'ec_paramgen_curve:prime256v1', '-keyout', keyfile.name, '-nodes', '-x509', '-days', '365', '-subj', f'/CN={host}', '-out', certfile.name, ], check=True, shell=False) context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH) context.load_cert_chain(certfile.name, keyfile.name) yield context @contextlib.contextmanager def client_ssl_context(): try: context = ssl.create_default_context(ssl.Purpose.SERVER_AUTH) context.check_hostname = False context.verify_mode = ssl.CERT_NONE yield context finally: pass async def client_handle(reader, writer): ... async def main(host, port): with server_ssl_context(host) as server_context, client_ssl_context() as client_context: await asyncio.start_server(client_handle, host, port, ssl=server_context) reader, writer = await asyncio.open_connection(host, port, ssl=client_context) transport = writer._transport from asyncio.sslproto import _SSLProtocolTransport assert isinstance(transport, _SSLProtocolTransport) inner_transport = transport._ssl_protocol._transport from asyncio.selector_events import _SelectorSocketTransport assert isinstance(inner_transport, _SelectorSocketTransport) sock: socket.socket = inner_transport._sock assert isinstance(sock, socket.socket) # Simulate a broken pipe, this invokes the OS error "Bad file descriptor" # but triggers the same flow as "Broken Pipe" sock.close() # Invoke write so socket returns an OSError print('[Client] Sending x6: %r' % 'Hello, world!') # Increment _conn_lost to more than 5 to trigger the logging. # This silently fails, but the logging is triggered. for i in range(6): writer.write('Hello, world!'.encode()) await writer.drain() print(f"{inner_transport._conn_lost=}, {transport.is_closing()=}, {inner_transport.is_closing()=}") if __name__ == '__main__': logging.basicConfig(level=logging.DEBUG) asyncio.run(main('localhost', 8443), debug=True)

CPython versions tested on:

3.12

Operating systems tested on:

Linux, macOS

Linked PRs

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    Status

    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions