- Notifications
You must be signed in to change notification settings - Fork 234
Description
Describe the bug:
PostgreSQL has a something called dollar-quoted string constants. apm-agent-python's instrumentation.packages.dbapi2.scan function has logic to detect these dollar quoted string constants and skip them.
However, this logic/code path is also triggered when it comes across query parameters. After all, query parameters also start with a dollar character ($). For queries with a very large number of query parameters (eg, 3000-4000) this code path, as determined by running the app under PyCharm with yappi, is a severe bottle neck. Such a query can be handled in just tens of miliseconds (30-40 ms) by PostgreSQL, but it subsequently takes apm-agent-python more than 10 sec. (!!!). Almost all the time is spent in the skip_to function.
Leading to graphs like these:
What kind of a query takes up to 3000 query parameters? Queries with large VALUES lists.
SELECT * FROM (VALUES ($1::varchar, $2::varchar, $3::varchar), ($4::varchar, $5::varchar, $6::varchar), ($7::varchar, $8::varchar, $9::varchar), ... ($2992::varchar, $2993::varchar, $2994::varchar), ($2995::varchar, $2996::varchar, $2997::varchar), ($2998::varchar, $2999::varchar, $3000::varchar))BTW the above query is a simplified example. In our use case that large VALUES list is part of a much more elaborate CTE (WITH statement)
To Reproduce
- Use SQL queries with a large
VALUESlist consisting of query parameters.
Environment (please complete the following information)
- OS:
- Windows 10
- Linux (whatever is in the
python:3.10.11Docker image) - MacOS 12.6.6 on ARM
- .... in other words it is not OS specific
- Python version:
- 3.10.11
- 3.11.4
- Framework and version [e.g. Django 2.1]:
- FastAPI 0.96.0
- APM Server version:
- 8.6.2
- Agent version:
- (elastic-apm) 6.16.1