CDC in FULL mode on aura with Neo4jGraphQLSubscriptionsCDCEngine-enabled neo4j-graphql-server leads to approx. 116 failed queries / minute #6237
Description
Describe the bug
Having CDC enabled in FULL mode on neo4j Aura after some days we get these errors within the neo4j-graphql server approx 1-2 times/second:
Neo4jError: Given ChangeIdentifier describes a transaction that occurred before any enrichment records exist at captureStacktrace (/usr/src/neo4j-graphql-enterprise-server/.output/server/node_modules/neo4j-driver-core/lib/result.js:624:17) at new Result (/usr/src/neo4j-graphql-enterprise-server/.output/server/node_modules/neo4j-driver-core/lib/result.js:112:23) at newCompletedResult (/usr/src/neo4j-graphql-enterprise-server/.output/server/node_modules/neo4j-driver-core/lib/transaction.js:528:12) at Object.run (/usr/src/neo4j-graphql-enterprise-server/.output/server/node_modules/neo4j-driver-core/lib/transaction.js:360:20) at Transaction.run (/usr/src/neo4j-graphql-enterprise-server/.output/server/node_modules/neo4j-driver-core/lib/transaction.js:181:34) at ManagedTransaction.run (/usr/src/neo4j-graphql-enterprise-server/.output/server/node_modules/neo4j-driver-core/lib/transaction-managed.js:54:21) at QueryExecutor.<anonymous> (/usr/src/neo4j-graphql-enterprise-server/.output/server/node_modules/neo4j-driver-core/lib/internal/query-executor.js:94:57) at step (/usr/src/neo4j-graphql-enterprise-server/.output/server/node_modules/neo4j-driver-core/lib/internal/query-executor.js:49:23) at Object.next (/usr/src/neo4j-graphql-enterprise-server/.output/server/node_modules/neo4j-driver-core/lib/internal/query-executor.js:30:53) at /usr/src/neo4j-graphql-enterprise-server/.output/server/node_modules/neo4j-driver-core/lib/internal/query-executor.js:24:71 { constructor: [Function: Neo4jError] { isRetriable: [Function (anonymous)] }, gqlStatus: '52N37', gqlStatusDescription: 'error: procedure exception - procedure execution error. Execution of the procedure db.cdc.query() failed.', diagnosticRecord: { OPERATION: '', OPERATION_CODE: '0', CURRENT_SCHEMA: '/' }, classification: 'UNKNOWN', rawClassification: undefined, code: 'Neo.ClientError.ChangeDataCapture.InvalidIdentifier', retriable: false, [cause]: GQLError: Given ChangeIdentifier describes a transaction that occurred before any enrichment records exist at new GQLError (/usr/src/neo4j-graphql-enterprise-server/.output/server/node_modules/neo4j-driver-core/lib/error.js:117:24) at newGQLError (/usr/src/neo4j-graphql-enterprise-server/.output/server/node_modules/neo4j-driver-core/lib/error.js:261:12) at ResponseHandler._handleErrorCause (/usr/src/neo4j-graphql-enterprise-server/.output/server/node_modules/neo4j-driver-bolt-connection/lib/bolt/response-handler.js:199:57) at ResponseHandler._handleErrorPayload (/usr/src/neo4j-graphql-enterprise-server/.output/server/node_modules/neo4j-driver-bolt-connection/lib/bolt/response-handler.js:193:50) ... 4 lines matching cause stack trace ... at Dechunker.write (/usr/src/neo4j-graphql-enterprise-server/.output/server/node_modules/neo4j-driver-bolt-connection/lib/channel/chunking.js:206:32) at channel.onmessage (/usr/src/neo4j-graphql-enterprise-server/.output/server/node_modules/neo4j-driver-bolt-connection/lib/bolt/create.js:69:63) { constructor: [Function: GQLError], gqlStatus: '52N16', gqlStatusDescription: 'error: procedure exception - invalid procedure argument list. Invalid arguments to procedure.', diagnosticRecord: { OPERATION: '', OPERATION_CODE: '0', CURRENT_SCHEMA: '/', _classification: 'CLIENT_ERROR' }, classification: 'CLIENT_ERROR', rawClassification: 'CLIENT_ERROR', [cause]: GQLError: 52N29: Given ChangeIdentifier describes a transaction that occurred before any enrichment records exist. at new GQLError (/usr/src/neo4j-graphql-enterprise-server/.output/server/node_modules/neo4j-driver-core/lib/error.js:117:24) at newGQLError (/usr/src/neo4j-graphql-enterprise-server/.output/server/node_modules/neo4j-driver-core/lib/error.js:261:12) at ResponseHandler._handleErrorCause (/usr/src/neo4j-graphql-enterprise-server/.output/server/node_modules/neo4j-driver-bolt-connection/lib/bolt/response-handler.js:199:57) at ResponseHandler._handleErrorCause (/usr/src/neo4j-graphql-enterprise-server/.output/server/node_modules/neo4j-driver-bolt-connection/lib/bolt/response-handler.js:198:50) at ResponseHandler._handleErrorPayload (/usr/src/neo4j-graphql-enterprise-server/.output/server/node_modules/neo4j-driver-bolt-connection/lib/bolt/response-handler.js:193:50) at ResponseHandler.handleResponse (/usr/src/neo4j-graphql-enterprise-server/.output/server/node_modules/neo4j-driver-bolt-connection/lib/bolt/response-handler.js:116:49) at dechunker.onmessage (/usr/src/neo4j-graphql-enterprise-server/.output/server/node_modules/neo4j-driver-bolt-connection/lib/bolt/create.js:73:33) at Dechunker._onHeader (/usr/src/neo4j-graphql-enterprise-server/.output/server/node_modules/neo4j-driver-bolt-connection/lib/channel/chunking.js:196:18) at Dechunker.AWAITING_CHUNK (/usr/src/neo4j-graphql-enterprise-server/.output/server/node_modules/neo4j-driver-bolt-connection/lib/channel/chunking.js:149:25) at Dechunker.write (/usr/src/neo4j-graphql-enterprise-server/.output/server/node_modules/neo4j-driver-bolt-connection/lib/channel/chunking.js:206:32) { constructor: [Function: GQLError], cause: undefined, gqlStatus: '52N29', gqlStatusDescription: 'error: procedure exception - outdated change identifier. Given ChangeIdentifier describes a transaction that occurred before any enrichment records exist.', diagnosticRecord: [Object], classification: 'CLIENT_ERROR', rawClassification: 'CLIENT_ERROR' } } } Accordingly in neo4j aura we record these failed queries:
CALL db.cdc.query(******, [{ select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }, { select: ******, labels: [******], txMetadata: { app: $param0 } }])They fail with status codes 52N37 and 52N16 with no further description.
They occurr ~1168 times per 10 minutes.
Subscriptions are configured via:
new Neo4jGraphQLSubscriptionsCDCEngine({ driver, onlyGraphQLEvents: true, })No other configurations, just using the default.
The only way to solve this is to restart the neo4h-graphql server.
Am I doing/configuring something wrong here?
System (please complete the following information):
- Version: @neo4j/graphql@6.6.1]