Skip to content

Support CDC mode for Mixed Index mutations #4873

@porunov

Description

@porunov

Mixed Indexes are part of an external system and that's why stale indexes may appear in the system during failures. This can sometimes lead to permanent stale indexes and requires either in-flight index fixing when the index is found or proper transaction recoveries to align data between different storage systems (for example Cassandra and ElasticSearch).
This proposal it to allow using built-in change data capture (CDC) when it's available for mixed index mutations.
In such case, this will guarantee eventual consistency between mixed indexes and actual graph data.

Cassandra natively supports CDC (doc) and there is also available Debezium Connector for Cassandra.

To support this we should change JanusGraph Mixed index mutation mode to skip any mutations during original transaction and instead relay on the Debezium Connector to process Mixed Index mutations later. Alternatively, we could do both (mutate mixed index during original transaction and then trigger mutations again during CDC processing - just to ensure consistency in case of failures).

Related work on transaction recovery process #4872

Metadata

Metadata

Assignees

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions