CNDB-17012: Improve TrieMemoryIndex row count estimation

[pkolaczk]

Fixes a bug in TermsDistribution#toBigDecimal which didn't preserve
order for some types and broke the assertion
in TrieMemoryIndex#estimateNumRowsMatchingRange.

Additionally, replaces the row count estimation algorithm with a better
one:

• simpler
• more efficient (no need to search and iterate the trie)
• more accurate (especially for wider ranges)

[github-actions]

Checklist before you submit for review

• This PR adheres to the Definition of Done
• Make sure there is a PR in the CNDB project updating the Converged Cassandra version
• Use NoSpamLogger for log lines that may appear frequently in the logs
• Verify test results on Butler
• Test coverage for new/modified code is > 80%
• Proper code formatting
• Proper title for each commit staring with the project-issue number, like CNDB-1234
• Each commit has a meaningful description
• Each commit is not very long and contains related changes
• Renames, moves and reformatting are in distinct commits
• All new files should contain the DataStax copyright header instead of the Apache License one

[adelapena]

Why a limited alphabet, if we are testing all of UTF-8?

[pkolaczk]

The problem is this simplified estimation isn't very accurate if the data is not distributed uniformly. This ticket really switched the estimation from a horrible algorithm to a slightly less horrible ;). And this test is just a very basic sanity check that the algorithm works fine if the main assumption (= data distributed uniformly) kinda holds.

If we use a wider character set, there will be likely huge holes between the code points, and modeling real distribution of characters used by some real text (as in some existing natural language) becomes hairy very quickly. I got already much worse results when using a set of {a-z, 0-9}, because of a hole between encodings of letters and numbers.

Btw - this test wasn't passing earlier even for the simplified character set like that.
Im afraid we can't make this test better without further improvement of the estimation algorithm itself. We'd probably need to introduce histograms similar way like we have for sstables, but that is far out of scope of this ticket.

[cassci-bot]

❌ Build ds-cassandra-pr-gate/PR-2262 rejected by Butler

---

7 regressions found
See build details here

---

Found 7 new test failures

Test	Explanation	Runs	Upstream
o.a.c.index.sai.QueryContextTest.testWideTableScoreOrdered[ca] (compression)	REGRESSION	🔴🔵	0 / 22
o.a.c.index.sai.cql.NonNumericTermsDistributionTest.testAsciiIndexEstimates (compression)	REGRESSION	🔵🔴	0 / 22
o.a.c.index.sai.cql.NonNumericTermsDistributionTest.testTimestampIndexEstimates (compression)	REGRESSION	🔵🔴	0 / 22
o.a.c.index.sai.cql.NonNumericTermsDistributionTest.testUtf8IndexEstimates (compression)	REGRESSION	🔵🔴	0 / 22
o.a.c.index.sai.cql.VectorCompaction100dTest.testOneToManyCompaction[ec false]	REGRESSION	🔴⚪	0 / 22
o.a.c.index.sai.cql.VectorCompaction2dTest.testZeroOrOneToManyCompaction[ca false] (compression)	REGRESSION	🔴🔵	0 / 22
o.a.c.index.sai.cql.VectorSiftSmallTest.testRerankKZeroOrderMatchesFullPrecisionSimilarity[ec true false]	REGRESSION	🔴⚪	0 / 22

---

Found 7 known test failures

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
93.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud