2

I'm getting wildly different ranking/scoring between SQL and Lucene given the following query:

[pseudo code] (statut* within 3 of interpret*) AND contradict

I don't think this is an issue with the parser because all of the results seem to conform to the query requirements. However, in the top 1000 of results, I only get 172 common results. Since all the results from both Lucene and SQL both seem to conform to the query requirements, my only remaining guess is that the scoring is somehow radically different. I've had trouble finding any information on how SQL handles scoring, nor on comparing SQL and Lucene scoring. I'm not necessarily expecting the same results set from the two engines, but I was expecting more than 10% similarity and I need to at least be able to explain the huge discrepancy.

How can I explain this significant discrepancy?

Improve this question
1
  • 2
    How can I explain this significant discrepancy? What makes you think that two different products should use a similar ranking/scoring algorithm? Just use whatever you like. Or develop your own scoring algorithm and use it in Lucene.Net(I don't say it would be easy) Commented Jul 9, 2012 at 21:04

1 Answer 1

Reset to default
2

From the documentation on MSDN titled "How Search Query Results Are Ranked" (emphasis mine):

Full-text search in SQL Server can generate an optional score (or rank value) that indicates the relevance of the data returned by a full-text query. This rank value is calculated on every row and can be used as an ordering criteria to sort the result set of a given query by relevance. The rank values indicate only a relative order of relevance of the rows in the result set. The actual values are unimportant and typically differ each time the query is run. The rank value does not hold any significance across queries.

That said, it SQL full text searches places no real value on the result; the only value it has in relation to other rows in the result.

Compare that to scoring in Lucene, which is completely dependent on how you index the documents, whether or not the documents and/or fields are boosted, filters, etc.

Scoring in Lucene is also consistent, unlike in SQL server, where there are no guarantees. It's even reflected in the name, that the result from a full-text query in SQL server is a rank value and not a score, like it is in Lucene.

The values are not completely comparable, but that's understandable, because the results wouldn't be the same either.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.