use correct tokenizer when building termsetquery #4983
Conversation
| &self, | ||
| schema: &TantivySchema, | ||
| tokenizer_manager: &TokenizerManager, | ||
| tokenizer: Option<String>, | ||
| ) -> Result<TantivyQueryAst, InvalidQuery> { | ||
| let full_text_params = FullTextParams { |
There was a problem hiding this comment.
I don't think this works.
What happens if your tokenizer generates two tokens?
You will end up making it into a disjunction.
so roller-coaster will match roller-blade... I'm not sure what the right spec is but this one is too naive.
There was a problem hiding this comment.
we could use get_normalizer() instead, which would just lowercase (if required), and never split tokens. That would make roller-coaster match nothing instead of things it shouldn't. Properly matching would require not creating a TermSetQuery, but a bool query of what works in a termset on one side, and what doesn't on the other, and i'm not sure that's worth the extra trouble
|
I think we want this to behave such as:
should behave exactly like
|
Description
fix #4945
How was this PR tested?
added integration test (also fix an integration test not cleaning up behind)