/es/_msearch returns mixed results

[pabloveliz]

TLDR;

• how to use grafana with zincsearch using lucene queries
• how to query zincsearch for logical gorups ie: a AND (b OR c)

Briefing

I don't know hot to classify this, so here are the facts:

I'm trying to use grafana (GF) with zincsearch (ZS) 0.4.8 (using the elasticsearch (ES) plugin and the /es endpoint) and I'm not getting the same results as ES.

GF uses Lucene query syntax, which works as expected in ES, suppose we have this data:

game	platform	week
God of Tar	PS2	24
Nano Droids	PS2	30
Legend of Zira	XBOX	24
Remote Remorse	XBOX	40
Grant Andreas	EPIC	20
Saint Col	EPIC	24

A Lucene query in GF like this:
week:24 AND platform:(PS2 OR XBOX)
brings only 2 records in ES (as expected), but in ZS brings the whole set.

So, I tried to figure out why.. after some packet capturing, I notice that the GF plugin uses the endpoint _msearch for getting the data. With this I made a test case in order to replicate the problem, and.. boy oh boy, the problem is bigger.

The response from ZS is not the whole set, is the whole sets (yes, plural). Somehow it returns all the data in the indexes.

So basically, my main question is:

• how can I use GF with ZS with Lucene query, or ...
• how to write the Lucene query above to return those 2 records ONLY

How to replicate

In the test case I added a second data set as:

date	vendor	week
2023-08-13	sony	33
2023-08-13	enix	33

# Run version 0.4.8 os ZS
ZINC_FIRST_ADMIN_USER=admin ZINC_FIRST_ADMIN_PASSWORD=admin ./zincsearch

# Get the list of indexes (empty)
curl admin:admin@localhost:4080/api/index

{"list":[],"page":{"page_num":1,"page_size":0,"total":0}}

# inject first data set
curl -X POST admin:admin@localhost:4080/es/_bulk -H "Content-Type: application/json" -d '
{"index": {"_index": "sample", "_type": "_doc"}}
{"@timestamp": "2023-08-13T12:00:00Z", "game": "God of Tar", "platform": "PS2", "week": 24}
{"index": {"_index": "sample", "_type": "_doc"}}
{"@timestamp": "2023-08-13T12:00:00Z", "game": "Nano Droids", "platform": "PS2", "week": 30}
{"index": {"_index": "sample", "_type": "_doc"}}
{"@timestamp": "2023-08-13T12:00:00Z", "game": "Legend of Zira", "platform": "XBOX", "week": 24}
{"index": {"_index": "sample", "_type": "_doc"}}
{"@timestamp": "2023-08-13T12:00:00Z", "game": "Remote Remorse", "platform": "XBOX", "week": 40}
{"index": {"_index": "sample", "_type": "_doc"}}
{"@timestamp": "2023-08-13T12:00:00Z", "game": "Grant Andreas", "platform": "EPIC", "week": 20}
{"index": {"_index": "sample", "_type": "_doc"}}
{"@timestamp": "2023-08-13T12:00:00Z", "game": "Saint Col", "platform": "EPIC", "week": 24}
'

{"took":0,"errors":false,"items":[{"index":{"_index":"sample","_type":"_doc","_id":"218Lh2WKM2k","_version":1,"result":"created","status":200,"_shards":{"total":1,"successful":1,"failed":0},"_seq_no":1,"_primary_term":1}},{"index":{"_index":"sample","_type":"_doc","_id":"218Lh30ryJa","_version":1,"result":"created","status":200,"_shards":{"total":1,"successful":1,"failed":0},"_seq_no":2,"_primary_term":1}},{"index":{"_index":"sample","_type":"_doc","_id":"218Lh30ryJb","_version":1,"result":"created","status":200,"_shards":{"total":1,"successful":1,"failed":0},"_seq_no":3,"_primary_term":1}},{"index":{"_index":"sample","_type":"_doc","_id":"218Lh30J9Re","_version":1,"result":"created","status":200,"_shards":{"total":1,"successful":1,"failed":0},"_seq_no":4,"_primary_term":1}},{"index":{"_index":"sample","_type":"_doc","_id":"218Lh30J9Rf","_version":1,"result":"created","status":200,"_shards":{"total":1,"successful":1,"failed":0},"_seq_no":5,"_primary_term":1}},{"index":{"_index":"sample","_type":"_doc","_id":"218Lh30J9Rg","_version":1,"result":"created","status":200,"_shards":{"total":1,"successful":1,"failed":0},"_seq_no":6,"_primary_term":1}}]}

# inject second data set
curl -X POST admin:admin@localhost:4080/es/_bulk -H "Content-Type: application/json" -d '
{"index": {"_index": "sale-2023.08", "_type": "_doc"}}
{"@timestamp": "2023-08-13T12:00:00Z", "date": "2023-08-13", "vendor": "sony", "week": 33}
{"index": {"_index": "sale-2023.08", "_type": "_doc"}}
{"@timestamp": "2023-08-13T12:00:00Z", "date": "2023-08-13", "vendor": "enix", "week": 33}
'

{"took":0,"errors":false,"items":[{"index":{"_index":"sale-2023.08","_type":"_doc","_id":"218LlAmkEdW","_version":1,"result":"created","status":200,"_shards":{"total":1,"successful":1,"failed":0},"_seq_no":7,"_primary_term":1}},{"index":{"_index":"sale-2023.08","_type":"_doc","_id":"218LlApseEE","_version":1,"result":"created","status":200,"_shards":{"total":1,"successful":1,"failed":0},"_seq_no":8,"_primary_term":1}}]}

# Get the list of indexes again (only 2 indexes)
curl admin:admin@localhost:4080/api/index

{"list":[{"shard_num":3,"name":"sale-2023.08","storage_type":"disk","settings":{},"mappings":{"properties":{"@timestamp":{"type":"date","index":true,"store":false,"sortable":true,"aggregatable":true,"highlightable":false},"_id":{"type":"keyword","index":true,"store":false,"sortable":true,"aggregatable":true,"highlightable":false},"date":{"type":"text","index":true,"store":false,"sortable":false,"aggregatable":false,"highlightable":false},"vendor":{"type":"text","index":true,"store":false,"sortable":false,"aggregatable":false,"highlightable":false},"week":{"type":"numeric","index":true,"store":false,"sortable":true,"aggregatable":true,"highlightable":false}}},"shards":{"218LlAmkD9S":{"shard_num":1,"id":"218LlAmkD9S","node_id":"","shards":[{"id":0,"stats":{"doc_time_min":1691928000000000000,"doc_time_max":1691928000000000000,"doc_num":1,"storage_size":4314,"wal_size":0}}],"stats":{"doc_time_min":1691928000000000000,"doc_time_max":1691928000000000000,"doc_num":1,"storage_size":4314,"wal_size":0}},"218LlAmkEdW":{"shard_num":1,"id":"218LlAmkEdW","node_id":"","shards":[{"id":0,"stats":{"doc_time_min":0,"doc_time_max":0,"doc_num":0,"storage_size":0,"wal_size":0}}],"stats":{"doc_time_min":0,"doc_time_max":0,"doc_num":0,"storage_size":0,"wal_size":0}},"218LlAmkFi0":{"shard_num":1,"id":"218LlAmkFi0","node_id":"","shards":[{"id":0,"stats":{"doc_time_min":1691928000000000000,"doc_time_max":1691928000000000000,"doc_num":1,"storage_size":4315,"wal_size":0}}],"stats":{"doc_time_min":1691928000000000000,"doc_time_max":1691928000000000000,"doc_num":1,"storage_size":4315,"wal_size":0}}},"stats":{"doc_time_min":0,"doc_time_max":0,"doc_num":2,"storage_size":8629,"wal_size":2},"version":"0.4.8"},{"shard_num":3,"name":"sample","storage_type":"disk","settings":{},"mappings":{"properties":{"@timestamp":{"type":"date","index":true,"store":false,"sortable":true,"aggregatable":true,"highlightable":false},"_id":{"type":"keyword","index":true,"store":false,"sortable":true,"aggregatable":true,"highlightable":false},"game":{"type":"text","index":true,"store":false,"sortable":false,"aggregatable":false,"highlightable":false},"platform":{"type":"text","index":true,"store":false,"sortable":false,"aggregatable":false,"highlightable":false},"week":{"type":"numeric","index":true,"store":false,"sortable":true,"aggregatable":true,"highlightable":false}}},"shards":{"218Lh2WKKYg":{"shard_num":1,"id":"218Lh2WKKYg","node_id":"","shards":[{"id":0,"stats":{"doc_time_min":1691928000000000000,"doc_time_max":1691928000000000000,"doc_num":2,"storage_size":6154,"wal_size":0}}],"stats":{"doc_time_min":1691928000000000000,"doc_time_max":1691928000000000000,"doc_num":2,"storage_size":6154,"wal_size":0}},"218Lh2WKM2k":{"shard_num":1,"id":"218Lh2WKM2k","node_id":"","shards":[{"id":0,"stats":{"doc_time_min":1691928000000000000,"doc_time_max":1691928000000000000,"doc_num":1,"storage_size":4253,"wal_size":0}}],"stats":{"doc_time_min":1691928000000000000,"doc_time_max":1691928000000000000,"doc_num":1,"storage_size":4253,"wal_size":0}},"218Lh2WKN6o":{"shard_num":1,"id":"218Lh2WKN6o","node_id":"","shards":[{"id":0,"stats":{"doc_time_min":1691928000000000000,"doc_time_max":1691928000000000000,"doc_num":3,"storage_size":7766,"wal_size":0}}],"stats":{"doc_time_min":1691928000000000000,"doc_time_max":1691928000000000000,"doc_num":3,"storage_size":7766,"wal_size":0}}},"stats":{"doc_time_min":0,"doc_time_max":0,"doc_num":6,"storage_size":18173,"wal_size":0},"version":"0.4.8"}],"page":{"page_num":1,"page_size":0,"total":2}}

# execute lucene query for sample, should return 2 records as ES
curl -X POST -H 'Content-Type: application/json' admin:admin@localhost:4080/es/_msearch -d '
{"search_type":"query_then_fetch","ignore_unavailable":true,"index":"sample"}
{"size":500,"query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":1672552800000,"lte":1704088799999,"format":"epoch_millis"}}},{"query_string":{"analyze_wildcard":true,"query":"week:24 AND platform:(PS2 OR XBOX)"}}]}},"sort":[{"@timestamp":{"order":"desc","unmapped_type":"boolean"}},{"_doc":{"order":"desc"}}],"script_fields":{}}
'

{"responses":[
  {"took":0,"timed_out":false,"_shards":{"total":6,"successful":6,"skipped":0,"failed":0},"hits":{"total":{"value":8},"max_score":1,"hits":[
    {"_index":"sale-2023.08","_type":"_doc","_id":"218HL9uvD5C","_score":1,"@timestamp":"2023-08-13T12:00:00Z","_source":{"@timestamp":"2023-08-13T12:00:00Z","date":"2023-08-13","vendor":"sony","week":33}},
    {"_index":"sale-2023.08","_type":"_doc","_id":"218HL9xlCog","_score":1,"@timestamp":"2023-08-13T12:00:00Z","_source":{"@timestamp":"2023-08-13T12:00:00Z","date":"2023-08-13","vendor":"enix","week":33}},
    {"_index":"sample","_type":"_doc","_id":"218HKlt8lUI","_score":1,"@timestamp":"2023-08-13T12:00:00Z","_source":{"@timestamp":"2023-08-13T12:00:00Z","game":"God of Tar","platform":"PS2","week":24}},
    {"_index":"sample","_type":"_doc","_id":"218HKlz5VE5","_score":1,"@timestamp":"2023-08-13T12:00:00Z","_source":{"@timestamp":"2023-08-13T12:00:00Z","game":"Legend of Zira","platform":"XBOX","week":24}},
    {"_index":"sample","_type":"_doc","_id":"218HKlz5VE7","_score":1,"@timestamp":"2023-08-13T12:00:00Z","_source":{"@timestamp":"2023-08-13T12:00:00Z","game":"Grant Andreas","platform":"EPIC","week":20}},
    {"_index":"sample","_type":"_doc","_id":"218HKlz5VE6","_score":1,"@timestamp":"2023-08-13T12:00:00Z","_source":{"@timestamp":"2023-08-13T12:00:00Z","game":"Remote Remorse","platform":"XBOX","week":40}},
    {"_index":"sample","_type":"_doc","_id":"218HKlz5VE4","_score":1,"@timestamp":"2023-08-13T12:00:00Z","_source":{"@timestamp":"2023-08-13T12:00:00Z","game":"Nano Droids","platform":"PS2","week":30}},
    {"_index":"sample","_type":"_doc","_id":"218HKlz5VE8","_score":1,"@timestamp":"2023-08-13T12:00:00Z","_source":{"@timestamp":"2023-08-13T12:00:00Z","game":"Saint Col","platform":"EPIC","week":24}}
  ]}}]}

# same query from grafana to ES
curl -X POST -H 'Content-Type: application/json' localhost:9200/_msearch -d '
{"search_type":"query_then_fetch","ignore_unavailable":true,"index":"sample"}
{"size":500,"query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":1672552800000,"lte":1704088799999,"format":"epoch_millis"}}},{"query_string":{"analyze_wildcard":true,"query":"week:24 AND platform:(PS2 OR XBOX)"}}]}},"sort":[{"@timestamp":{"order":"desc","unmapped_type":"boolean"}},{"_doc":{"order":"desc"}}],"script_fields":{}}
'

{"took":0,"responses":[
  {"took":0,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":2,"relation":"eq"},"max_score":null,"hits":[
    {"_index":"sample","_type":"_doc","_id":"SQa8G4oBy0WhxzYxNTov","_score":null,"_source":{"@timestamp": "2023-08-13T12:00:00Z", "game": "Legend of Zira", "platform": "XBOX", "week": 24},"sort":[1691928000000,2]},
    {"_index":"sample","_type":"_doc","_id":"Rwa8G4oBy0WhxzYxNTov","_score":null,"_source":{"@timestamp": "2023-08-13T12:00:00Z", "game": "God of Tar", "platform": "PS2", "week": 24},"sort":[1691928000000,0]}
  ]},"status":200}]}

[pabloveliz]

options := querystr.DefaultOptions()

// TODO fields
// TODO default_field
// TODO default_operator
// TODO boost

zer, _ := zincanalysis.QueryAnalyzer(analyzers, value.Analyzer)

Maybe this has something to do with it ?