(This is most likely a validation problem. We need to early, with rate limited logging, invalid index patterns)

We don't attempt to insert into an index named _bulk ever. I think that might be the bug. Nothing had changed on our side. After the last up date today, we started getting that error

hum, @esatterwhite could you provide us with a sample of the requests your client sent?
i couldn't find a way to reproduce that issue in which it worked before, and fails now, but I do get the same error message if i switch from POST to PUT when using the bulk API. Elastic documentation doesn't seem to say that's possible, but actually Elastic accepts such query, and Quickwit thinks it's a normal, non bulk insertion, for an index called _bulk

@esatterwhite is probably right. If you observed it for PUT, it is probably a side effect of shedloading I introduced in
b97f398

Confirmed, I manage to reproduce with

export QW_INGEST_MAX_IN_FLIGHT=0


http POST http://localhost:7280/api/v1/_elastic/_bulk/ < es_batch2.json
HTTP/1.1 400 Bad Request
content-type: application/json
date: Fri, 17 May 2024 15:42:57 GMT
transfer-encoding: chunked
vary: origin
vary: access-control-request-method
vary: access-control-request-headers

{
    "message": "index ID pattern `_bulk` is invalid: patterns must match the following regular expression: `^[a-zA-Z\\*][a-zA-Z0-9-_\\.\\*]{0,254}$`"
}

I still don't know which endpoint it is rerouted too though...