[Feature Request] Support for Experimental Codecs

[sarthakaggarwal97]

Is your feature request related to a problem? Please describe

With the introduction of new custom codecs, it gets essential to bake the new codecs before marking them ready for production. This helps us to avoid potential index corruption risks, and allows for the community feedback to flow in before we make the codecs as generally available.

Earlier, we used to have sandbox for codecs OpenSearch, with the introduction of the separate repository for custom codecs, it gets important to have a similar functionality available in the codecs plugin as well.

Describe the solution you'd like

We can extend the existing CodecSettings interface to know whether the codec is experimental upon index creation.

Related component

Indexing

Describe alternatives you've considered

No response

Additional context

Coming from opensearch-project/custom-codecs#148

[andrross]

@sarthakaggarwal97 Could the custom-codecs plugin itself create an register a static setting like codecs.experimental.enabled and only create and add the new codec if that setting is explicitly set to true?

[sarthakaggarwal97]

@andrross Do you mean a static cluster setting? It would be like a feature flag right. Please correct me if I am wrong.

We might have to utilize this interface so that we can mark codecs as experimental. Another benefit of an interface change is that it gives a way for other plugins as well to use it. We have plugins like Security Analytics, K-nn who also currently maintain their own codecs, and might have similar use cases in the future.

I feel it will be a combination of both the setting and a clean way to mark the new codecs as experimental. We can then have assertions in place to block the create index call.

[andrross]

@sarthakaggarwal97 Yeah it would functionally work the same way as a feature flag, but just be scoped to the custom-codecs plugin in this case. I think in order to determine exactly how we implement this feature we'll want to clearly define the user experience. If we extend the CodecSettings interface, how do we implement the behavior? What's a system admin's experience for enabling and disabling experimental codecs?

[sarthakaggarwal97]

@andrross since the validation to select the codec happens in EngineConfig, we might need to implement the feature flag within OpenSearch itself. CodecSettings interface would help us know whether the particular codec is experimental or not, while the feature flag would make experimental codecs available via CodecService.

Tagging @reta @mgodwan @backslasht for inputs.

[reta]

@andrross since the validation to select the codec happens in EngineConfig, we might need to implement the feature flag within OpenSearch itself. CodecSettings interface would help us know whether the particular codec is experimental or not, while the feature flag would make experimental codecs available via CodecService.

I would 💯 agree with @andrross that experimental codecs should be contained in custom-codecs plugin, including the experimental support. The CodecSettings is optional and we should not expect any codec to implement it (vs Apache Lucene's Codec which is mandatory).

[sarthakaggarwal97]

I guess we should be able to have the setting in custom-codecs. We might have to still expose the experimental via the CodecSettings interface. If we would not validate the experimental nature of codecs in the EngineConfig, and the codecs are not available in CodecService (because they are experimental), the shards would fail.

Apologies if I am missing something here, please correct me if I am wrong.

[reta]

Apologies if I am missing something here, please correct me if I am wrong.

I would encourage to not rely on CodecSettings anyhow (for this specific feature)

[andrross]

If we would not validate the experimental nature of codecs in the EngineConfig, and the codecs are not available in CodecService (because they are experimental), the shards would fail.

@sarthakaggarwal97 Can you elaborate a bit on what you mean here? I get that there are sharp edges on the user experience (i.e. if you disable an experimental codec then any existing indexes using that codec will fail), but that is sort of expected for these opt-in experimental features until they are stabilized and fully supported. To me, the simplest option here is to just let custom-codecs choose whether to register experimental codecs at startup based on the existence of a static setting. What about that implementation doesn't work with the experience you have in mind?