ZincSearch future

[prabhatsharma]

ZincSearch is a general purpose search engine that uses inverted indexes and is primarily used for the below 2 use cases

1. app search
2. log search

During our early months as an open source software based company, we realized that index based log search engine is not a good technology for log search and majority of the users of ZincSearch were using it for log search. Hence we decided to build https://github.com/openobserve/openobserve

ZincSearch will be maintained and supported going forward, but only for the use case of app search. For any log search use cases preferred tool would be OpenObserve.

[gaby]

@prabhatsharma Are there plans to address all the open dependencies PR's?

Seems like contributions stopped and moved to ZO.

[prabhatsharma]

We will merge dependencies PRs. The majority of the contributions will continue to ZO.

[gaby]

Thanks! Looking forward :-)

[gaby]

@prabhatsharma @hengfeiyang It seems like the whole migration from ZincSearch to ZincObserve is very confusing... The last update to the readme on this repo here states that ZincObserve is fully ES api compatible, but that's not true?

It would be very helpful if someone went through both repo and cleanup all the confusion. ZincSearch was originally under ZincLabs and was migrated to this new GitHub Org, while ZincObserve was moved over to the original ZincLabs to make it look like it has more followers.

The documentation are pointing to features from ZincSearch while being published as ZincObserve. It's just very confusing to find any information now, especially given that ZincObserve is not production ready and API keeps changing on a daily basis.

On another note it would help if all the dependency PR's were merged for ZincSearch and new release created.

[prabhatsharma]

@gaby Appreciate your concern. We understand that there is confusion and you are not the only one. Let me give you some background on what is going on that will give you insight into the changes that are happening (They are still in progress and there is more to come).

I initially built ZincSearch for the use of log search as a hobby project as an alternative to elasticsearch with a focus on ease of use and low resource consumption. Once we started a commercial open source company in April 2022, we wanted to make sure that the offering that we had for log search was at least 10x better than existing platforms. We could have gotten 5x better with ZincSearch for log search in this regard (but not 10x). We researched and came up with ZincObserve that is 10x (actually a lot more. 140x on storage, stateless nodes, ingest and query functions, better UI than Kibana, ....) better than existing log search products. We are set out to build the best log search and observability product in the world. This requires extreme focus when you have a small team working towards that goal. Let's also understand that 90% of the users who use ZincSearch are using it for log search.

As a commercial open source company, we would like to solve the problem that we started to solve when we started the company which is log search. ZincSearch is a general purpose search engine that works for app search and log search. When you focus on solving a particular problem then you can build a product tailored to that problem which is far superior to the general purpose solution for that use case and ZincObserve is that tailored solution to log search and observability problem.

We also need to avoid any confusion on what we stand for as a commercial open source company.

We stand for building the best log search and observability solution in the world.

That is why ZincSearch has been moved to its own repo which is a better solution for app search than log search.

ZincObserve will be the product that will have our focus because it solves the log search problem even for existing ZincSearch users (90% users) much better. Wouldn't you want us to solve the problems and build solutions for 90% the users far more effectively than 100% of users in a less effective fashion? What is your use case?

For app search, I would recommend that people move on to vector search (There are specific products in that category) using products like milvus, weaviate, qdrant and more. There is yet another category for app search which is super fast type ahead search - typesense and meilisearch are good products in that category.

ZincObserve APIs for log search should get stable sooner than later.

Now a couple of specific points that you mentioned -

ZincObserve is not fully ES compatible

• ZincObserve is compatible with ElaticSearch using zPlane which is our enterprise offering. We will state this more clearly.

ZincSearch was originally under ZincLabs and was migrated to this new GitHub Org, while ZincObserve was moved over to the original ZincLabs to make it look like it has more followers.

• I have stated the reasons why we moved ZincSearch to its own repo above. ZincObserve has been part of ZincLabs org from the day its first line of code was written and has not been moved to it. I am not sure how we create the impression that ZincObserve has more followers. It's a product we hope to have more users in due course of time but that is not the case today. I am not sure, how we can even create that kind of impression for an open-source project on github that has its metrics (stars, forks, contributors, downloads, etc) publicly visible to anyone. That being said I would like to understand and fix that, as it's not the right thing to do if there is such a projection even if it is happening unintentionally. We could use some help here. Point us to the specific things and we will fix them.

It's just very confusing to find any information now, especially given that ZincObserve is not production ready and API keeps changing on a daily basis.

• Yes, APIs are constantly being added and some of them change too. This shows the speed and the amount of effort going into making the best log search and observability software in the world. Breaking changes to ZincObserve APIs for log search should stop sometime this or next month. It's our goal to avoid any confusion. What information are you looking for? Allow us to help by asking questions on github issues or on slack channel.

That being said, ZincSearch has a large user base (Many hundreds of users) running it in production environments. We will continue to ensure that we stand by these users and either help them migrate to ZincObserve for their use cases of log search or support their existing installations for app search.

Thank you for supporting us. We would appreciate any additional feedback you may have.

Dependency PRs are a pain 😭 There are so many of them and you can merge only one of them at a time. After you merge the first one, you need to update the next one by rebasing it and waiting for the CI to do its job, and so on with every PR. We will get to it eventually, but I guess it's a catching-up game and more PRs will come up again. These PRs don't really add any extra features to ZincSearch and are painful and hence have lower priority.

[gaby]

@prabhatsharma Yeah, even to me the main use case has been logs, syslog, and k8s logs. For which case ZO has been a blessing. For anything that uses ES api using ZincSearch has been the way to go.

Thank you for the suggestion of alternative apps to use. At least in my case i'm trying to fully replace ElasticSearch with something else to ingest Elastic Beats data and display it with Kibana. Even though both ZincSearch and ZO offer a UI it seems miles behind Kibana, which is understandable. I also tried sending Elastic Beats data to ZO but it wasn't happy about the amount of different fields and how/which to index. I also tried enabling indexing/search for all the fields but that made ZO super slow.

I was not aware about zPlane, any reason why this was a made an enterprise feature for ZO given that it was OSS for ZincSearch?

Would it help if people did PR's were multiple dependency updates were bundled together?

[prabhatsharma]

I know, for app search with ES API compatibility, ZincSearch is still the best choice for single node installations.

Upcoming release of Logs UI in OpenObserve has many new features which make it better than kibana. Would love some brutally honest feedback once you use it. Visualization and dashboard still need some more work. Do let us know, what is it that you would like to see.

ZO intentionally has no indexing, just partitioning. If have large amount of data you should partition based on the fields on which you would be querying. Performance gains of partitioning will be available only for the new records that get ingested though.

We plan to make zPlane source available with no restrictions on non-commercial use. We are still discussing this internally. Keeping the ES compatibility in ZincSearch, made it somewhat difficult to prioritize things on the API front that we wanted to build, hence we decided to push it outside of ZincObserve.