HSE 2.0 Pre-release

[hse-root]

We are excited to announce the pre-release of HSE 2.0. This pre-release represents a significant change to how HSE applications are deployed and managed. With HSE 2.0, we hope to address much of the feedback we have received from the user community.

HSE 2.0 also represents a transition in how HSE is developed. HSE has been open source and available on GitHub since April 27, 2020. However, GitHub pull requests have not been part of our development workflow. That is changing with HSE 2.0, and soon you will see all changes to HSE as GitHub PRs.

We look forward to your feedback as we continue to make new release candidates available leading up to the full HSE 2.0 release.

Summary of Changes

HSE now runs entirely in user space on common Linux file systems. Specifically, the mpool kernel module is no longer required. This means that HSE 2.0 storage is not backward compatible with HSE 1.x.

Configuration and logging mechanisms now conform to best practices for containerized and cloud-native applications. This makes it easier to develop HSE applications that can run anywhere.

Transaction and cursor semantics now closely align with those found in other databases and storage engines. This makes it easier to port applications written for other storage engines to HSE.

We made syntactic changes to the native C API making it easier to both maintain backward compatibility going forward and develop alternate language bindings. This means that HSE 2.0 APIs are not backward compatible with HSE 1.x, though porting to HSE 2.0 is straight-forward.

Python language bindings are available for the HSE 2.0 API. We now use these for much of our test infrastructure, making is easier and faster to develop test cases.

In addition to these user-visible changes, we also redesigned the HSE data ingest pipeline to make it simpler, easier to maintain, more flexible, and better suited for future work that we are planning.

Status

The HSE 2.0 pre-release is ready for you to test and provide feedback. We are still working through known issues that we expect to address in successive release candidates. The performance of HSE 2.0 is generally on par with HSE 1.x, though there are still gaps that we are working to close.

We are also migrating tests to the HSE project repos and integrating them with GitHub Actions. This is an on-going process but what is available in the pre-release provides good coverage.

Call to Action

• Read the documentation to understand how HSE has changed.
• Test the latest release candidate and provide your feedback.
• Migrate your HSE 1.x applications to HSE 2.0. While HSE 1.x is not going away immediately, it is no longer being actively developed.

Cheers,
HSE Project Team

[kyoungho-koo]

Hello. I was doing research using HSE version 1.8.0.
I had very positive thoughts about the HSE structure using the existing Mpool.
Using Mpool eliminates the file system overhead.
This is thought to provide more efficient storage access.
And I know there have been attempts to bring Mpool to the Linux kernel mainstream.
What is the reason for not maintaining the Mpool structure?

[smoyergh]

As you note we submitted mpool to Linux upstream. Unfortunately, the upstream maintainers decided that object storage functionality (like mpool) should reside in user space rather than the kernel.

This decision made it necessary to migrate HSE to file-based storage. Having HSE depend on mpool as an out-of-tree kernel module is both a barrier to adoption and a maintenance burden.