Clean Termination

[victorstewart]

what is the correct termination sequence? first hse_kvdb_kvs_close each kvs then call hse_kvdb_fini?

and what if an application terminates without conducting that shutdown sequence? maybe a scheduler just kills the process. will the kvdb and kvs(s) still always be left in a recoverable state? Maybe just any data not flushed to disk between the last flush and the termination be lost?

and if an application throws a nonrecoverable signal like a SIGSEGV, should we conduct that sequence in the signal catcher?

[tristan957]

Hi Victor,

I am a newish contributor to the project and had a similar question a week or two back. My understanding is that terminating a program while KVSs/KVDBs are open is fine in the sense that you will be able to re-open the KVSs/KVDBs next time you start a program, and be able to retrieve data that had previously been flushed to disk.

The correct sequence for shutting down an HSE application is

1. determine what to do with any active transactions (commit/abort/orphan)
2. for each kvs, call hse_kvdb_kvs_close()
3. for each kvdb, call hse_kvdb_close()
4. after all kvdbs have been closed, call hse_kvdb_fini()

You can remember this sequence as the opposite of what you did to start your application.

I would definitely recommend hooking up a signal handler to ensure data integrity with regard to unexpected application terminations.

I will confirm the data integrity question tomorrow and get back to you. I will also make it a point to read the documentation, and see where docs may be lacking in this regard, so that I can update them for our upcoming 1.9 release.

[victorstewart]

thanks so much @tristan957 ! I'll leave this open for now until we come to further closure on that topic.

[victorstewart]

oh and just to correct a typo, it's hse_kvdb_kvs_close not hse_kvs_close.

and none of these functions state it in their comments, but I assume one or all of these functions flush their latest data state to disk?

[davidboles]

oh and just to correct a typo, it's hse_kvdb_kvs_close not hse_kvs_close.

and none of these functions state it in their comments, but I assume one or all of these functions flush their latest data state to disk?

That is correct - the sequence given by Tristan is the clean shutdown path that will cause all data to be flushed to media. If you do not perform this sequence, then depending on how fast the application exits and whether it has called hse_kvdb_sync() there may be a window of data which does not make it to media. The database will open to a consistent state regardless.

If you were to finish all operations, then call and return from hse_kvdb_sync(), and then exit in any way all data will be on media. Startup may/will be slightly faster as the engine doesn't have to replay anything to determine what the clean state is, so it's better to do the clean shutdown path for controlled termination.

[victorstewart]

I implemented that logic in a catch-all signal catcher.

My two termination paths are either... 1) the application crashed or 2) the scheduler signaled the application to gracefully shutdown with a SIGINT. So both handled.