Custom state backends

[mjsqu]

I managed to stand up a working feature branch of meltano using AWS SSM Parameter Store as its state backend. The following state commands were analogous to the boto3 ssm client functions:

meltano state ... command	ssm client function
list	get_parameters_by_path(Path=)
set	put_parameter(Name=,Value=)
get	get_parameter(Name=)
clear	delete_parameter(Name=)

The coding I had to do was quite minimal as the meltano/src/core/state_store modules are really well written and provide a lot of the functionality already. The SSM module I added was a mash-up of the s3.py and the db.py modules:

• s3.py - contributed logic for connection to aws via boto3
• db.py - contributed the get() and set() methods amongst others, and meant I didn't need to declare my backend as a 'filesystem' type

It would be interesting to explore how meltano could provide a method of registering state backends in the project in the same way that taps/targets can be registered, should users wish to store their state somewhere different, such as:

• AWS Systems Manager Parameter Store (SSM)
• Google Cloud DataStore
• Azure App Configuration
• Any NoSQL service that provides key-value pair storage
• A REST API which exposes GET/PUT/DELETE endpoints for each meltano state operation

[mjsqu]

From: https://meltano.slack.com/archives/C013Z450LCD/p1700177112720099?thread_ts=1700118139.892419&cid=C013Z450LCD

(https://discuss.meltano.com/c/contributing - when the above Slack message expires)

[mjsqu]

And finally, providing some credentials, I was able to demonstrate working list, set and get commands:

Thanks for the help!

I'd really appreciate it if you could review my pyproject.toml as I've never written a Python package before, let alone a plugin - and also it would be good to get some guidance on naming conventions for the package, as I will probably try and work on Google Cloud DataStore and Azure next and want to lock in a solid naming standard, ideas:

meltano-state-aws-ssm-<backend|plugin|pluggable>
meltano-state-gcs-datastore-<backend|plugin|pluggable>

[edgarrmondragon]

Hey @mjsqu! I'll give your pyproject.toml a look but I actually similarly created a sample "plugin" package, though it uses Hatch instead of Poetry: https://github.com/edgarrmondragon/meltano-rocksdb-state-backend

[mjsqu]

Even better, I'll look over your repo as well! Thank you.

[mjsqu]

meltano-<service>-state-backend is a good naming convention, I'll rename mine

[mjsqu]

I imagine this could live elsewhere in future, but I made a very early start on a cookiecutter template for state-backends:
https://github.com/mjsqu/cookiecutter-meltano-state-backend