Implement the existing AWS API for all list and describe calls

[jhspaybar]

Rather than a separate client and separate API I'd prefer to just use the existing AWS SDK and command line tools to query my cluster state from this system, just without the risk of throttling that exists when speaking to the hosted API.

It'd also be cool if it implemented the various write APIs as well and forwarded them, but that's probably less valuable for me.

[jhspaybar]

Is this something you'd be open to taking as a pull request? It isn't clear to me where I can help.

[simplycloud]

@jhspaybar we are currently thinking to add cluster-state-service api support into the ecs-cli. this would allow us to continue to add functionality and features to a single common toolset for working with Amazon ECS.

we hope to have a proposal for what that might look like shortly and i'll reference it here so that anyone can comment and share ideas / feedback.

[jhspaybar]

Thanks @simplycloud for the reply. That does sound helpful. The primary use case I'm thinking about though is if I were to write a scheduler in Java using the existing AWS SDK for ECS and start to run into throttles, I'd like to be able to just change my target to something like localhost:1234 where the ecs-cluster-state agent is running and have everything continue to work just without risk of throttling from a flurry of describes.

[kiranmeduri]

I like the idea of @jhspaybar to conform to AWS ECS APIs for the following reasons

• Leverage AWS SDK across multiple languages as a path to interact with cluster-state-service.
• Scheduler or other client code may not need to change if they already are using AWS ECS APIs for getting cluster-state.

However I have following concerns which need to be addressed

• Authn/Authz support similar to AWS service (sigv4) (http://docs.aws.amazon.com/general/latest/gr/signature-version-4.html). At first we can ignore the verification, but need to set right expectation.
• Maintaining compatibility of API may become a blocker for experimentation. To solve this we may need to support two flavors of API which can bloat the code-base.
• Richer APIs may not be available via AWS SDK which can be useful for schedulers (e.g. streaming/watch style APIs)
• ECS will have APIs not supported by cluster-state-service which can cause confusion. For e.g. TaskDefinition management APIs.

Regarding the write APIs I agree that they can pass-through cluster-state-service. This can enable better task lifecycle management.

[simplycloud]

@jhspaybar @kiranmeduri suggestions on what we should do next here? I do have a few concerns around tying Blox to closely to the ECS APIs right now. We may want to add APIs and functionality specifically within Blox that is not supported in ECS and thus not part of the AWS SDK.

[jhspaybar]

My plan was to eventually return here with a pull request of some sort to get a conversation going if nothing else. I'm just getting some free time again and was going to modify the cloud formation template for Kinesis and then contribute a SQLite datastore first though.

[kylbarnes]

@jhspaybar let me know if you have any questions about the CloudFormation template stuff, as I did the originals. Before you get started on a second datastore though, we should all talk and come up with some common interfaces that will allow the datastores to be pluggable, as we'll probably want to add another like DynamoDB in the near future, and I don't want to see you get too far down the path of SQLite before we start thinking about the next datastore. Pluggable datastores can be complicated, and we should plan this out before any work starts.

[jhspaybar]

I've never used cloud formation before, so I'm currently trying to figure out how to parameterize sqs vs Kinesis. If this is obvious/easy for you, I wouldn't be broken up to see you contribute it, otherwise I'll get this figured out eventually :). As for the datastore, for now I was just expecting to implement the handful of methods on the datastore interface related to key/value storage of the full JSON blobs.

[yrsurya]

Any information on when V1.0 of blox is going to be release?