Enable Etcd compaction

[kylbarnes]

Currently, the cluster-state-service stores the entire ECS event-stream in Etcd. Over time, this can add up to quite a bit of data. We should enable automatic Etcd compaction by either enabling the --auto-compaction-retention=hours flag on the Etcd service, or by creating a job that periodically compacts Etcd.

[kylbarnes]

In the meantime, Etcd can be manually compacted by using the following steps. Replace abcdef1234 with your Etcd docker container id.

# Find current Etcd revision.
$ docker exec -it abcdef1234 /bin/sh
$ etcdctl -w json get --order=DESCEND --sort-by=MODIFY --limit=1 ecs
{"header":{"cluster_id":1234,"member_id":1234,"revision":50000,"raft_term":1}}

# Grab the "revision" value from the previous command.
# It's probably a good idea to subtract 1,000 from the revision number to keep the latest 1,000 mutations.
$ docker exec -it abcdef1234 /bin/sh
$ etcdctl compaction 49000
compacted revision 49000

# Wait for compaction to complete.
$ docker logs abcdef1234 2>&1 | grep compact
2017-05-15 12:00:00 | mvcc: store.index: compact 49000
2017-05-15 12:00:50 | mvcc: finished scheduled compaction at 49000 (took 50s)

# Defrag Etcd to free up disk space.
$ docker exec -it abcdef1234 /bin/sh
$ etcdctl defrag
Finished defragmenting etcd member[127.0.0.1:2379]