Skip to content
This repository has been archived by the owner on Jan 23, 2022. It is now read-only.
/ s3-buddy Public archive

S3 wrapper that makes your life easier

License

Notifications You must be signed in to change notification settings

higee/s3-buddy

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

12 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Table of Contents

Python Version MIT License

app.py : get basic stats of S3 buckets 🎮

app2.py : dynamically rename S3 objects within/between buckets 🎮



1. app.py : get basic stats of S3 buckets

About

  • aws cli lets you figure out the number of S3 objects and their size .
  • What if you want to know hourly/daily/monthly stats within a given range of specific dates?

Demo

demo

Prerequisite

  • the profile in your credentials must have an access to S3 buckets
$ cat ~/.aws/credentials
>>>
[higee]
aws_access_key_id = something
aws_secret_access_key = something

Install

$ cd {somewhere}
$ git clone https://github.com/higee/s3-buddy
$ cd s3-buddy
$ pip install -r requirements.txt

Example

$ python app.py --help
$ python app.py --start-date 2019-02-01 --end-date 2019-02-01 \
                --bucket higee-bucket --path incoming --interval hour \
                --profile higee
$ python app.py --start-date 2019-02-01 --end-date 2019-02-03 \
                --bucket higee-bucket --path incoming --interval day \
                --profile higee
$ python app.py --start-date 2019-01-01 --end-date 2019-03-01 \
                --bucket higee-bucket --path incoming --interval month \
                --profile higee
$ python app.py --start-date 2019-02-01 --end-date 2020-01-01 \
                --bucket higee-bucket --path incoming --interval year \
                --profile higee
$ python app.py --start-date 2019-02-01 --end-date 2019-02-01 \
                --bucket higee-bucket --path incoming --interval hour \
                --delta 6 --profile higee


2. app2.py : dynamically rename S3 objects within/between buckets

About

  • aws cli lets you copy or move objects.
  • What if you need to dynamically rename keys or folders as well?
  • Term key and folder are slightly different from original meanings
    • example : s3://higee-bucket/incoming/2019/01/01/00/higee.log
      • bucket : higee-bucket
      • folder : incoming/2019/01/01/00/
      • key : higee.log
  • this app renames objects by creating copies with new name.
  • So set delete to True if you want to remove them.

Demo

demo

Prerequisite

  • the profile in your credentials must have an access to S3 buckets
$ cat ~/.aws/credentials
>>>
[higee]
aws_access_key_id = something
aws_secret_access_key = something

Install

$ cd {somewhere}
$ git clone https://github.com/higee/s3-buddy
$ cd s3-buddy
$ pip install -r requirements.txt

Arguments

argument description required default
profile needs access to source-bucket and dest-bucket
source-bucket S3 bucket with objects to be renamed
dest-bucket S3 bucket where renamed objects will be moved to source-bucket
dryrun whether you want to test it first True
delete whether you want to remove source objects False
source-folder-key-prefix prefix of S3 objects you want to rename
source-key-str sub-string of key to be replaced by dest-key-str
dest-key-str sub-string that will replace source-key-str
source-folder-str original folder name
dest-folder-str new folder name that will replace dest-folder-str

Example

$ python app2.py --help
$ python app2.py --source-folder-key-prefix incoming/2019/01/01/01/higee-staging \
                 --source-bucket higee-bucket --source-key-str staging \
                 --dest-key-str prod --profile higee
$ python app2.py --source-bucket higee-bucket --dest-bucket higee-bucket-2 \
                 --source-folder-key-prefix incoming/2019/01/01/02/higee-staging \
                 --source-key-str staging --dest-key-str dev --profile higee
$ python app2.py --source-folder-key-prefix incoming/2019/02 \
                 --source-bucket higee-bucket --source-folder-str incoming \
                 --dest-folder-str processed --profile higee
$ python app2.py --source-bucket higee-bucket --dest-bucket higee-bucket-2 \
                 --source-folder-key-prefix incoming/2019/03 \
                 --source-folder-str incoming --dest-folder-str processed \
                 --profile higee
  • 💣 execute the command without dryrun 💣
    • source : s3://higee-bucket/incoming/2019/01/01/01/higee*
    • dest : s3://higee-bucket/incoming/2019/01/01/01/higee-staging*
    • dryurn : False
    • delete source objects : False
$ python app2.py --source-folder-key-prefix incoming/2019/01/01/01 \
                 --source-bucket higee-bucket --source-key-str higee \
                 --dest-key-str higee-staging --dryrun False --profile higee
$ python app2.py --source-folder-key-prefix incoming/2019/01 \
                 --source-bucket higee-bucket --source-key-str higee \
                 --dest-key-str higee-staging --dryrun False --delete True \
                 --profile higee
$ python app2.py --source-bucket higee-bucket --dest-bucket higee-bucket-2 \
                 --source-folder-key-prefix incoming/2019/01/01/01/02/higee-staging \
                 --source-key-str staging --dest-key-str dev --profile higee

About

S3 wrapper that makes your life easier

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages