SSU Configuration Management: Packer & Ansible

Nowaday

There are two major parts involved: ansible and packer.

Packer is used to create an image from defined base image, copying ansible directory to that image and running locally a given ansible playbook.

Ansible is used to keep codified configuration of

• 'immutable' or occasionally modified image components(like packages, static parts of configuration etc.);
• 'dynamic' configuration for instances, depending on a set of instance properties
  • AWS region
  • Environment
  • Instance tags
  • etc.

Prepare

• Export samples

$ echo "export EDITOR=vim" >> ~/.bashrc
$ source ~/.bashrc
$ export AWS_PROFILE=
$ export AWS_ACCESS_KEY_ID=
$ export AWS_SECRET_ACCESS_KEY=
$ export AWS_DEFAULT_REGION=eu-west-1

$ echo "127.0.0.1" > ~ansible/inventory/hosts
$ export ANSIBLE_ANSIBLE_SSH_KEYDIR=`pwd`
$ export ANSIBLE_INVENTORY_ENABLED=amazon.aws.aws_ec2

$ ANSIBLE_PIPELINING(/etc/ansible/ansible.cfg) = True
$ DEFAULT_HOST_LIST(/etc/ansible/ansible.cfg) = ['/etc/ansible/hosts']
$ DEFAULT_REMOTE_USER(/etc/ansible/ansible.cfg) = myuser
$ RETRY_FILES_ENABLED(/etc/ansible/ansible.cfg) = False
$ ANSIBLE_VAULT_PASSWORD_FILE(/etc/ansible/ansible.cfg) =~/bin/ansible-vault-pass

$ export PACKER_LOG_PATH="packer.log"
$ export PACKER_LOG=1

• AWS Account and Cli

pip install awscli
aws configure --profile ssu-cm
export  AWS_PROFILE=ssu-cm
aws get-caller-identity

• Packer

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt-get update && sudo apt-get install packer
sudo cp /usr/bin/packer /usr/local/bin/packer

which packer
/usr/local/bin/packer
packer version
Packer v1.7.2
packer -autocomplete-install

• VirtualBox

• Ansible

Install Ansible by using virtualenv

$ sudo easy_install pip
$ sudo pip install -r ./requirements.txt
$ ansible --version

Troubleshooting SSH Connections

Sometimes when attempting an initial Ansible ping, it can fail in a very confusing way. If this happens, try adding something like this to your ~/.ssh/config file:

Host <hostname-prefix>*
  Port 22
  User <ssh-login-username>
  StrictHostKeyChecking=no
  UserKnownHostsFile=/dev/null

How to develop AMIs with Packer and Ansible

Things I wish they'd told me

Use in packer templates

Before using ansible provisioning in packer next packages should be installed into image:

• python >= 2.7
• python-setuptools
• python-pip
• ansible >= 2.4

{
  ...
  "provisioners": [
    ...
    {
      "type": "shell",
      "inline": [
        "sleep 10",
        "sudo yum install -y python2 python-setuptools",
        "curl 'https://bootstrap.pypa.io/get-pip.py' | sudo python",
        "sudo pip install ansible"
      ]
    },
    {
      "type": "ansible-local",
      "playbook_file": "../ansible/run_playbook.yaml",
      "playbook_dir": "../ansible",
      "role_paths": [
        "../ansible/roles"
      ],
      "clean_staging_directory": true
    }
    ...
  ]
  ...
}

Difference between ansible running remotely and locally

When packer starts ansible for ansible-local provisioner internal ansible variable inventory_hostname is '127.0.0.1'. This way we can distinguish between AMI builds and regular ansible runs in role files:

- name: Example AMI stage
  block:
    - name: Run on image build only
      command: echo "local"
  when: inventory_hostname == '127.0.0.1'
  
- name: Example live instance stage
  block:
    - name: Run on remote instance only
      command: echo "remote"
  when: inventory_hostname != '127.0.0.1'  

Remote 'live' systems operations

We are using dynamic inventory based on ansible recommended script with a minor changes in code and configuration. Example:

⟫ AWS_PROFILE=project EC2_INI_PATH=./inventory/ec2.ini ANSIBLE_INVENTORY=./inventory/ec2.py \
    ansible -m ping tag_Project_SSU_CM
project_devops_eu_west_1_openvpn | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    },
    "changed": false,
    "ping": "pong"
}
...

Develop the Ansible role locally

When you try to debug something in packer by making a change in the Ansible role you want to see the effect immediately. To do that just symlink your project in ~/.ansible/roles with whatever name you are referencing it with. So, for example, a project refered to as ansible-role-test which is locally in ansible-role-test:

`ln -s $(pwd)/ansible-role-test ~/.ansible/roles/ansible-role-test`

Don't run packer with -debug

If you do this IT WILL ASK YOU FOR INPUT BETWEEN EVERY STEP. This is perhaps one of the most annoying debug behavior ever.

Take advantage of the Ansible debugger with -on-error=ask and a specified keypair

If you've got a box set up (maybe from running packer with -on-error=ask) you can run the ansible debugger. It's a lot like pdb for Ansible.

Now packer will run until an error and then ask you what to do:

==> my-test-ami: [c] Clean up and exit, [a] abort without cleanup, or [r] retry step (build may fail even if retry succeeds)?

By specifying the keypair you can use a key you already have in AWS to SSH in:

  "builders": [{
            "ssh_keypair_name": "the_key_is_automation",
            "ssh_private_key_file": "~/.ssh/tempKeys/the_key_is_automation.pem",
            "name": "my-test-ami",
            "type": "amazon-ebs",
            ...

Taking a look at what's wrong becomes as easy as SSH:

ssh -i ~/.ssh/tempkeys/the_key_is_automation.pem ec2-user@xx.xx.xx.xx

Delete the AMI created by Packer

First, get the AMI ID value using:

aws ec2 describe-images --filters "Name=tag:Name,Values=my-server*" -region=eu-west-1 --query 'Images[*].{ID:ImageId}'

Then delete the AMI

aws ec2 deregister-image --image-id ami-<value>

Launch an instance from the AMI

First, I launch a new instance from my AMI. I use the same subnet as for Packer, take the ID of the Dedicated Host I created earlier on, and I specify an IAM Role, which has the required permissions to use SSM.

aws ec2 run-instances --instance-type mac1.metal \
                      --subnet-id subnet-0d3d002af8EXAMPLE \
                      --image-id ami-0d6fb7542bb0a8da3 \
                      --region us-east-2 \
                      --placement HostId=h-03464da766df06f5c \
                      --iam-instance-profile Name=SSMInstanceRole

Connecting to the instance

I can start an SSH session to the instance via SSM with the following command:

aws ssm start-session --target <YOUR_INSTANCE_ID> --region us-east-2

Session Manager

When using session_manager the machine running Packer must have the AWS Session Manager Plugin installed and within the users' system path. Connectivity via the session_manager interface establishes a secure tunnel between the local host and the remote host on an available local port to the specified ssh_port. See Session Manager Connections for more information.

Assume Role Configuration

source "amazon-ebs" "example" {
    assume_role {
        role_arn     = "arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME"
        session_name = "SESSION_NAME"
        external_id  = "EXTERNAL_ID"
    }
}

Packer Filters

Examples

  security_group_filter {
    filters = {
      "tag:Class": "packer"
    }
  }
  vpc_filter {
    filters = {
      "tag:Name": "myVPC",
      "isDefault": "false"
    }
  }
  subnet_filter {
    filters = {
      "state": "available",
      "tag:Name": "*public*"
    }
    random = true
  }

This example uses a amazon-ami data source rather than a specific AMI. this allows us to use the same filter regardless of what region we're in, among other benefits.

data "amazon-ami" "example" {
  filters = {
    virtualization-type = "hvm"
    name                = "*Windows_Server-2012*English-64Bit-Base*"
    root-device-type    = "ebs"
  }
  owners      = ["amazon"]
  most_recent = true
  # Access Region Configuration
  region      = "us-east-1"
}

  source_ami_filter {
    filters = {
       virtualization-type = "hvm"
       name = "ubuntu/images/\*ubuntu-xenial-16.04-amd64-server-\*"
       root-device-type = "ebs"
    }
    owners = ["099720109477"]
    most_recent = true
  }

Limiting Playbook/Task Runs

When writing Ansible, sometimes it is tedious to make a change in a playbook or task, then run the playbook It can sometimes be very helpful to run a module directly as shown above, but only against a single development host.

Limit to one or more hosts This is required when one wants to run a playbook against a host group, but only against one or more members of that group.

Limit to one host

ansible-playbook playbooks/PLAYBOOK_NAME.yml --limit "host1"

Limit to multiple hosts

ansible-playbook playbooks/PLAYBOOK_NAME.yml --limit "host1,host2"

Negated limit. NOTE: Single quotes MUST be used to prevent bash interpolation.

ansible-playbook playbooks/PLAYBOOK_NAME.yml --limit 'all:!host1'

Limit to host group

ansible-playbook playbooks/PLAYBOOK_NAME.yml --limit 'group1'

Limiting Tasks with Tags

Limit to all tags matching install

ansible-playbook playbooks/PLAYBOOK_NAME.yml --tags 'install'

Skip any tag matching sudoers

ansible-playbook playbooks/PLAYBOOK_NAME.yml --skip-tags 'sudoers'

Troubleshooting Ansible http://docs.ansible.com/ansible/playbooks_checkmode.html

• Pre-commit

pip3 install pre-commit pre-commit-hooks setup-cfg-fmt
sudo apt install shellcheck
curl https://raw.githubusercontent.com/terraform-linters/tflint/master/install_linux.sh | sudo bash
pre-commit install
pre-commit run -a

Run GitHub SuperLinter localy https://github.com/github/super-linter

export cm_repo_path=`pwd` \
    && echo "set cm_repo_path to $cm_repo_path" \
    && docker run -e RUN_LOCAL=true -v $cm_repo_path:/tmp/lint github/super-linter