Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a feature to clone modules via SSH keys during terraform_validate stage #426

Open
gartemiev opened this issue Aug 11, 2022 · 8 comments
Labels
area/docker documentation Improvements or additions to documentation estimate/2h Need 2 hours to be done good first issue Good for newcomers

Comments

@gartemiev
Copy link

It would be great to add a feature to be able to clone modules via SSH keys during terraform_validate stage.

@gartemiev gartemiev added the feature New feature or request label Aug 11, 2022
@MaxymVlasov
Copy link
Collaborator

That's from https://ukrops.slack.com/archives/C3R001CHW/p1659634407658399

TL;DR Possible solution: apk add --no-cache openssh in the last Dockefile stage.

@MaxymVlasov MaxymVlasov added the good first issue Good for newcomers label Aug 11, 2022
@tofupup
Copy link
Contributor

tofupup commented Sep 7, 2022

@MaxymVlasov @yermulnik Wanted to get your thoughts on implementation here before I start working this...I think this moves out of just the terraform_validate hook, since terraform init gets called in common::terraform_init by at least terraform_validate and terraform_providers_lock currently. Please let me know if I'm missing a simpler solution here.

Primary issues are getting SSH keys and public SSH host keys inside the container.

I think my preference initially is to provide ssh-agent access via docker bind (and having a section in the README about running the container with SSH access), but not support binding the user's .ssh directory (since those keys would be more likely to have a passphrase). And for the known_hosts, I think the 2nd option I list below (ssh-keyscan), though needing more code in _common.sh, is better than binding a known_hosts file as the hooks will then run even if the user has not previously connected to the host.

For keys:

  • If ssh-agent is running when docker is called, we'd probably want to give access to that instead of trying to copy in key files (which may not even exist in the case of running the container on a remote system, where ssh-agent access is passed through ssh -A or similar). So having the user execute something like:
docker run -v ${SSH_AUTH_SOCK}:/run/ssh.socket -e SSH_AUTH_SOCK=/run/ssh.socket

would be necessary.

  • If there's no ssh-agent, the user would have to provide a passphraseless key, either by binding it via the docker run or maybe passing via an environment option, but neither of these seem ideal.

For public host keys:

  • Require the user to provide a known_hosts file that contains the remote host signatures via binding

or

  • Before running terraform init, we have to scan for any source = URLs that match the three SSH styles supported by terraform:
  source = "git@github.com:hashicorp/example.git"
  source = "git::ssh://username@example.com/storage.git"
  source = "git::username@example.com:storage.git"

it's easy enough to parse out the hostnames from these, and run ssh-keyscan with output to a temporary known_hosts file. But it does add complexity to the common::terraform_init function. This would work even if the user is using a private custom SSH git repo.

@gartemiev do you think this would meet your needs? Is there a case that makes more sense to you?

@yermulnik
Copy link
Collaborator

Well, yeah, that's a bit of a hassle 🤔
Mounting auth socket would mean some of the headless setups might fail as they might not have ssh agent running (or ssh keys added to it).
On the other hand mounting .ssh dir (or explicitly priv/pub keys only) would mean a need to provide a passphrase if key requires it, which again might be a point of failure for headless setups.
There's another option to globally replace GIT URLs to use https://, which might not work for some setups with self-hosted GIT implementations where access via HTTPS isn't provided.

I'm leaning to an option to mount .ssh dir (or explicitly priv/pub keys only) along with requirement for users to provide passphrase-less key in case they're running pre-commit-terraform Docker container against GitHub-hosted Terraform modules. This would allow headless setups to get provided with such a key to access GH via SSH protocol. Kinda a big caution message in the README on how to do that with as less hassle as possible (plus what Max wrote re ssh installation into Docker image).

What I actually mean is that we should not interfere with any user-related stuff, but only provide an option on how to overcome GitHub's limitation with user-side actions.

Does it make sense to you folks?

@tofupup
Copy link
Contributor

tofupup commented Sep 8, 2022

I spent some time experimenting with this, and I think @yermulnik's plan for just providing documentation on how to solve this (with openssh-client installed in the image) is workable. Especially trying to follow @MaxymVlasov's thoughts on nocode. Depending on what exactly the user needs, it might take some setup on their side, but once setup I think it would work well. We could also provide a doc section on binding the ssh-agent socket where useful.

We could put a simple plan in the README, but having a full explanation on some of the setup options may necessitate a separate file (or a wiki entry if the project had one).

I've put notes below (primarily for myself), but if agreed that @yermulnik's plan makes sense, I can put together a pass at docs and the minor Dockerfile change. What are thoughts on having a separate file for more thorough documentation vs a wiki?

Notes

For host keys, while not ideal I think using SSH option StrictHostKeyChecking=no might be acceptable, since we're already using passphraseless keys.

If the user wants to bind their existing .ssh directory, and not have to modify their config, they can use -e GIT_SSH_COMMAND="/usr/bin/ssh -o StrictHostKeyChecking=no" when running the container, and the host key problem should be resolved.

On Github this plan would allow using read-only repository level deployment keys, or account level SSH keys. Other systems (private hosted git repos, Codecommit, etc) shouldn't have an issue either. One issue with Github repository deployment keys is they have to be unique per repository, so if there are multiple private github repos referenced in the code, the source = statements have to refer to a separate Host entries for each repository. I put an example of how I think this would have to be handled below.

As an example, let's say I have 2 Github repositories, tofupup/private-tf-security-group and tofupup/private-tf-vpc. For read-only deployment keys for each repository:

  1. (optional) create a .ssh directory for this project
$ PROJSSH="/home/john/src/proj-sshdir"
$ mkdir -p $PROJSSH
  1. Create passphraseless SSH keys for both repos:
$ ssh-keygen -t ed25519 -C "github_security_group_buildkey" -N "" -f "$PROJSSH/id_github_security_group_buildkey"
$ ssh-keygen -t ed25519 -C "github_vpc_buildkey" -N "" -f "$PROJSSH/id_github_vpc_buildkey"
  1. Add keys to respective repositories (via web or gh CLI)
$ gh repo deploy-key add "$PROJSSH/id_github_security_group_buildkey.pub" -t "john security_group buildkey" -R tofupup/private-tf-security-group
$ gh repo deploy-key add "$PROJSSH/id_github_vpc_buildkey.pub" -t "john vpc buildkey" -R tofupup/private-tf-vpc
  1. Create ssh config
❯ cat $PROJSSH/config
Host gh_security_group
        Hostname github.com
        ControlMaster no
        IdentitiesOnly yes
        IdentityFile ~/.ssh/id_github_security_group_buildkey
        StrictHostKeyChecking no

Host gh_vpc
        Hostname github.com
        ControlMaster no
        IdentitiesOnly yes
        IdentityFile ~/.ssh/id_github_vpc_buildkey
        StrictHostKeyChecking no
  1. source statements in module must reference these new hostnames:
  source = "git::ssh://git@gh_security_group/tofupup/private-tf-security-group"
  source = "git::ssh://git@gh_vpc/tofupup/private-tf-vpc"

@MaxymVlasov
Copy link
Collaborator

MaxymVlasov commented Sep 8, 2022

A separate file is OK, put it in the docs/ folder.

Wiki has lack of search, so for me it is useless.

I could introduce the mkdocs->gh-pages integration later, if we will have many docs
(I'll already have needed skeleton and settings, so it should be done in less than 4 hours, when the time comes)

@yermulnik
Copy link
Collaborator

+1 to file instead of wiki.

@tofupup Thanks for your time and effort. What you wrote in the latest post looks reasonable and good to me.
The only usual nitpick bit would be to wrap var into double quotes mkdir -p "$PROJSSH" 😂

@MaxymVlasov MaxymVlasov changed the title Add a feture to clone modules via SSH keys during terraform_validate stage Add a feature to clone modules via SSH keys during terraform_validate stage Oct 6, 2022
@github-actions
Copy link

github-actions bot commented Nov 6, 2022

This issue has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this issue will be closed in 10 days

@github-actions github-actions bot added the stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 6, 2022
@MaxymVlasov MaxymVlasov added documentation Improvements or additions to documentation and removed stale Denotes an issue or PR has remained open with no activity and has become stale. labels Nov 7, 2022
@github-actions
Copy link

github-actions bot commented Dec 8, 2022

This issue has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this issue will be closed in 10 days

@github-actions github-actions bot added the stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 8, 2022
@MaxymVlasov MaxymVlasov removed the stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 8, 2022
@MaxymVlasov MaxymVlasov added estimate/2h Need 2 hours to be done and removed feature New feature or request hacktoberfest labels Dec 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/docker documentation Improvements or additions to documentation estimate/2h Need 2 hours to be done good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

4 participants