Testing repositories with private submodules in Travis

Recently I’ve been working on a project we’re testing in Travis CI and we need to include some submodules referencing private GitHub repositories then run the tests on the combined code. Naturally Travis fails to retrieve the submoduled repositories, because it’s not authorised to do so. This is how we’re getting around that…

Here’s the .travis.yml file:

git:
  # Tell Travis not to try to load submodules
  # We'll do that further down
  submodules: false

before_install:
  # This is just updating all the things
  - sudo apt-get update > /dev/null
  # The .ssh directory probably exists, but why take the risk?
  - mkdir -p ~/.ssh
  # This command clears all SSH keys from the current session,
  # preventing the GitHub repository deploy key from being used
  - ssh-add -D
  # This adds a directive to the SSH config which specifies our
  # deploy user's key is used for authenticating with all connections
  # to github.com
  - echo -e "Host github.com\n  User my-machine-user\n  IdentityFile ${TRAVIS_BUILD_DIR}/ci/ssh/insecure-key\n" >> ~/.ssh/config
  # SSH keys will be unusable unless they are unreadable by
  # anyone other than the owner
  - chmod 600 ${TRAVIS_BUILD_DIR}/ci/ssh/*
  # We need to add the SSH signature for GitHub to the SSH known_hosts
  - cp ${TRAVIS_BUILD_DIR}/ci/ssh/known_hosts ~/.ssh/known_hosts
  # Now update the submodules we're using
  - git submodule update --init --recursive

script:

  # Run the tests
  - make test

Here’s an overview of the steps we’re going to take:

1. Stop Travis from processing submodules automattically
2. Specify the submodule URLs in SSH format
3. Ship a private and public keypair for a GitHub machine user
4. Configure SSH to use the private key for the GitHub machine user
5. Process the submodules

We specify the repository URLs in the SSH form, e.g. git@github.com:Account/repository.git, because this allows us to use public/private key to authenticate with GitHub over SSH, which allows us to authenticate in our headless environment (Travis) with no user input.

In the ci/ssh/ directory we have the public AND PRIVATE keypair for a GitHub machine user with the username “my-machine-user”, and we’ve added the public key to their account on GitHub. Note that the , but we don’t care much because the user only has read permissions. This machine user should never have more than read permissions on any GitHub account, because we’re distributing the private key and the private key has no passphrase; overall though, in our particular situation we don’t care much because the code is open source and we’re merely being shy, not enacting security requirements.

To configure SSH to use the machine user’s private key to authenticate all SSH connections, we print some information to ~/.ssh/config/, i.e. echo -e "Host github.com\n User my-machine-user\n… etc. You’ll need to replace my-machine-user with the username of your actual machine user.

One issue I got stuck on for a while, is that Travis will have already authenticated with GitHub using the deploy key for the primary repository, which authenticates them as a user equivalent to that repository (who has no privileges on other repositories). To get around this, we use ssh-add -D to clear all keys from the SSH agent, and the agent then uses the key specified in the config.

Photo by Max Wolfe