workflow-tools: start using GitHub Actions at scale

Microservice architecture may have dozens or even hundreds of lookalikes services that require similar CI/CD workflows. With infrastructure as code approach taken by the GitHub Actions, why not using code generation? Provisioning a repository for a new microservice may also be automated.

I’ve just opensourced an internal tool we use here at ANNA to automate GitHub Actions-based CI/CD workflow generation. workflow-tools is a collection of CLI tools used to:

  • generate GitHub Actions workflows from a Jijna2-template
  • set GitHub secrets (used by the workflows)

Examples

workflow-tools

Let’s consider a real world example: setting up a GitHub Actions workflow for the workflow-tools repository itself. We need:

  1. Generate a GitHub Action workflow using workflow_generator tool
  2. Set GitHub Secrets the workflow needs using workflow_secret tool

Generating workflow

1
2
3
4
5
WORKFLOW_RUNNER_VERSION=ubuntu-latest \
workflow_generator \
docs/examples/master.tmpl \
~/PATH-TO-YOUR-REPO/.github/workflows/master.yml \
-e docs/examples/envfile

First, we define a Jinja2-template for the workflow (see a master.yml at line 3). Variables to be substituted should be marked up this way:

[[ workflow.your_variable ]]

When rendering the resulting file, workflow-generator tool substitutes the markup with the value of corresponding environment variable:

WORKFLOW_YOUR_VARIABLE

The environment variable can be set globally, for a single command run, or can be read from the envfile specified by the option flag -e (see an envfile at line 5).

Envfile comes in handy when a template uses many variables at once. It’s also easier to share variables between the templates using envfile. The envfile has the lowest precedence, it can be overridden (line 1).

Setting secrets

Now that we generated the workflow, let’s set a GitHub secret used by the template:

1
2
3
4
5
6
7
workflow_secret \
  --owner=anna-money \
  --repo=workflow-tools \
  --token="YOUR-PERSONAL-ACCESS-TOKEN" \
 update \
   --key=PYPI_PUSH_USER \
   --value=YOUR_SECRET_VALUE

First, we need to get a personal access token (see line 4). workflow_secret tool has multiple commands (see Tools autodocs). To set up new or update existing secret update command is used (line 5). The command accepts --key and --value options (lines 6, 7). workflow_secret also have tool-wide options used for each command: --owner (GitHub user, line 2), --repo (GitHub repository name, line 3) and --token.

Finally, let’s check what secrets are set for the repository:

1
2
3
4
5
workflow_secret \
  --owner=anna-money \
  --repo=workflow-tools \
  --token="YOUR-PERSONAL-ACCESS-TOKEN" \
 list

Why workflow-tools?

It’s a common practice to use templating in system administration, especially once DevOps practices have spread widely. Some tools Ansible template module have embedded templating mechanisms. It may be of great help where the tools originally fit. But it may be an overhead when applied to GitHub Actions configuration.

Other tools like envsubst have been there for a long time and proofed being useful. But one-size-fits-all approach may affect usability in a negative way.

The workflow-tools approach is to make Jinja2 standalone compiler with usability focused on the GitHub Actions workflows.

GitHub secrets setting tool from the workflow-tools package follows the same approach: simple CLI that does one thing and does it well, but not too general-purpose as it could be.

Who should use workflow-tools?

If you are using GitHub Actions and similar workflows are needed for multiple repositories, then workflow-tools package is for you.