Building the perfect GitHub CI workflow for your frontend team

What would constitute a "good" GitHub CI workflow?

I'm going to throw my best "engineer response" at this question: it depends! Your team might have specific needs or objectives that would make some of my tips not as useful to you as they could be. However, for this article, we need some guidelines that I think would be universal when it comes to building efficient GitHub workflows, such as:

• cost-saving: bring the "build minutes" down to the lowest possible value to not have a massive bill at the end of the month.
• efficient: your team's time is precious, the workflow should be as fast as possible, but also fast to fail if something were to go wrong
• well-architected: each step has a purpose, and might depend on other steps. This also means not running "useless steps".

Now that we've established those guidelines, let's take a look at one of the most important tips of this article.

One workflow to rule them all

Let's consider a typical set of tasks a frontend team would run on every PR:

1. Lint
2. Formatting
3. Type checking
4. Unit test
5. Build
6. End-to-end tests, maybe on different browsers

Running those in separate workflows might look like the most straightforward way to architect those tasks. However, if something as simple as the lint task fails, there's no way you can stop your expensive tasks like build or your end-to-end tests from running. And that, my friends, is not very efficient.

Workflows run in parallel, and there's no way for them to interact with one another. Thus, you can't cancel a workflow due to another workflow's failed state. You're stuck running all the workflows in every PR.

To address this, I chose to combine all my workflows into one. All the tasks that were independent workflows before became part of the same unique workflow, but this time, as jobs.

1
# In this example, lint-format is a job among many others in a bigger GitHub workflow.
2
# This job has 3 steps: Checking out the code, running the lint command, and running the formatting command.
3

4
jobs:
5
  lint-format:
6
    runs-on: ubuntu-latest
7
    strategy:
8
      matrix:
9
      node: [12]
10
    steps:
11
      - name: Checkout Commit
12
      uses: actions/checkout@v2
13
      - name: Use Node.js ${{ matrix.node }}
14
      uses: actions/setup-node@v1
15
      with:
16
        node-version: ${{ matrix.node }}
17
      - name: Run lint
18
      run: |
19
        yarn lint
20
      - name: Run prettier
21
      run: |
22
        yarn format

The cool thing about jobs is that you can run them sequentially or parallel as you please! GitHub provides a handy keyword called needs that lets you set one or several jobs as dependencies, thus preventing a given job to start unless the dependent jobs have successfully run. This allows us to:

• Fail the workflow fast. If a key job fails, the workflow is marked as failed on your PR as soon as possible
• Avoid running useless expensive tasks on a "doomed to fail" workflow run

1
# In this workflow excerpt, the type-check and unit-test jobs run in parallel whereas the
2
# build job "needs" these 2 jobs to be successful to be kicked off.
3
# Thus, if any of type-check or unit-test were to fail, the build job will not start and the
4
# whole workflow will be marked as "failed".
5

6
jobs:
7
  type-check:
8
    runs-on: ubuntu-latest
9
    strategy:
10
      matrix:
11
      node: [12]
12
    steps:
13
      - name: Checkout Commit
14
      uses: actions/checkout@v2
15
      - name: Use Node.js ${{ matrix.node }}
16
      uses: actions/setup-node@v1
17
      with:
18
        node-version: ${{ matrix.node }}
19
      - name: Check types
20
      run: |
21
        yarn type-check
22
  unit-test:
23
    runs-on: ubuntu-latest
24
    strategy:
25
      matrix:
26
      node: [12]
27
    steps:
28
      - name: Checkout Commit
29
      uses: actions/checkout@v2
30
      - name: Use Node.js ${{ matrix.node }}
31
      uses: actions/setup-node@v1
32
      with:
33
        node-version: ${{ matrix.node }}
34
      - name: Run test
35
      run: |
36
        yarn test
37
  build:
38
    runs-on: ubuntu-latest
39
    strategy:
40
      matrix:
41
      node: [12]
42
    needs: [type-check, unit-test]
43
    steps:
44
      - name: Checkout Commit
45
      uses: actions/checkout@v2
46
      - name: Use Node.js ${{ matrix.node }}
47
      uses: actions/setup-node@v1
48
      with:
49
        node-version: ${{ matrix.node }}
50
      - name: Run build
51
      run: |
52
        yarn build

You may be wondering: what job should be run in parallel and what job needs to be run sequentially? That will depend on the needs of your team.

On my end, I tend to parallelize unit testing, linting, and type-checking for example. These steps are generally fast and inexpensive to run thus I do not feel they need to depend on each other in most cases. However, I'd require a job such as build to only run if those three jobs above are successful, i.e. run it sequentially.

The screenshot below features the GitHub Workflow powering the CI for this blog. Yours will probably end up sharing some similarities:

As you can see, by combining all our workflows into one, and carefully choosing which job to parallelize or run sequentially, we end up having better visibility on how our CI pipeline functions and the dependencies between each of its steps.

Sharing is caring

Now that all the CI steps are combined into one single workflow, the main challenge is to find out how we can make them as efficient as possible by sharing critical job outputs.

However, it's not very obvious from the get-go how one can share job outputs with other jobs on GitHub CI. There are two ways that I found to be "efficient":

1. leveraging caching with actions/cache
2. uploading/downloading artifacts using respectively actions/upload-artifact and actions/download-artifact

The first one is "great" but only for tasks that are repetitive and have outputs that do not change much over time like installing NPM dependencies.

1
jobs:
2
  # As its name stands for, this jobs will install the npm dependencies and cache them
3
  # unless they have been cached in a previous workflow run and remained unchanged.
4
  install-cache:
5
    runs-on: ubuntu-latest
6
    strategy:
7
      matrix:
8
        node-version: [12]
9
    steps:
10
      - name: Checkout Commit
11
        uses: actions/checkout@v2
12
      - name: Use Node.js ${{ matrix.node }}
13
        uses: actions/setup-node@v1
14
        with:
15
          node-version: ${{ matrix.node }}
16
      - name: Cache yarn dependencies
17
        uses: actions/cache@v2
18
        id: cache-dependencies
19
        with:
20
          path: node_modules
21
          key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
22
          restore-keys: |
23
            ${{ runner.os }}-yarn-
24
      - name: Install Dependencies
25
        # Check for `cache-hit` (`steps.cache-dependencies.cache-hit != 'true'`)
26
        # If there's a cache hit, we skip this step (the dependencies are already available)
27
        # If there's no cache hit, we run "yarn install"
28
        if: steps.cache-dependencies.outputs.cache-hit != 'true'
29
        run: |
30
          yarn install --force --non-interactive
31
  # This job requires some dependencies to be installed to run. Thus we'll restore
32
  # the dependencies that have been previously cached and use them here.
33
  type-check:
34
    runs-on: ubuntu-latest
35
    strategy:
36
      matrix:
37
        node: [12]
38
    needs: install-cache
39
    steps:
40
      - name: Checkout Commit
41
        uses: actions/checkout@v2
42
      - name: Use Node.js ${{ matrix.node }}
43
        uses: actions/setup-node@v1
44
        with:
45
          node-version: ${{ matrix.node }}
46
      # Here we use actions/cache again but this time only to restore the dependencies
47
      # At this stage of the workflow we're sure that the dependencies have been installed and cached
48
      # either on this same run, or on a previous CI run. Thus we can skip trying to run "yarn install".
49
      - name: Restore yarn dependencies
50
        uses: actions/cache@v2
51
        id: cache-dependencies
52
        with:
53
          path: node_modules
54
          key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
55
          restore-keys: |
56
            ${{ runner.os }}-yarn-
57
      - name: Check types
58
        run: |
59
          yarn type-check

Using artifacts, however, is what made a significant difference in the efficiency of my GitHub CI workflows.

For example, if you have 2 jobs that respectively run your e2e tests on firefox and chrome, you do not want to build your frontend twice as this could significantly increase the number of "billable minutes" for your CI run. The optimal solution here would consist of having a build job before your end-to-end tests running only once and then share the build artifacts with your chrome-end-to-end and firefox-end-to-end jobs.

To achieve this, we need to leverage actions/upload-artifact and actions/download-artifact:

• once the build is successful, use actions/upload-artifact to upload your build artifacts
• then use action/download-artifact on any jobs you want to pull that build output and use it

It's important to note that this trick only works because we're running every single CI step in the same workflow. You can only download artifacts in a workflow that were uploaded during the same workflow run.

1
# This example showcases how you can share the build output of a "build" job with two following jobs that need
2
# the output to run their respective tasks.
3

4
jobs:
5
  build:
6
    ...
7
    steps:
8
      ...
9
      - name: Run build
10
        run: |
11
          yarn build
12
      # This step in the build job will upload the build output generated by the previous step
13
      - name: Upload build artifacts
14
        uses: actions/upload-artifact@v2
15
        with:
16
          # Give a unique name to your artifacts so they can be easily retrieved
17
          name: build-output
18
          # This example is based of a Next.JS build output, thus the .next path.
19
          # The path might need to be changed based on your build settings or the framework your team is using.
20
          path: .next
21
  e2e-tests-chrome:
22
    ...
23
    needs: build
24
    steps:
25
      ...
26
      # Here we restore the build output generated in the previous job by downloading the artifact we uploaded
27
      - name: Download build artifacts
28
        uses: actions/download-artifact@v2
29
        with:
30
          name: build-output
31
          # Specify the path in which you wish to place your artiface.
32
          # Here I restore them in the .next folder since it's necessary to run the next start command later on
33
          path: .next
34
      - name: Run cypress
35
        uses: cypress-io/github-action@v2.10.1
36
        with:
37
          start: next start
38
          browser: chrome
39
  e2e-tests-firefox:
40
    ...
41
    needs: build
42
    steps:
43
      ...
44
      # Here we restore the same build output as we did in the e2e-tests-chrome job
45
      - name: Download build artifacts
46
        uses: actions/download-artifact@v2
47
        with:
48
          name: build-output
49
          path: .next
50
      - name: Run cypress
51
        uses: cypress-io/github-action@v2.10.1
52
        with:
53
          start: next start
54
          browser: firefox

1
jobs:
2
  build:
3
    ...
4
    steps:
5
      ...
6
      - name: Run build
7
        run: |
8
          yarn build
9
      - name: Upload build artifacts
10
        uses: actions/upload-artifact@v2
11
        with:
12
          name: build-output
13
          path: .next
14
          retention-days: 1

"You are terminated"

My last tip, and perhaps my favorite due to its simplicity is terminating duplicate workflow runs.

It happens to me very often: I'm done with a current branch and decide to push my code and open a PR, thus triggering a workflow run. Then a few seconds later noticed I forgot to run that one console.log or made a typo somewhere and need to push an extra change, thus triggering yet another workflow run.

By default, there's nothing that will stop the first workflow to run, it will continue until it's finished, thus wasting precious billing minutes that could have had a better use.

To prevent such a thing from happening, GitHub recently introduced the notion of workflow concurrency.

With the concurrency keyword you can create a concurrency group for your workflow (or a job). This will mark any workflow run from that same concurrency group as "pending" if any run is currently in progress. You can also decide to cancel any in-progress workflow of the same concurrency group whenever a new workflow is added to the queue.

1
name: CI
2

3
on:
4
  pull_request:
5
    branches:
6
      - main
7

8
concurrency:
9
  # Here the group is defined by the head_ref of the PR
10
  group: ${{ github.head_ref }}
11
  # Here we specify that we'll cancel any "in progress" workflow of the same group. Thus if we push, ammend a commit and push
12
  # again the previous workflow will be cancelled, thus saving us github action build minutes and avoid any conflicts
13
  cancel-in-progress: true
14

15
jobs:
16
  install-cache:
17
  ...

Doing this at the workflow level will ensure that any old or outdated workflows that are in progress will get canceled when we push a new change and triggering a new workflow thus saving your team's precious time and money.

Conclusion

So now that we went through all the tips to build the perfect Github CI workflow to power the CI needs of a frontend team, let's take a look at how they hold up against the guidelines we've established earlier:

Is it cost-saving? Yes! We made sure to share the output of expensive steps such as build and to cache repetitive steps that we would have needed to run throughout the workflow like installing our dependencies.

Is it efficient? More efficient than running every job in a separate workflow for sure! Not only we are parallelizing independent jobs like e2e-tests-firefox and e2e-tests-chrome, we're also making sure to cancel any duplicate workflows thanks to the use of concurrency groups.

Is it well architected? As we saw in the screenshot showcased earlier in this blog post, it's now easy to visualize all the steps and their dependencies. Combining every task into one workflow and architecting those jobs using the needs keyword made the whole CI pipeline way easier to understand.

Need a full example? Don't you worry, I got you covered 🙌! You can find my Github CI workflow featuring all the tips and examples of this article on the GitHub repository of this blog. It's fairly dense and long, thus why I did not directly integrate it here directly as it might have been distracting.

I hope some of the tips I introduced in this blog post will help you and your team perfect your own GitHub workflows and thus achieve a fast and reliable CI pipeline for your favorite frontend projects! Are there any other tips that you wish I had introduced in this article? Other GitHub CI secrets that empowered your team worth mentioning? As always, do not hesitate to reach out! I'd love learn more about what worked for you and test them out to further improve this article!