Home

Building the perfect GitHub CI workflow for your frontend team

August 3, 2021

/ 15 min read /

0 Likes β€’

0 Replies β€’

0 Mentions

Last Updated August 3, 2021

You've probably noticed if you've been following me for a while that I'm a ✨ big fan ✨ of automation. I wrote about automated CI/CD a year ago and also talked a lot about the concepts surrounding this subject, but never really touched upon the tools I use for my CI jobs and how I use them. One such tool that has really worked for me, especially as a frontend engineer, is GitHub CI.

For over a year now, it's been my service of choice for automated CI/CD pipelines. The workflow syntax is easy to get started with, and has an extended set of features to help you craft your CI experience the way you and your team may want it.

However, even after a year, there still is a lot that I'm learning about this tool every day. When I got started with it, there was no set rule on how to properly architect your workflows, and there's a lot of tips, tricks I discovered along the way to build what I would qualify as "the perfect GitHub CI workflow" (at least to my eyes πŸ˜„). This article aims to gather those tips and good practices I've been using for personal projects and at work and show you how you can use all of those in a single workflow to power the CI/CD pipeline of your frontend team.

What would constitute a "good" GitHub CI workflow?

I'm going to throw my best "engineer response" at this question: it depends! Your team might have specific needs or objectives that would make some of my tips not as useful to you as they could be. However, for this article, we need some guidelines that I think would be universal when it comes to building efficient GitHub workflows, such as:

  • cost-saving: bring the "build minutes" down to the lowest possible value to not have a massive bill at the end of the month.
  • efficient: your team's time is precious, the workflow should be as fast as possible, but also fast to fail if something were to go wrong
  • well-architected: each step has a purpose, and might depend on other steps. This also means not running "useless steps".

Now that we've established those guidelines, let's take a look at one of the most important tips of this article.

One workflow to rule them all

Let's consider a typical set of tasks a frontend team would run on every PR:

  1. Lint
  2. Formatting
  3. Type checking
  4. Unit test
  5. Build
  6. End-to-end tests, maybe on different browsers

Running those in separate workflows might look like the most straightforward way to architect those tasks. However, if something as simple as the lint task fails, there's no way you can stop your expensive tasks like build or your end-to-end tests from running. And that, my friends, is not very efficient.

Workflows run in parallel, and there's no way for them to interact with one another. Thus, you can't cancel a workflow due to another workflow's failed state. You're stuck running all the workflows in every PR.

To address this, I chose to combine all my workflows into one. All the tasks that were independent workflows before became part of the same unique workflow, but this time, as jobs.

Excerpt of a Github CI workflow job.

1
# In this example, lint-format is a job among many others in a bigger GitHub workflow.
2
# This job has 3 steps: Checking out the code, running the lint command, and running the formatting command.
3
4
jobs:
5
lint-format:
6
runs-on: ubuntu-latest
7
strategy:
8
matrix:
9
node: [12]
10
steps:
11
- name: Checkout Commit
12
uses: actions/checkout@v2
13
- name: Use Node.js ${{ matrix.node }}
14
uses: actions/setup-node@v1
15
with:
16
node-version: ${{ matrix.node }}
17
- name: Run lint
18
run: |
19
yarn lint
20
- name: Run prettier
21
run: |
22
yarn format

The cool thing about jobs is that you can run them sequentially or parallel as you please! GitHub provides a handy keyword called needs that lets you set one or several jobs as dependencies, thus preventing a given job to start unless the dependent jobs have successfully run. This allows us to:

  • Fail the workflow fast. If a key job fails, the workflow is marked as failed on your PR as soon as possible
  • Avoid running useless expensive tasks on a "doomed to fail" workflow run

Example of jobs running in parallel and sequentially

1
# In this workflow excerpt, the type-check and unit-test jobs run in parallel whereas the
2
# build job "needs" these 2 jobs to be successful to be kicked off.
3
# Thus, if any of type-check or unit-test were to fail, the build job will not start and the
4
# whole workflow will be marked as "failed".
5
6
jobs:
7
type-check:
8
runs-on: ubuntu-latest
9
strategy:
10
matrix:
11
node: [12]
12
steps:
13
- name: Checkout Commit
14
uses: actions/checkout@v2
15
- name: Use Node.js ${{ matrix.node }}
16
uses: actions/setup-node@v1
17
with:
18
node-version: ${{ matrix.node }}
19
- name: Check types
20
run: |
21
yarn type-check
22
unit-test:
23
runs-on: ubuntu-latest
24
strategy:
25
matrix:
26
node: [12]
27
steps:
28
- name: Checkout Commit
29
uses: actions/checkout@v2
30
- name: Use Node.js ${{ matrix.node }}
31
uses: actions/setup-node@v1
32
with:
33
node-version: ${{ matrix.node }}
34
- name: Run test
35
run: |
36
yarn test
37
build:
38
runs-on: ubuntu-latest
39
strategy:
40
matrix:
41
node: [12]
42
needs: [type-check, unit-test]
43
steps:
44
- name: Checkout Commit
45
uses: actions/checkout@v2
46
- name: Use Node.js ${{ matrix.node }}
47
uses: actions/setup-node@v1
48
with:
49
node-version: ${{ matrix.node }}
50
- name: Run build
51
run: |
52
yarn build

You may be wondering: what job should be run in parallel and what job needs to be run sequentially? That will depend on the needs of your team.

On my end, I tend to parallelize unit testing, linting, and type-checking for example. These steps are generally fast and inexpensive to run thus I do not feel they need to depend on each other in most cases. However, I'd require a job such as build to only run if those three jobs above are successful, i.e. run it sequentially.

The screenshot below features the GitHub Workflow powering the CI for this blog. Yours will probably end up sharing some similarities:

Screenshot of a successful Github workflow run showcasing each jobs and their dependencies with one another.
Screenshot of a successful Github workflow run showcasing each jobs and their dependencies with one another.
Screenshot of a failed Github workflow run. Notice how failing the 'build' job did not triggered the two parallel e2e jobs as they were dependent on this job to be successful.
Screenshot of a failed Github workflow run. Notice how failing the 'build' job did not triggered the two parallel e2e jobs as they were dependent on this job to be successful.

As you can see, by combining all our workflows into one, and carefully choosing which job to parallelize or run sequentially, we end up having better visibility on how our CI pipeline functions and the dependencies between each of its steps.

Sharing is caring

Now that all the CI steps are combined into one single workflow, the main challenge is to find out how we can make them as efficient as possible by sharing critical job outputs.

However, it's not very obvious from the get-go how one can share job outputs with other jobs on GitHub CI. There are two ways that I found to be "efficient":

  1. leveraging caching with actions/cache
  2. uploading/downloading artifacts using respectively actions/upload-artifact and actions/download-artifact

The first one is "great" but only for tasks that are repetitive and have outputs that do not change much over time like installing NPM dependencies.

Example of sharing npm dependencies through multiple GitHub CI jobs

1
jobs:
2
# As its name stands for, this jobs will install the npm dependencies and cache them
3
# unless they have been cached in a previous workflow run and remained unchanged.
4
install-cache:
5
runs-on: ubuntu-latest
6
strategy:
7
matrix:
8
node-version: [12]
9
steps:
10
- name: Checkout Commit
11
uses: actions/checkout@v2
12
- name: Use Node.js ${{ matrix.node }}
13
uses: actions/setup-node@v1
14
with:
15
node-version: ${{ matrix.node }}
16
- name: Cache yarn dependencies
17
uses: actions/cache@v2
18
id: cache-dependencies
19
with:
20
path: node_modules
21
key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
22
restore-keys: |
23
${{ runner.os }}-yarn-
24
- name: Install Dependencies
25
# Check for `cache-hit` (`steps.cache-dependencies.cache-hit != 'true'`)
26
# If there's a cache hit, we skip this step (the dependencies are already available)
27
# If there's no cache hit, we run "yarn install"
28
if: steps.cache-dependencies.outputs.cache-hit != 'true'
29
run: |
30
yarn install --force --non-interactive
31
# This job requires some dependencies to be installed to run. Thus we'll restore
32
# the dependencies that have been previously cached and use them here.
33
type-check:
34
runs-on: ubuntu-latest
35
strategy:
36
matrix:
37
node: [12]
38
needs: install-cache
39
steps:
40
- name: Checkout Commit
41
uses: actions/checkout@v2
42
- name: Use Node.js ${{ matrix.node }}
43
uses: actions/setup-node@v1
44
with:
45
node-version: ${{ matrix.node }}
46
# Here we use actions/cache again but this time only to restore the dependencies
47
# At this stage of the workflow we're sure that the dependencies have been installed and cached
48
# either on this same run, or on a previous CI run. Thus we can skip trying to run "yarn install".
49
- name: Restore yarn dependencies
50
uses: actions/cache@v2
51
id: cache-dependencies
52
with:
53
path: node_modules
54
key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
55
restore-keys: |
56
${{ runner.os }}-yarn-
57
- name: Check types
58
run: |
59
yarn type-check

Using artifacts, however, is what made a significant difference in the efficiency of my GitHub CI workflows.

For example, if you have 2 jobs that respectively run your e2e tests on firefox and chrome, you do not want to build your frontend twice as this could significantly increase the number of "billable minutes" for your CI run. The optimal solution here would consist of having a build job before your end-to-end tests running only once and then share the build artifacts with your chrome-end-to-end and firefox-end-to-end jobs.

To achieve this, we need to leverage actions/upload-artifact and actions/download-artifact:

  • once the build is successful, use actions/upload-artifact to upload your build artifacts
  • then use action/download-artifact on any jobs you want to pull that build output and use it

It's important to note that this trick only works because we're running every single CI step in the same workflow. You can only download artifacts in a workflow that were uploaded during the same workflow run.

Uploading and downloading artifacts to share the build output

1
# This example showcases how you can share the build output of a "build" job with two following jobs that need
2
# the output to run their respective tasks.
3
4
jobs:
5
build:
6
...
7
steps:
8
...
9
- name: Run build
10
run: |
11
yarn build
12
# This step in the build job will upload the build output generated by the previous step
13
- name: Upload build artifacts
14
uses: actions/upload-artifact@v2
15
with:
16
# Give a unique name to your artifacts so they can be easily retrieved
17
name: build-output
18
# This example is based of a Next.JS build output, thus the .next path.
19
# The path might need to be changed based on your build settings or the framework your team is using.
20
path: .next
21
e2e-tests-chrome:
22
...
23
needs: build
24
steps:
25
...
26
# Here we restore the build output generated in the previous job by downloading the artifact we uploaded
27
- name: Download build artifacts
28
uses: actions/download-artifact@v2
29
with:
30
name: build-output
31
# Specify the path in which you wish to place your artiface.
32
# Here I restore them in the .next folder since it's necessary to run the next start command later on
33
path: .next
34
- name: Run cypress
35
uses: cypress-io/github-action@v2.10.1
36
with:
37
start: next start
38
browser: chrome
39
e2e-tests-firefox:
40
...
41
needs: build
42
steps:
43
...
44
# Here we restore the same build output as we did in the e2e-tests-chrome job
45
- name: Download build artifacts
46
uses: actions/download-artifact@v2
47
with:
48
name: build-output
49
path: .next
50
- name: Run cypress
51
uses: cypress-io/github-action@v2.10.1
52
with:
53
start: next start
54
browser: firefox

Setting the retention days option when uploading artifacts

1
jobs:
2
build:
3
...
4
steps:
5
...
6
- name: Run build
7
run: |
8
yarn build
9
- name: Upload build artifacts
10
uses: actions/upload-artifact@v2
11
with:
12
name: build-output
13
path: .next
14
retention-days: 1

"You are terminated"

My last tip, and perhaps my favorite due to its simplicity is terminating duplicate workflow runs.

It happens to me very often: I'm done with a current branch and decide to push my code and open a PR, thus triggering a workflow run. Then a few seconds later noticed I forgot to run that one console.log or made a typo somewhere and need to push an extra change, thus triggering yet another workflow run.

By default, there's nothing that will stop the first workflow to run, it will continue until it's finished, thus wasting precious billing minutes that could have had a better use.

To prevent such a thing from happening, GitHub recently introduced the notion of workflow concurrency.

With the concurrency keyword you can create a concurrency group for your workflow (or a job). This will mark any workflow run from that same concurrency group as "pending" if any run is currently in progress. You can also decide to cancel any in-progress workflow of the same concurrency group whenever a new workflow is added to the queue.

Example of GitHub workflow using concurrency groups

1
name: CI
2
3
on:
4
pull_request:
5
branches:
6
- main
7
8
concurrency:
9
# Here the group is defined by the head_ref of the PR
10
group: ${{ github.head_ref }}
11
# Here we specify that we'll cancel any "in progress" workflow of the same group. Thus if we push, ammend a commit and push
12
# again the previous workflow will be cancelled, thus saving us github action build minutes and avoid any conflicts
13
cancel-in-progress: true
14
15
jobs:
16
install-cache:
17
...

Doing this at the workflow level will ensure that any old or outdated workflows that are in progress will get canceled when we push a new change and triggering a new workflow thus saving your team's precious time and money.

Conclusion

So now that we went through all the tips to build the perfect Github CI workflow to power the CI needs of a frontend team, let's take a look at how they hold up against the guidelines we've established earlier:

Is it cost-saving? Yes! We made sure to share the output of expensive steps such as build and to cache repetitive steps that we would have needed to run throughout the workflow like installing our dependencies.

Is it efficient? More efficient than running every job in a separate workflow for sure! Not only we are parallelizing independent jobs like e2e-tests-firefox and e2e-tests-chrome, we're also making sure to cancel any duplicate workflows thanks to the use of concurrency groups.

Is it well architected? As we saw in the screenshot showcased earlier in this blog post, it's now easy to visualize all the steps and their dependencies. Combining every task into one workflow and architecting those jobs using the needs keyword made the whole CI pipeline way easier to understand.

Need a full example? Don't you worry, I got you covered πŸ™Œ! You can find my Github CI workflow featuring all the tips and examples of this article on the GitHub repository of this blog. It's fairly dense and long, thus why I did not directly integrate it here directly as it might have been distracting.

I hope some of the tips I introduced in this blog post will help you and your team perfect your own GitHub workflows and thus achieve a fast and reliable CI pipeline for your favorite frontend projects! Are there any other tips that you wish I had introduced in this article? Other GitHub CI secrets that empowered your team worth mentioning? As always, do not hesitate to reach out! I'd love learn more about what worked for you and test them out to further improve this article!

Fetching Replies...

Do you have any questions, comments or simply wish to contact me privately? Don’t hesitate to shoot me a DM on Twitter.


Have a wonderful day.
Maxime

Subscribe to my newsletter

Get email from me about my ideas, frontend development resources and tips as well as exclusive previews of upcoming articles.



Β© 2021 Maxime Heckel β€”β€” SF/NY