Last month we introduced our free eBook Addressing Uncertainty in MultiSector Dynamics Research. Did you know that, to date, we have published 74 revisions? Without an automated release and deployment process, this would have been very tedious! But, with the help of GitHub Actions, every small edit, minor clarification, or major improvement can go live within moments of merging the code. Read on to learn how the eBook team leverages GitHub Actions for CI/CD (Continuous Integration / Continuous Delivery).
GitHub Workflow
A reliable CI/CD strategy depends on a robust code review process—continuously delivering bugs and typos will not impress anyone! There are many acceptable workflows; the best one to use will depend on team composition and codebase. In our case, a feature branching workflow suffices:
The feature branching workflow consists of a main code branch representing published assets. New features or bug fixes require authors to create a new branch from main, implement their changes, and then submit a pull request back into main. A pull request is a formal code review process that gives other authors a chance to provide feedback on the proposed changes. Once consensus has been reached, the feature branch is merged into the main branch, kicking off the CI/CD process.
Automation
While the code review process is designed to catch content and conceptual errors, subtle process and system based errors can often slip through the cracks. Thus the first step in the CI/CD process should be running a suite of automated tests that span a range of systems, behaviors, and any known pain points. The depth and breadth of these tests should be sufficient to ensure an adequate degree of publication readiness without being overly burdensome to maintain. The test suite can grow over time!
For the eBook, our tests simply ensure that the Python dependencies install correctly on Linux, Mac, and Windows containers, that the supporting code can be imported on these systems without error, and that the HTML and PDF versions of the publication generate successfully. If any tests fail, the publication process is cancelled and the authors are notified with details of the failure.
This test, release, and publication process is orchestrated by GitHub Actions. Read on to learn more!
GitHub Actions
GitHub Actions are available to any project hosted in GitHub—totally free for public repositories, and free to a limited extent for private repositories. An Action can be defined as a unit of work performed using a virtual machine in response to an event. In GitHub, Actions are defined using YAML files places into the .github/workflows directory of a repository. YAML (YAML Ain’t Markup Language) is a concise, human-readable syntax for conveying semi-structured data. The minimal content of a GitHub Action includes a name, one or more event triggers, and one or more jobs. A job consists of a name, one or more virtual machine targets, and one or more steps. For example, the eBook test Action looks like this:
.github/workflows/01_test.yml
name: Test
on:
push:
branches: [ main ]
jobs:
test:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
steps:
- uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@master
with:
python-version: 3.9
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Run tests
run: |
pip install pytest
pytest
There is a lot going on here, so let’s take it step by step!
name: Test
on:
push:
branches: [ main ]This snippet gives our Action a name, and specifies that it should trigger on updates to the main branch of the repository.
jobs:
test:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ ubuntu-latest, macos-latest, windows-latest ]This snippet defines a job within our “Test” Action named “test”, and then uses special syntax to declare that the job should be run on three different virtual machines: the latest Ubuntu Linux, macOS, and Windows containers. Running the tests on multiple operating systems helps catch bugs with system-specific dependencies.
steps:
- uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@master
with:
python-version: 3.9
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Run tests
run: |
pip install pytest
pytestThis snippet outlines the actual units of work within the job; each “-” separates a unique task. The uses syntax is special in that it allows one to leverage tasks written by others hosted in the GitHub Actions Marketplace. The actions/checkout@v2 task clones the repository onto the virtual machine, and the actions/setup-python@master task installs and configures the specified Python version. The final two steps use the run directive to invoke custom code, in this case installing Python dependencies and running the Python test suites.
Deployment
Once the tests successfully pass, it’s time to publish! Since the eBook is essentially a web app, GitHub Pages is the perfect deployment platform. Pages hosts the content of a branch as a website, and is free for public repositories.
If you followed along with the previous eBook post, you learned about the Python, Sphinx, and Restructured Text workflow for compiling the eBook content into a polished product. Let’s create a GitHub Action to compile the eBook and deploy it to GitHub Pages! Here’s the full YAML file:
.github/workflows/02_deploy.yml
name: Deploy
on:
push:
branches: [ main ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
with:
python-version: '3.9'
- name: Install latex dependencies
run: sudo apt-get update -y && sudo apt-get install -y texlive latexmk texlive-latex-recommended texlive-latex-extra texlive-fonts-recommended ghostscript
- name: Update pip and install python dependencies
working-directory: 'docs/'
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Build html and pdf ebook
working-directory: 'docs/'
run: |
make html latexpdf --keep-going LATEXMKOPTS="-interaction=nonstopmode" || true
make latexpdf --keep-going LATEXMKOPTS="-interaction=nonstopmode" || true
make latexpdf --keep-going LATEXMKOPTS="-interaction=nonstopmode" || true
continue-on-error: true
- name: Get current datetime in ISO format
id: date
run: echo "::set-output name=date::$(date -u +'%Y-%m-%d')"
- name: Create GitHub release
id: gh_release
uses: softprops/action-gh-release@v1
with:
files: docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.pdf
tag_name: ${{ steps.date.outputs.date }}v${{ github.run_number }}
- name: Commit the compiled files
run: |
cd docs/build/html
git init
git add -A
git config --local user.email "action@github.com"
git config --local user.name "GitHub Action"
git commit -m 'deploy' -a || true
- name: Push changes to gh-pages
uses: ad-m/github-push-action@master
with:
branch: gh-pages
directory: docs/build/html
force: true
github_token: ${{ secrets.GITHUB_TOKEN }}
A lot to unpack here! Let’s take it step by step. As before, we start by naming the Action, triggering it on updates to the main branch, declaring that it should run only on an ubuntu-latest virtual machine, checking out out the code, and setting up Python. Then we get into the new job steps:
- name: Install latex dependencies
run: sudo apt-get update -y && sudo apt-get install -y texlive latexmk texlive-latex-recommended texlive-latex-extra texlive-fonts-recommended ghostscriptThis step installs all the operating system dependencies needed to support the Latex syntax and compilation to PDF. There was some trial and error involved in getting this right, but once correct it should be pretty stable.
- name: Build html and pdf ebook
working-directory: 'docs/'
run: |
make html latexpdf --keep-going LATEXMKOPTS="-interaction=nonstopmode" || true
make latexpdf --keep-going LATEXMKOPTS="-interaction=nonstopmode" || true
make latexpdf --keep-going LATEXMKOPTS="-interaction=nonstopmode" || true
continue-on-error: trueThis step runs the Sphinx makefile to compile the HTML and PDF versions of the eBook. The verbosity and repetitiveness of these commands works around some unusual oddities of the Latex and PDF compilation. --keep-going LATEXMKOPTS="-interaction=nonstopmode" prevents the command from waiting for user input. || true and the repeated make latexpdf lines allow the PDF engine to fully resolve all the references in the restructured text files; otherwise the PDF file would be incomplete and garbled (this one stumped us for awhile!).
- name: Get current datetime in ISO format
id: date
run: echo "::set-output name=date::$(date -u +'%Y-%m-%d')"
- name: Create GitHub release
id: gh_release
uses: softprops/action-gh-release@v1
with:
files: docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.pdf
tag_name: ${{ steps.date.outputs.date }}v${{ github.run_number }}To make it easier to chronologically place our eBook releases, we wanted to include a date stamp in our version tags. The first step above assigns the date to a variable. The second step above creates and tags an official GitHub release (using the date and an auto-incrementing run number), and includes the PDF version of the eBook as an asset attached to the release.
- name: Commit the compiled files
run: |
cd docs/build/html
git init
git add -A
git config --local user.email "action@github.com"
git config --local user.name "GitHub Action"
git commit -m 'deploy' -a || true
- name: Push changes to gh-pages
uses: ad-m/github-push-action@master
with:
branch: gh-pages
directory: docs/build/html
force: true
github_token: ${{ secrets.GITHUB_TOKEN }}These two steps cause the GitHub Actions user to commit the compiled HTML files and force push them to the gh-pages branch of our repository, using a secret token. This is a common “hack” to enable publishing only the desired web assets and not the entire repository. Never force push to other shared code branches!
Action Status
Check the status of the CI/CD pipeline using the Actions tab of the GitHub repository. Successful Actions show a check mark, in progress Actions show a spinner, and failed Actions show an X. Clicking into a particular Action will show more details, log messages, and error traces if relevant.
Slack Integration
To take our workflow to the next level (and to avoid the need to read even more email 😅 ), we added the GitHub app to our eBook Slack channel. This adds a bot that can subscribe to GitHub repositories and report on activity; for instance: new issues, new pull requests, and new releases. We can then discuss and iterate inline in Slack, without having to jump to other apps or sites.
To add the GitHub bot to a channel, right click on the channel name, select “Open channel details”, and navigate to the “Integrations” tab. From here, you can choose “Add apps” and search for GitHub. Once added, type a bot command such as /github subscribe [repository name] to start receiving notifications in the channel. The bot can also be used to open and close issues!
Conclusion
Using the GitHub Actions workflow to automate the testing and publication of our eBook enabled our team to focus more on the content and quality of release rather than stumbling over the intricacies of the publication process. CI/CD has been a powerful tool in software communities, but can greatly benefit researchers and academics as well. We hope our learnings presented above will speed you along in your own workflows!
Water Programming: A Collaborative Research Blog 