This article covers adding accessibility tests to your site using Pa11y (pa11y-ci with axe) and Cypress (with cypress-axe) in GitLab CI/CD. I use a Jekyll website as an example, but any website technology that runs in CI/CD can leverage this setup.
Prep your website
In addition to getting your website to run in CI/CD, I recommend enabling an XML sitemap feature. A sitemap allows the accessibility tests to parse all URLs to find accessibility issues across the site. I recommend the jekyll-sitemap plugin for Jekyll sites.
Collecting a list of all major URLs is a good alternate step if a sitemap is not possible. The URLs should cover all potential layouts of the website, such as pages with the highest traffic or the most landings. This approach won't catch all accessibility issues, especially content level concerns, but it will test the layout and main pages.
This scenario requires the npm or yarn package managers. I used npm for this article. If your project doesn't have npm initialized, run the npm init command to create the package.json file.
Begin with Pa11y
Pa11y is a free and open source software that tests websites for accessibility issues. Pa11y-ci is the command line utility geared towards continuous integration (CI). Install pa11y-ci as a development dependency with npm:
$ npm i --save-dev pa11y-ciAfter you complete the installation, edit the package.json and add the following commands to the scripts section:
"start-detached": "bundle exec jekyll serve --detach",
"pa11y-ci:home": "pa11y-ci http://127.0.0.1:4000",
"pa11y-ci:sitemap": "pa11y-ci --sitemap http://127.0.0.1:4000/sitemap.xml --sitemap-find https://accessibility.civicactions.com --sitemap-replace http://127.0.0.1:4000 --sitemap-exclude \"/*.pdf\""- start-detached: Starts the web server that will run Jekyll for testing.
- pa11y-ci:home: Runs pa11y-ci tests on the home page. Useful for troubleshooting.
- pa11y-ci:sitemap: Runs pa11y-ci tests using the sitemap and excludes PDFs. The sitemap will refer to the live site URLs, so replace those with local URLs for testing in the CI pipeline.
Add a JSON file named .pa11yci that configures pa11y-ci with various options. Here is a sample file:
{
"defaults": {
"concurrency": 1,
"standard": "WCAG2AA",
"runners": ["axe", "htmlcs"],
"ignore": [
"color-contrast",
"frame-tested"
],
"chromeLaunchConfig": {
"args": ["--disable-dev-shm-usage", "--no-sandbox", "--disable-gpu"]
},
"reporters": [
"cli",
["./pa11y-reporter-junit.js", { "fileName": "./pa11y-report-junit.xml" }]
]
}
}- concurrency: I reduced this set to 1 because increasing it caused errors (https://github.com/pa11y/pa11y-ci/issues/168 covers the bug, which might be fixed).
- standard: I have stuck with the default WCAG2AA as the goal for this site.
- runners: I ran axe (run tests using axe-core) and htmlcs (default, run tests using HTML CodeSniffer) to cover all potential accessibility issues.
- ignore: With newer versions of axe and some of the changes to the site, I ran into color contrast false positives. I also have an embedded iframe that requires separate testing that axe will report about. I have follow-up issues to examine the axe results, so I am ignoring those criteria for now.
- chromeLaunchConfig: pa11y-ci uses Chrome, and I found that the GitLab CI pipeline requires that the Chrome browser runs properly in the pipeline.
- reports: I use the default command line reporter, but I also added a custom reporter that reports on the pa11y-ci results in a junit format. This came in handy for reporting the results in the GitLab CI pipeline.
That's it. Run this setup locally using npm, and you will see the following output (truncated for brevity):
dmundra in ~/workspace/accessibility/accessibility on branch main > npm run start-detached
> start-detached
> bundle exec jekyll serve --detach
Configuration file: /Users/dmundra/workspace/accessibility/accessibility/_config.yml
Source: /Users/dmundra/workspace/accessibility/accessibility
Destination: /Users/dmundra/workspace/accessibility/accessibility/_site
Incremental build: disabled. Enable with --incremental
Generating...
done in 8.217 seconds.
Auto-regeneration: disabled when running server detached.
Server address: http://127.0.0.1:4000
Server detached with pid '14850'. Run `pkill -f jekyll' or `kill -9 14850' to stop the server.
dmundra in ~/workspace/accessibility/accessibility on branch main > npm run pa11y-ci:sitemap
> pa11y-ci:sitemap
> pa11y-ci --sitemap http://localhost:4000/sitemap.xml --sitemap-exclude "/*.pdf"
Running Pa11y on 110 URLs:
> http://localhost:4000/guide/glossary - 0 errors
> http://localhost:4000/guide/introduction - 0 errors
> http://localhost:4000/guide/history - 0 errors
> http://localhost:4000/guide/design - 0 errors
...
✔ 110/110 URLs passedThe site passes the tests. Here is an example job running in GitLab. The pa11y configuration continues to test all site pages for accessibility issues and report on them.
What does an error look like? Here is an example:
> http://localhost:4000/guide/introduction - 1 errors
Errors in http://localhost:4000/guide/introduction:
• <ul> and <ol> must only directly contain <li>, <script> or <template>
elements (https://dequeuniversity.com/rules/axe/3.5/list?application=axeAPI)
(#main-content > div:nth-child(2) > div > div > div > div:nth-child(1) > nav
> ul)
<ul class="usa-sidenav">
You get a count of the number of errors at a given URL and then details on the accessibility issue. It also displays a link to the criteria being violated and the location in the HTML of the issue.
Try Cypress
Cypress is a JavaScript testing framework and is very helpful in writing tests that interact with the site and assert that features work as expected. The setup for Cypress is very similar to pa11y-ci in terms of installation with npm.
$ npm i --save-dev cypress cypress-axe cypress-real-eventsAfter the installation is complete, edit the package.json and add the following commands to the scripts section:
"cypress-tests": "cypress run --browser chrome --headless"- cypress-tests: Run the Cypress tests with a headless Chrome browser.
When launching Cypress for the first time, you get a wizard to create the configuration file. Here is a sample file:
const { defineConfig } = require('cypress')
module.exports = defineConfig({
video: true,
videosFolder: 'cypress/results',
reporter: 'junit',
reporterOptions: {
mochaFile: 'cypress/results/junit.[hash].xml',
toConsole: false,
},
screenshotsFolder: 'cypress/results/screenshots',
e2e: {
// We've imported your old cypress plugins here.
// You may want to clean this up later by importing these.
setupNodeEvents(on, config) {
return require('./cypress/plugins/index.js')(on, config)
},
baseUrl: 'http://localhost:4000',
},
})- video: Take videos of the tests, which are helpful for troubleshooting.
- videosFolder: Defines the video storage folder.
- reporter: Set to junit to make it easier to report the results in the GitLab CI pipeline.
- reporterOptions: Includes a path for the junit files and the keyword [hash] to preserve unique reports for each test file (otherwise, the file is overwritten). Skip the console output for the reporter and use the default output.
- screenshotsFolder: Defines the screenshot storage folder (useful for troubleshooting).
- e2e: References the local URL of the site and the plugins.
After setting up Cypress and writing some tests (see below for examples), run the tests locally using npm. You will see the following output (truncated for brevity):
dmundra in ~/workspace/accessibility/accessibility on branch main > npm run cypress-tests
> cypress-tests
> cypress run --browser chrome --headless
====================================================================================================
(Run Starting)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Cypress: 11.2.0 │
│ Browser: Chrome 109 (headless) │
│ Node Version: v18.10.0 (/usr/local/Cellar/node/18.10.0/bin/node) │
│ Specs: 5 found (accordion.cy.js, home.cy.js, images.cy.js, menu.cy.js, search.cy.js) │
│ Searched: cypress/e2e/**/*.cy.{js,jsx,ts,tsx} │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────
Running: search.cy.js (5 of 5)
(Results)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Tests: 1 │
│ Passing: 1 │
│ Failing: 0 │
│ Pending: 0 │
│ Skipped: 0 │
│ Screenshots: 0 │
│ Video: true │
│ Duration: 2 seconds │
│ Spec Ran: search.cy.js │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
(Video)
- Started processing: Compressing to 32 CRF
- Finished processing: /Users/dmundra/workspace/accessibility/accessibility/cypres (0 seconds)
s/results/search.cy.js.mp4
...
(Run Finished)
Spec Tests Passing Failing Pending Skipped
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ ✔ search.cy.js 00:02 1 1 - - - │
...While Pa11y-ci can test interactivity, Cypress and its plugins can do much more. For a Jekyll site, I found that pa11y-ci did not catch any accessibility issues in mobile drop-down menu, dynamic search, or accordion features. I ran Cypress tests to interact with the elements (like performing searches, clicking menus, or clicking the accordion) and then checked if the results still passed accessibility tests. Here is the search example:
describe('Search', () => {
it('should be accessible', () => {
cy.visit('/search')
cy.get('#search-input').type('accessibility')
cy.checkA11yWithMultipleViewPorts()
})
})Here is a quick video of the running test.
The above test visits the search page, types the word "accessibility" in the search field, and then checks the results for accessibility issues. I use the cypress-axe plugin to check accessibility issues with axe core, just like pa11y-ci. I have wrapped the cypress-axe functions in a function to test multiple window sizes and report on the issues in a table format.
I also use the plugin cypress-real-events to interact with the site with a keyboard to check that the features are keyboard-accessible. Keyboard accessibility is a critical consideration (Operable principle of WCAG), and having an automated test that can confirm the features are keyboard accessible means that, maybe, there is one less test to run manually. You can see an example of the test here.
Here is an example of what an error looks like:
Running: a11y/anonymous_a11y.cy.js (1 of 36)
cy.log(): Accessibility scanning: Home (/)
cy.log(): 4 accessibility violations were detected
┌─────────┬────────────────────────┬────────────┬────────────────────────────────────────────────────────────────────────────────────┬───────┐
│ (index) │ id │ impact │ description │ nodes │
├─────────┼────────────────────────┼────────────┼────────────────────────────────────────────────────────────────────────────────────┼───────┤
│ 0 │ 'image-alt' │ 'critical' │ 'Ensures <img> elements have alternate text or a role of none or presentation' │ 4 │
│ 1 │ 'link-name' │ 'serious' │ 'Ensures links have discernible text' │ 3 │
│ 2 │ 'page-has-heading-one' │ 'moderate' │ 'Ensure that the page, or at least one of its frames contains a level-one heading' │ 1 │
│ 3 │ 'region' │ 'moderate' │ 'Ensures all page content is contained by landmarks' │ 2 │
└─────────┴────────────────────────┴────────────┴────────────────────────────────────────────────────────────────────────────────────┴───────┘Cypress logs provide a count of the number of errors at a given URL and then details on what the accessibility issue is, the impact, and its location.
You can find additional details and examples in the Cypress folder.
Use the GitLab CI/CD
Now that you have pa11y-ci and Cypress running locally, see how to run automated accessibility tests in GitLab using CI/CD features. The GitLab repository is available here. Here is the .gitlab-ci.yml file setup:
stages:
- test
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- node_modules/
- .npm/
- vendor/ruby
default:
image: ruby:2
before_script:
- apt-get update
- apt-get install -yq gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libnss3 lsb-release xdg-utils wget libgbm1 xvfb
- apt-get install -y nodejs npm
- bundle install -j $(nproc) --path vendor/ruby
- npm ci --cache .npm --prefer-offline
- npm run start-detached
pa11y-tests:
stage: test
script:
- npm run pa11y-ci:sitemap
artifacts:
when: always
reports:
junit:
- pa11y-report-junit.xml
expire_in: 1 day
cypress-tests:
stage: test
script:
# Install chrome browser manually, taken from https://github.com/cypress-io/cypress-docker-images/blob/master/browsers/node16.14.2-slim-chrome100-ff99-edge/Dockerfile#L48
- wget --no-verbose -O /usr/src/google-chrome-stable_current_amd64.deb "http://dl.google.com/linux/chrome/deb/pool/main/g/google-chrome-stable/google-chrome-stable_105.0.5195.125-1_amd64.deb"
- dpkg -i /usr/src/google-chrome-stable_current_amd64.deb
- rm -f /usr/src/google-chrome-stable_current_amd64.deb
- npm run cypress-tests
artifacts:
when: always
paths:
- cypress/results/
reports:
junit:
- cypress/results/*.xml
expire_in: 1 dayThe file currently defines only one stage test and caches folders that store dependencies when installed. Then:
- Steps used by all stages:
- Use the Ruby version 2 image because it is compatible with the current Jekyll installation.
- I install many dependencies based on the documentation at running puppeteer on GitLab. Install
nodeandnpmto install site dependencies. - Install the Jekyll Ruby dependencies.
- Install the Cypress and pa11y-ci dependencies via
npm. - Start the web server.
- Run the pa11y-ci to test the site and capture the output to a file.*
- Install the Chrome browser dependencies in cypress-tests using steps provided by Cypress in their Docker image configurations. Run the Cypress tests and capture the output to files.*
* Capture the output of Cypress and pa11y-ci tests as junit XML files.
Here is an example screenshot of the GitLab pipeline (taken from https://gitlab.com/civicactions/accessibility/-/pipelines/744894072):
(Daniel Mundra, CC BY-SA 4.0)
Here is an example of the test results in the same pipeline:
(Daniel Mundra, CC BY-SA 4.0)
GitLab CI/CD automatically take junit XML files and outputs them in a clear format. Cypress tests provide the junit XML output as part of their features (see above). I created a custom reporter for pa11y-ci to output the format in junit (credit to macieklewkowicz/pa11y-reporter-junit).
Note: GitLab version 12.8+ supports Pa11y accessibility tests (see https://docs.gitlab.com/ee/ci/testing/accessibility_testing.html for details). The above setup allows for customization of the pa11y-ci and also targeting of local URLs. I recommend using their options for live sites.
Wrap up
Using the above steps, you can provide accessibility testing for your site locally and in CI. This process helps you track and fix accessibility issues on your site and in the content. An important caveat about automated testing is that it only catches 57% of issues, so you definitely want to include manual testing with your accessibility testing.
Further reading and examples
- Automated accessibility testing: Leveraging GitHub Actions and pa11y-ci with axe.
- Automated accessibility testing with Travis CI.
- How to Automate Web Accessibility Testing.
- Setting up Cypress with axe for accessibility - Tim Deschryver.
- GitLab CI | Cypress Documentation.
Thank you to Marissa Fox and Mike Gifford for your support, thoughts, and feedback.
1 Comment