What Automated Accessibility Testing Catches

Automated accessibility testing belongs in every build, and it will never tell you that your site works. Teams that believe only the first half of that sentence trust their scanner too much; teams that believe only the second half skip a cheap, fast safety net.

The evidence supports both halves. Deque's own 2021 study of first-time audits found that automated axe testing caught 57% of issues, measured by volume. In 2017, the UK Government Digital Service ran ten automated tools against a page built to fail, and the best one found 41% of the known barriers. Those figures sound contradictory until you notice that they count different things, and the difference should shape how you build your pipeline.

Counting issues is not counting criteria

Deque counted issues. Some issue types, such as low-contrast text, missing form labels, and empty links, recur on page after page, and machines detect them well. They dominate any count by volume, so an automated tool's share of the total looks large. It's also a vendor measuring its own tool, which is worth keeping in mind. That doesn't make the number wrong, but it isn't neutral.

The GOV.UK test counted barriers, roughly one of each kind, on a single page. Variety is hard on automated tools, because each distinct barrier type needs its own rule, and many types can't be written as a rule at all.

Neither study measured the question auditors care about most: how many WCAG success criteria a machine can fully evaluate. Of the 55 A and AA criteria in WCAG 2.2, only a small number can be decided by a machine in most cases. Many more can be partly checked; a tool can confirm that a captions track exists, but not that the captions are accurate. A large group needs human judgment entirely, such as whether headings describe their sections (2.4.6 Headings and Labels) or whether the reading order preserves meaning (1.3.2 Meaningful Sequence).

There's a severity problem too. A page with four hundred contrast failures and one keyboard trap in the checkout produces a scanner report that is mostly about contrast. The trap is the one that stops a sale.

Tools also disagree with each other, because each vendor writes its own rules. W3C's Accessibility Conformance Testing (ACT) rules exist to fix that: they spell out, criterion by criterion, exactly what a test checks and what counts as a pass. When you compare tools, ask which ACT rules each one implements and how consistently. A tool that can't answer is asking you to trust its rule names.

So a clean automated run means one specific thing: no detectable issues, in the states you tested. Both qualifiers matter.

Machine-decidable versus judgment

Topic	A machine can decide	A person has to judge
Images	Does each image have a text alternative?	Does the alt text say what the image means in this context?
Forms	Does each input have an accessible name?	Does the label say what to enter, and does the error say how to fix it?
Contrast	Does text on a solid background meet the ratio?	Is text over a photo or gradient readable everywhere it sits?
ARIA	Are roles, states, and properties valid and allowed?	Does the widget behave as its role promises, by keyboard and screen reader?
Structure	Are headings and landmarks present?	Do headings describe the content, and is the order meaningful?
Keyboard	Does anything carry a positive `tabindex`?	Can you finish the journey by keyboard, with visible focus and no traps?
Media	Is there a captions track?	Are the captions accurate and in sync?
Language	Does the page declare a language?	Are passages in other languages marked?

The right-hand column is where the barriers that actually block people tend to live.

CI checks that fail for the right reasons

Build automation in three layers, each catching problems as early and as cheaply as possible.

Lint rules in the editor. Plugins such as eslint-plugin-jsx-a11y for React or eslint-plugin-vuejs-accessibility for Vue flag images without alt, click handlers on non-interactive elements, and invalid ARIA as you type. They are crude, but instant.

Component checks in Storybook. The accessibility addon runs axe-core against each story. Its value depends on your stories. Write one for every state, including open, error, disabled, loading, and empty, because a component's bugs live in its states.

End-to-end checks with Playwright or Cypress. Run axe-core against your key routes in a real browser. The step teams skip is the important one: axe sees only the DOM as it exists at that moment. Open the menu, submit the form with errors, open the dialog, and scan again.

import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';

const WCAG_AA = ['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa', 'wcag22aa'];

test('checkout has no detectable A/AA violations, including the error state', async ({ page }) => {
  await page.goto('/checkout');
  let results = await new AxeBuilder({ page }).withTags(WCAG_AA).analyze();
  expect(results.violations).toEqual([]);

  // Scan the state that matters most: a failed submission.
  await page.getByRole('button', { name: 'Place order' }).click();
  await expect(page.getByRole('alert')).toBeVisible();

  results = await new AxeBuilder({ page }).withTags(WCAG_AA).analyze();
  expect(results.violations).toEqual([]);
});

Look at the locators. getByRole('button', { name: 'Place order' }) finds the button through the accessibility tree, the same way a screen reader does. If a refactor turns that button into an unlabeled div, the test fails before axe even runs. Role-based locators turn every functional test into a small accessibility test, at no extra cost.

A few practices keep the pipeline honest:

Baseline legacy code, then fail on anything new. Record existing violations by rule and selector, and fail the build only when a new one appears. Perfection shouldn't block shipping, but nothing should get worse.
Disable rules with a ticket number, never silently. Every exception should name an issue and an owner.
Don't exclude third-party widgets from scans. Track their violations separately and send them to the vendor. Your users can't tell whose code broke the page.
Let the test start the app. Playwright's webServer option boots your site inside the CI job, so accessibility checks run beside your unit tests rather than in a separate pipeline someone forgets.

The manual layer, sized to your release cadence

Automation covers the left column of the table. People cover the right, and the trick is to size the manual work so it actually happens.

On every pull request that touches the interface: a five-minute keyboard pass. Tab through the changed component. Operate it with Enter, Space, and the arrow keys. Press Escape and check that focus returns somewhere sensible. Confirm focus is always visible and never slides under a sticky header.

On every release: a screen reader smoke test of two or three critical journeys. Use NVDA with Chrome or Firefox on Windows, and VoiceOver with Safari on macOS or iOS. Listen for three things: every control has a sensible name and role, state changes are announced ("expanded", "selected", "3 items in cart"), and errors are spoken when they appear. Add a quick reflow check at a viewport 320 CSS pixels wide.

Every quarter: a full audit of a sample of templates against all 55 criteria, by someone who didn't build them. Feed the findings back into the first two layers. A focus trap found by hand once can become a Playwright assertion forever.

Write all three into your definition of done. Manual checks that live only in good intentions don't survive a deadline.

Making the two layers one process

The goal isn't to choose between scanners and people. It's to let machines handle everything machine-decidable, on every commit, so human time goes to the judgment calls. If you're setting this up, our accessible development and training team builds CI gates and trains developers in the keyboard and screen reader passes. Baclion Watch then runs weekly scans and quarterly manual checks, so regressions surface before your users find them.

What accessibility automation catches, and what it can't