Accessibility

Comparing results: automated vs manual accessibility testing

12. May 2025 Evelin Laupa-Soonsein & Mari-Ell Mets

The accessibility requirements that will come into force from 28 June 2025 require service providers to assess the compliance of their digital environments with the European Union standard EN 301 549 V3.2.1. There are two options for evaluation: relatively resource-intensive manual testing versus attractive free automated testing tools. In this article, we'll explore the difference between automated and manual accessibility testing results.

Automated testing tools are useful for checking the accessibility of digital environments and provide a good starting point for identifying issues, but their capabilities are limited. High-quality software development plays an important role in accessibility, as our audits show that nearly 75% of accessibility issues stem from front-end code. Unfortunately, automated tests cannot detect all of these problems.

Automated tools are able to check whether some predetermined technical conditions are met (e.g., whether <input> is connected to <label>), but they can't see the big picture and can't understand the actual problems in the code (e.g. if the elements are grouped visually but not in the code).

What's more, automated tools can also bring out incorrect issues, highlighting problems that don't actually exist or, conversely, failing to identify real problems.

Accessibility is not just a technical requirement, but first and foremost a human-centred approach. Manual testing is necessary to understand how a web page or application actually performs for different users. Manual testing helps to uncover more complex issues and validate findings from automated tools.

Testing

Our aim was to find out the differences between the results of automated and manual accessibility testing. For testing, we chose the front page of an international public institution. We tested it with automated tools as well as manually. For automated testing, we used free browser extensions: WAVE Evaluation Tool, AXE DevTools, ARC Toolkit, and Siteimpove Accessibility Checker.

Manual testing for the desktop version was done using Chrome browser and JAWS screen reader, and for the mobile version on an iOS device with VoiceOver screen reader and an Android device with TalkBack screen reader.

Results

The summarised results in Figure 1 show that automated tools identified significantly fewer accessibility issues than manual testing.

However, the actual number of findings is even smaller, as most automated tools report the same error for each code element separately, which creates repetitions. In Table 1, where we display results by findings, we have grouped repetitive findings into one to make them more comparable to the results of manual testing.

Figure 1. Total errors found

For example, if the automatic tool pointed out four separate errors – one for each incorrect list element– then in Table 1 it is displayed as one error: "one list is read out by the screen reader as four separate lists".

Automated tools also highlight parsing errors, but these were not taken into account in manual testing, as in WCAG latest version 2.2, the "Parsing" criterion has been removed and does not need to be tested.

The results of different automated tools are also not fully comparable with each other, as each tool reports findings in a different way. Most automated tools also highlighted "warnings and best practices."

In these cases, it may be unclear whether it is a violation of an actual accessibility criterion or simply an opportunity to improve the user experience. If the manual test reported something as an issue, but automated tool put it under "warnings / best practices", we still marked it as “yes” in Table 1.

This shows that while automated tests bring up a variety of findings, they still require knowledge about accessibility and human revision to validate them. The following table shows the results of manual testing and whether automated tests found them or not.

Testing results

The findings are presented by criteria, along with a brief summary.

Contrast (Minimum)

The contrast ratio of the text and the background on the button is below the required limit of 4,5:1.

Bypass Blocks

There is no "Jump to main content” link that would allow user to go directly from the beginning of the page to the content of the page.

Focus Visible

There is no visible focus style on the button.
The focus style of the button exists but is practically invisible.

Focus Order

On mobile, the keyboard focus can move under the menu while the menu is open.
The link is focused twice with keyboard which makes navigation slower.

Headings and Labels

The date format is ambiguous and not clear.

Info and Relationships

The text in the table is read out as a link by screen readers.
The list is read out by a screen reader as four separate lists.
An element is read out by screen readers as a list, but it is not actually a list.
Not all navigation elements are inside the navigation for the screen reader.
The page lacks regions that help screen readers understand and navigate the page.
The code lacks the first-level heading <h1>. The headings are not hierarchically marked up in the code, which creates confusion for the screen reader user.

Keyboard

Users can't move onto the search results with the keyboard.

Link Purpose

External links are distinguishable visually, but not with a screen reader.

Label in Name

The visible text of the button and the text in the code do not match, which causes problems for both screen reader and voice command users.

Meaningful Sequence

The focus of the screen reader jumps to an unexpected place.
The same text is read out by a screen reader several times in a row.

Name, Role, Value

The screen reader doesn't say whether an item is in an open or closed state.
The mobile menu covers the close button, and the menu cannot be closed with voice commands.
The selected menu item is distinguishable visually, but not with a screen reader.
Users can't select search results with a screen reader or voice commands.
Navigation is incorrectly marked up as a context menu bar.

Non-Text Content

When an icon button is focused, a sequence of numbers is read out with a screen reader, which does not describe the function of the button.
When the logo is focused, the screen reader reads out text that is not actually on the logo.
When focusing an icon, the screen reader reads out text that doesn't describe the meaning of the icon.

On Focus

The search box closes unexpectedly.

Pointer Cancellation

The close button in the search box is activated when the mouse is pressed (not clicked).

Pointer Gestures

Certain actions can only be performed by dragging your finger along a certain trajectory.

Reflow

In the mobile view, things move on top of each other, and a horizontal scroll bar appears.

Status Messages

The screen reader doesn't notify user when search results are displayed.
When the search icon is clicked, a search box opens, but the screen reader doesn't notify the user that it happened.

Text Resize

When using browser magnification, things shift on top of each other.

Use of Color

Links are distinguished from the rest of the text only by a different color.

User preference

The text doesn't respond to the browser's font size settings.
The page foes not respond to the browser's dark mode settings.

The findings are presented by criteria, along with a brief summary.

Contrast (Minimum)

The contrast ratio of the text and the background on the button is below the required limit of 4,5:1.

Info and Relationships

The code lacks the first-level heading <h1>. The headings are not hierarchically marked up in the code, which creates confusion for the screen reader user.

Name, Role, Value

Partially identified problem: Navigation is incorrectly marked up as a context menu bar.

The findings are presented by criteria, along with a brief summary.

Contrast (Minimum)

The contrast ratio of the text and the background on the button is below the required limit of 4,5:1.

Bypass Blocks

There is no "Jump to main content” link that would allow user to go directly from the beginning of the page to the content of the page.

Info and Relationships

The list is read out by a screen reader as four separate lists.
An element is read out by screen readers as a list, but it is not actually a list.
Partially identified problem: The page lacks regions that help screen readers understand and navigate the page.
The code lacks the first-level heading <h1>. The headings are not hierarchically marked up in the code, which creates confusion for the screen reader user.

Name, Role, Value

Partially identified problem: Navigation is incorrectly marked up as a context menu bar.

The findings are presented by criteria, along with a brief summary.

Contrast (Minimum)

The contrast ratio of the text and the background on the button is below the required limit of 4,5:1.

Info and Relationships

The list is read out by a screen reader as four separate lists.
An element is read out by screen readers as a list, but it is not actually a list.
The page lacks regions that help screen readers understand and navigate the page.
The code lacks the first-level heading <h1>. The headings are not hierarchically marked up in the code, which creates confusion for the screen reader user.

Name, Role, Value

Partially identified problem: Navigation is incorrectly marked up as a context menu bar.

Use of Color

Links are distinguished from the rest of the text only by a different color.

The findings are presented by criteria, along with a brief summary.

Contrast (Minimum)

The contrast ratio of the text and the background on the button is below the required limit of 4,5:1.

Bypass Blocks

There is no "Jump to main content” link that would allow user to go directly from the beginning of the page to the content of the page.

Info and Relationships

The page lacks regions that help screen readers understand and navigate the page.
The code lacks the first-level heading <h1>. The headings are not hierarchically marked up in the code, which creates confusion for the screen reader user.

SOFTWARE DEVELOPMENT & ANALYTICS

STRATEGY & DESIGN

OTHER SERVICES

Comparing results: automated vs manual accessibility testing

Testing

Results

Testing results

Manual testing

WAVE Evaluation Tool

ARC Toolkit

AXE DevTools

Siteimpove Accessibility Checker

Wave Evaluation Tool

ARC Toolkit

Siteimprove Accessibility Checker

To summarise

Would you like to discuss your ideas?

More on this topic

5 UX design principles that improve service usability and support business growth

Shared goal, honest cooperation – website renewal for the Estonian University of Life Sciences

How a leading global sports technology company improved its user experience