Accessibility

Comparing results: automated vs manual accessibility testing

12. May 2025 Evelin Laupa-Soonsein & Mari-Ell Mets

The accessibility requirements that will come into force from 28 June 2025 require service providers to assess the compliance of their digital environments with the European Union standard EN 301 549 V3.2.1. There are two options for evaluation: relatively resource-intensive manual testing versus attractive free automated testing tools. In this article, we'll explore the difference between automated and manual accessibility testing results.

Automated testing tools are useful for checking the accessibility of digital environments and provide a good starting point for identifying issues, but their capabilities are limited. High-quality software development plays an important role in accessibility, as our audits show that nearly 75% of accessibility issues stem from front-end code. Unfortunately, automated tests cannot detect all of these problems.

Automated tools are able to check whether some predetermined technical conditions are met (e.g., whether <input> is connected to <label>), but they can't see the big picture and can't understand the actual problems in the code (e.g. if the elements are grouped visually but not in the code).

What's more, automated tools can also bring out incorrect issues, highlighting problems that don't actually exist or, conversely, failing to identify real problems.

Accessibility is not just a technical requirement, but first and foremost a human-centred approach. Manual testing is necessary to understand how a web page or application actually performs for different users. Manual testing helps to uncover more complex issues and validate findings from automated tools.

Testing

Our aim was to find out the differences between the results of automated and manual accessibility testing. For testing, we chose the front page of an international public institution. We tested it with automated tools as well as manually. For automated testing, we used free browser extensions: WAVE Evaluation Tool, AXE DevTools, ARC Toolkit, and Siteimpove Accessibility Checker.

Manual testing for the desktop version was done using Chrome browser and JAWS screen reader, and for the mobile version on an iOS device with VoiceOver screen reader and an Android device with TalkBack screen reader.

Results

The summarised results in Figure 1 show that automated tools identified significantly fewer accessibility issues than manual testing.

However, the actual number of findings is even smaller, as most automated tools report the same error for each code element separately, which creates repetitions. In Table 1, where we display results by findings, we have grouped repetitive findings into one to make them more comparable to the results of manual testing.

Figure 1. Total errors found

For example, if the automatic tool pointed out four separate errors – one for each incorrect list element– then in Table 1 it is displayed as one error: "one list is read out by the screen reader as four separate lists".

Automated tools also highlight parsing errors, but these were not taken into account in manual testing, as in WCAG latest version 2.2, the "Parsing" criterion has been removed and does not need to be tested.

The results of different automated tools are also not fully comparable with each other, as each tool reports findings in a different way. Most automated tools also highlighted "warnings and best practices."

In these cases, it may be unclear whether it is a violation of an actual accessibility criterion or simply an opportunity to improve the user experience. If the manual test reported something as an issue, but automated tool put it under "warnings / best practices", we still marked it as “yes” in Table 1.

This shows that while automated tests bring up a variety of findings, they still require knowledge about accessibility and human revision to validate them. The following table shows the results of manual testing and whether automated tests found them or not.

Testing results

The findings are presented by criteria, along with a brief summary.

Contrast (Minimum)

The contrast ratio of the text and the background on the button is below the required limit of 4,5:1.

Bypass Blocks

There is no "Jump to main content” link that would allow user to go directly from the beginning of the page to the content of the page.

Focus Visible

There is no visible focus style on the button.
The focus style of the button exists but is practically invisible.

Focus Order

On mobile, the keyboard focus can move under the menu while the menu is open.
The link is focused twice with keyboard which makes navigation slower.

Headings and Labels

The date format is ambiguous and not clear.

Info and Relationships

The text in the table is read out as a link by screen readers.
The list is read out by a screen reader as four separate lists.
An element is read out by screen readers as a list, but it is not actually a list.
Not all navigation elements are inside the navigation for the screen reader.
The page lacks regions that help screen readers understand and navigate the page.
The code lacks the first-level heading <h1>. The headings are not hierarchically marked up in the code, which creates confusion for the screen reader user.

Keyboard

Users can't move onto the search results with the keyboard.

Link Purpose

External links are distinguishable visually, but not with a screen reader.

Label in Name

The visible text of the button and the text in the code do not match, which causes problems for both screen reader and voice command users.

Meaningful Sequence

The focus of the screen reader jumps to an unexpected place.
The same text is read out by a screen reader several times in a row.

Name, Role, Value

The screen reader doesn't say whether an item is in an open or closed state.
The mobile menu covers the close button, and the menu cannot be closed with voice commands.
The selected menu item is distinguishable visually, but not with a screen reader.
Users can't select search results with a screen reader or voice commands.
Navigation is incorrectly marked up as a context menu bar.

Non-Text Content

When an icon button is focused, a sequence of numbers is read out with a screen reader, which does not describe the function of the button.
When the logo is focused, the screen reader reads out text that is not actually on the logo.
When focusing an icon, the screen reader reads out text that doesn't describe the meaning of the icon.

On Focus

The search box closes unexpectedly.

Pointer Cancellation

The close button in the search box is activated when the mouse is pressed (not clicked).

Pointer Gestures

Certain actions can only be performed by dragging your finger along a certain trajectory.

Reflow

In the mobile view, things move on top of each other, and a horizontal scroll bar appears.

Status Messages

The screen reader doesn't notify user when search results are displayed.
When the search icon is clicked, a search box opens, but the screen reader doesn't notify the user that it happened.

Text Resize

When using browser magnification, things shift on top of each other.

Use of Color

Links are distinguished from the rest of the text only by a different color.

User preference

The text doesn't respond to the browser's font size settings.
The page foes not respond to the browser's dark mode settings.

The findings are presented by criteria, along with a brief summary.

Contrast (Minimum)

The contrast ratio of the text and the background on the button is below the required limit of 4,5:1.

Info and Relationships

The code lacks the first-level heading <h1>. The headings are not hierarchically marked up in the code, which creates confusion for the screen reader user.

Name, Role, Value

Partially identified problem: Navigation is incorrectly marked up as a context menu bar.

The findings are presented by criteria, along with a brief summary.

Contrast (Minimum)

The contrast ratio of the text and the background on the button is below the required limit of 4,5:1.

Bypass Blocks

There is no "Jump to main content” link that would allow user to go directly from the beginning of the page to the content of the page.

Info and Relationships

The list is read out by a screen reader as four separate lists.
An element is read out by screen readers as a list, but it is not actually a list.
Partially identified problem: The page lacks regions that help screen readers understand and navigate the page.
The code lacks the first-level heading <h1>. The headings are not hierarchically marked up in the code, which creates confusion for the screen reader user.

Name, Role, Value

Partially identified problem: Navigation is incorrectly marked up as a context menu bar.

The findings are presented by criteria, along with a brief summary.

Contrast (Minimum)

The contrast ratio of the text and the background on the button is below the required limit of 4,5:1.

Info and Relationships

The list is read out by a screen reader as four separate lists.
An element is read out by screen readers as a list, but it is not actually a list.
The page lacks regions that help screen readers understand and navigate the page.
The code lacks the first-level heading <h1>. The headings are not hierarchically marked up in the code, which creates confusion for the screen reader user.

Name, Role, Value

Partially identified problem: Navigation is incorrectly marked up as a context menu bar.

Use of Color

Links are distinguished from the rest of the text only by a different color.

The findings are presented by criteria, along with a brief summary.

Contrast (Minimum)

The contrast ratio of the text and the background on the button is below the required limit of 4,5:1.

Bypass Blocks

There is no "Jump to main content” link that would allow user to go directly from the beginning of the page to the content of the page.

Info and Relationships

The page lacks regions that help screen readers understand and navigate the page.
The code lacks the first-level heading <h1>. The headings are not hierarchically marked up in the code, which creates confusion for the screen reader user.

Altogether, manual testing identified 36 issues, WAVE identified 3 issues, ARC and AXE each identified 7 issues, and Siteimprove identified 4 issues.

This overview and Figure 2 clearly show that automated accessibility tests identified very few accessibility issues compared to manual testing.

In most cases, automated tools were unable to detect problems related to screen reader behavior: for example, situations where screen reader read out text that did not describe the element, or where important messages were not read out automatically.

The automated tools also did not detect issues with focus visibility and order, which primarily affects people who navigate using keyboard or switches. For example, it was not detected if the keyboard focus was invisible, moved illogically, or jumped to an unexpected location.

Figure 2: Unique errors.

Three out of four automated tests also reported findings that were not reported during manual testing. We have highlighted these findings together with reasons why they were not reported in manual testing.

The findings are presented by criteria, together with a brief summary and our commentary.

Contrast (Minimum)

The contrast ratio between the text and the text background should be at least 4,5:1, but is 1:1.

Clarification: Visually, the text is white and the background is dark blue, which actually has a contrast ratio of 14.07:1, which is well above the required limit. In the code, the background color is added with inline CSS, which the automatic tool "can't see".

Undefined

The text in the title attribute is the same as the visible text or alternate text.

Clarification: The error was not detected during manual testing, as in this context it did not affect the screen readers and did not cause confusion for sighted people either.

The findings are presented by criteria, together with a brief summary and our commentary.

Labels or Instructions

A non-interactive element has a title attribute.

Clarification: The error was not detected during manual testing, as in this context it did not affect the screen readers and did not cause confusion for sighted people either.

Undefined

The text is formatted using italics in CSS.

Clarification: The visual style of one line of text in the footer was not considered a mistake in this context, because italics were not used to emphasize anything.

The findings are presented by criteria, together with a brief summary and our commentary.

Target Size (WCAG tase AAA)

Interactive elements should be at least 44 x 44 pixels in size but are smaller.

Clarification: Manual testing was based on the European Union standard EN 301 549, which does not require compliance with WCAG AAA level rules. If the client requests, we can also test digital environments against WCAG AAA rules and recommended best practices.

Info and Relationships

The table has a footer element, but it's empty.

Clarification: This was not revealed during manual testing, as it does not affect the operation of the screen reader and does not disturb other users either.

This overview shows that some of the findings reported by automated tools are not actually errors that would significantly affect the user experience or accessibility. This confirms that blindly trusting the results of automated tests can lead to inefficient use of resources, shifting attention to issues that are not actually an obstacle for users.

Therefore, it is important that the results of automated tests are complemented by expert review and manual testing so that resources are spent on solving real accessibility issues.

To summarise

All of the automated accessibility testing tools used for testing were equally effectively and were able to identify an average of 15% of accessibility issues on the website being tested, missing the majority of important issues.

In our opinion, one of the reasons is that when using a website or application, the user performs various actions such as pressing buttons, entering text or enlarging the page. These actions trigger various changes on the page, such as opening modals, accordions, or notifications.

However, automated tools do not have the capacity to simulate all these activities or analyze the changes that occur due to them. However, if user behaviour and context are not taken into account, many accessibility problems are overlooked.

There may be automated tools on the market that can also detect slightly more accessibility errors than the tools mentioned in this article. Nevertheless, we believe that there is still a long way to go before automated tools can replace manual testing – this will not happen anytime soon.

Would you like to discuss your ideas?

Book a free 30-minute accessibility consultation

Kadi Rosenthal

Accessibility Specialist

We will help you identify your accessibility needs, establish a clear plan for training or an audit and provide a concise overview of your website’s accessibility.

Book a consultation

Comparing results: automated vs manual accessibility testing

Testing

Results

Testing results

Manual testing

WAVE Evaluation Tool

ARC Toolkit

AXE DevTools

Siteimpove Accessibility Checker

Wave Evaluation Tool

ARC Toolkit

Siteimprove Accessibility Checker

To summarise

Would you like to discuss your ideas?

More on this topic

A certified WAS specialist helps ensure digital accessibility and increase the quality of service

5 UX design principles that improve service usability and support business growth

Shared goal, honest cooperation – website renewal for the Estonian University of Life Sciences