Accessibility Testing Standards

Testing For Web Accessibility

All WCAG Success Criteria are written as testable criteria for objectively determining if content satisfies them. While some of the testing can be automated using software evaluation programs, others require human testers for part or all of the test. Although content may satisfy the WCAG Success Criteria, the content may not always be usable by people with a wide variety of disabilities.

Testing the WCAG Success Criteria involve a combination of automated testing and human evaluation. Therefore, usability testing is recommended, in addition to the required functional testing. Usability testing aims to determine how well people can use the content for its intended purpose. It is recommended that users with disabilities be included in test groups when performing usability testing. The accessibility of a web site is important for attracting customers, but the usability of the web site will determine the level of success in which customers engage your organization.

Web Accessibility Testing Strategies

Accessibility Testing Overview

Testing for Accessibility and Usability

The accessibility of a web site is important for attracting customers, but the usability of the web site will determine the level of success in which customers engage your organization. Accessibility and usability are closely related, and the inclusive design model should integrate their goals, approaches, and guidelines to ensure web site development reflects expectations. Combining accessibility standards and usability processes with real people ensures that web design is technically and functionally usable by people with disabilities.

Accessibility means enabling people with diverse abilities and methods of access to effectively use your interface. The aim of accessibility is to remove barriers for perceiving, understanding and navigating your interface and ensure that nobody is excluded. Usability, on the other hand, is more to do with designing a user-friendly approach. It is concerned with how people interact with a website’s interface. That is, is it easy to do the things people want to do? Can people quickly find what they want without help? Do people make lots of mistakes using a website? Do people enjoy using the website? In general, usable websites benefit everyone. Whilst usability implies accessibility, the contrary is not necessarily true.

Designing For Accessibility and Usability

If designers, developers, and project managers approach accessibility as a checklist of meeting accessibility standards, the focus is only on the technical aspects of accessibility, and the human interaction aspect is often lost. In general, designers have to consider two different groups of users when aiming to make their interfaces accessible for all. people with differing technical abilities and people with differing physical abilities. Understanding the technical abilities of your users entails considering what devices they are using, where they will be accessing your content and how technically literate they are. Not only do designers have to keep user technical limitations in mind, but they also have to consider users with disabilities including visual, auditory, physical, speech and neurological disabilities.

So what does good usability look like? You will generally find that good usability goes unnoticed. In other words, a highly usable website means that your users are getting around so easily that they don’t even know how good they have it. The Internet has the potential to broaden the lives and increase the independence of people with disabilities. For people who can be physically as well as socially isolated, access to the Internet can offer information about social interaction, cultural activities, employment opportunities, and consumer goods. But, as statistics demonstrate, not many people with disabilities are able to take advantage of these possibilities, in large part because their needs have not been addressed by the web design community.

Testing Methodologies

There are three distinct methodologies, both in scientific literature and in common practice, to evaluate accessibility of websites and applications.

  • Analysis by accessibility experts: Depending on the needs, they refer directly to certain accessibility guidelines or perform the assessment as a practitioner experienced in the verification of accessibility of information systems.
  • Analysis through accessibility validators: These are automatic tools that can verify the compliance of a site or a web application with respect to the criteria established by certain accessibility guidelines. Commonly their evaluation is based on the source code analysis of the web application under consideration.
  • Analysis through Evaluation Groups: These groups are typically made up of people with various types of disabilities, who have the task of accessing the website in question in order to test its functionalities. These tests can be performed both in informal environments (home, work or school), and in specifically designed environments (laboratory or focus group), both in free form (without specific tasks), and task-driven (with predefined tasks).

Testing Resources

Artificial Intelligent Accessibility

Using AI Controlled Browsers

A self-voicing application is an application that provides an aural interface without requiring a separate screen reader. Traditionally, talking web browsers had been specially created, but a more recent trend has embedded artificial intelligent (AI) capabilities into mainstream web browsers with cloud based hosting services. AI image recognition is complicated, and AI in general is still a rapidly evolving technology just in its infancy.

In 2004, Opera Software created a self-voicing and speech-recognition extension for the Windows version of their web browser, and in 2005 Charles L. Chen created FireVox, an extension that adds speech capabilities to the Mozilla Firefox web browser on Mac, Windows, or Linux. The voice browser presents information aurally, using pre-recorded audio file playback or text-to-speech synthesis software. A voice browser obtains information using speech recognition and a keypad. As speech recognition and web technologies have matured, voice applications are deployed commercially in many industries and voice browsers are supplanting traditional proprietary interactive voice response (IVR) systems.

Artificial Intelligent (AI) Compliance

However, it is important to note that accessibility AI Overlays, despite vendor claims, cannot yet replace screen reader assistive technology, as they do not have the suffisticated screen reader interface required by blind users. The AI Overlay widget browsers may offer assistance to people who have trouble typing, moving a mouse, gesturing or reading a screen. Many people with a physical disabilities have difficulty using computers and handheld devices. There are a number of reasons why you might consider using self-voicing browsers and AI Overlays on a site, butWCAG compliance should not be one.

Automated Accessibility Evaluation

Using Automated Tools For Functional Testing

Accessibility evaluation tools perform web page automated checks, but each is designed for a specific target audience based on a specific conformance standard. So, it is important to choose an appropriate evaluation tool for the intended reporting purpose. Consider the standards and guidelines used by the different accessibility evaluation tools. There are currently two standards and guidelines most commonly used: the Web Content Accessibility Guidelines and the Section 508 of the United States Rehabilitation Act.

Automated Validation Tools (AVT)

Some tools are available at a web site where they are able to evaluate the content of a page quickly and easily without downloading or installing an application. Other options include tools that are created as extensions to a browser, and some tools require installation on your hard drive or server. Some tools are simple and limited and evaluate just one page at a time. While other tools are very detail oriented that are able to examine large sites and check for a variety of errors. Many tools can only perform an evaluation and may be free to use, but many commercially available tools provide repair guidance in addition to the evaluation process.

Accessibility report styles differ widely depending on the target audience, such as web designers and developers or executives and management, but in general tend to be very lengthy and challenging to decipher. Although evaluation tools can identify accessibility issues, they cannot determine if a product complies with expected conformance levels and is usable by assistive technologies. That is, tools can identify images that are missing alt text, but they cannot determine how appropriate the text is in describing the image.
Nine things automated accessibility tests can’t test (new window)

User Accessibility Evaluation

Performing Usability Testing

No automated evaluation tool can tell you if your site is accessible, or even compliant. Human testing is always necessary because accessibility is about the human experience. Accessibility evaluation is often limited to assessing conformance to accessibility standards, according to legal requirements or a defined business best practice. However, when the focus is only on the technical aspects of accessibility, the human interaction aspect can be lost. Many of the web page accessibility checks require human judgement and must be evaluated manually using different techniques. In some cases evaluation tools are prone to producing false or misleading results, and should not be used to determine conformance levels of accessibility.

Web accessibility evaluation tools can not determine the accessibility of Web sites, but can only assist in doing so. Usability evaluation methods can assess usable accessibility to ensure that accessibility solutions are usable by people with disabilities. Inclusion of people with disabilities in a collaborative group can contribute to a better understanding of accessibility issues within the organization. Websites often have text that is difficult to read, controls that are difficult to click, or audio and videos that are difficult to hear. Fortunately in many cases the user can customize their computer to improve the web browsing experience. This includes customization options in the operating system, in software such as web browsers and media players, and sometimes for hardware devices such as any external loud speakers or microphones. Sometimes changing the user’s web browser, using additional software or hardware, or otherwise customizing the computer can further improve the accessibility user experience.

User Diversity

How do people who cannot move their arms use your website? What about people who cannot see well or at all? Or people who have difficulty hearing or understanding, or have other accessibility needs? Each individual is unique. People have diverse abilities, skills, tools, preferences, and expectations that can impact how they use the Web. There are many reasons why people may be experiencing varying degrees of auditory, cognitive, neurological, physical, speech, and visual disabilities. For instance, some may have disabilities from birth, an illness, disease, or accident, or they may develop impairments with age. Some may not consider themselves to have disabilities even if they do experience such functional limitations. As of 2012, 53% of American adults ages 65 and older use the internet or email.
W3C/WAI: Accessibility is about people (new window)

Disability is part of the human condition. Almost every one of us will be permanently or temporarily disabled at some point in life. We must do more to break the barriers which segregate people with disabilities, in many cases forcing them to the margins of society. – Dr. Margaret Chan, WHO Director-General
ICF was officially endorsed by all 191 World Health Organization (WHO) Member States in the Fifty-fourth World Health Assembly on 22 May 2001 (resolution WHA 54.21) as the international standard to describe and measure health and disability.
International Classification of Functioning, Disability and Health (new window)

Interoperability Accessibility Evaluation

Accessibility Interoperability Testing

Typically, user interfaces are represented as a hierarchical tree. For example, an application window would contain several objects, the first of which might be a menu bar. The menu bar would contain a number of menus, each of which contains a number of menu items, and so on. The Accessibility Application Program Interface (AAPI) describes the object relationships to other objects, to provide context. Other features such as information about text formatting, applicable headers for content sections or table cells and things such as event notifications, are also available in the AAPI.

Technologies work together to extract accessibility information from a web interface and appropriately present it to the user.

  1. A web developer writes some HTML code.
  2. That HTML code is retrieved and parsed by a user agent browser.
  3. The user agent browser uses the appropriate AAPI Mapping standard to create, and maintain, an Accessibility Tree Document Object Model (DOM) representation with only HTML information that is relevant to the assistive technology screen reader.
  4. The Accessibility Tree is exposed via the platform AAPI.
  5. The assistive technology screen reader then interacts with the AAPI to retrieve information about the web page.
  6. The assistive technology screen reader reads the content to the end user via a refreshable braille display and/or synthesized speech output.
  7. A Key To Web Accessibility: By Leonie Watson and Chaals McCathie Nevile, March 2015 (new window)

Application Programming Interface Models

There is a contract model between user agents (browsers) and assistive technologies (screen readers). Each web page element has a Name, a Role, a Value and a State Interpreted by assistive technology Used in combination with the Accessibility Application Programming Interface (AAPI). Assistive technologies can make standard method calls to the operating system to get information about the objects on the screen. In browsers, the platform Accessibility API is used both to make information about the browser itself available to assistive technologies and to expose information about the currently rendered content. Browsers typically support one or more of the available AAPI for the platform they are running on. Typically, browsers use the HTML DOM, along with further information derived from style sheets, to generate an accessibility tree hierarchy of the content it is displaying, and it passes that information to the platform AAPI. Information such as the role, name and state of each object in the content, as well as how it relates to other objects in the content, can then be queried by assistive technologies.

While each operating system and vendor AAPI is different, there are some concepts all of them share.

  1. The tree, which models the entire interface as a tree of objects, exposed to assistive technology via the Accessibility API.
  2. The events, which let assistive technology know that a part of the tree has changed somehow.
  3. The actions, which come from assistive technology to ask the interface to change.

AAPI Models

Windows, Firefox, Chrome, Opera and Yandex, support MSAA, IAccessible and IAccessible2. While Internet Explorer supports MSAA, IAccessible and UIAExpress. Safari and Chrome support NSAccessibility on OSX and UIAccessibility on iOS. ChromeVox uses DOM access exclusively to provide web content information to assistive technologies, and does not use the Chrome Accessibility API.

User Experience Evaluation

Evaluating The User Experience

Evaluating the accessibility of Web content for people with disabilities requires diverse kinds of expertise and perspectives. Comprehensive and effective evaluations require testers with an understanding of Web technologies, evaluation tools, barriers that people with disabilities experience, assistive technologies and approaches that people with disabilities use, and accessibility guidelines and techniques. The scope of the accessibility evaluation will depend upon the skill level of the website developers in understanding standards, techniques, and user interaction. A standards review in the User-Centered Design process assesses whether a product conforms to specified interface design standard.

A best practice inclusion model requires standards that are adopted throughout the entire organization, so as to achieve a good user experience through the interoperability of policies and techniques.

  1. The developer did not markup/code the web page properly, or
  2. The browser or media player isn’t handling the markup properly, or
  3. The user’s Assistive Technology (AT) isn’t handling the markup properly, or
  4. The user doesn’t know how to use the browser, media player, and user agent keyboard access features, or
  5. The page is poorly designed and it is a general usability problem for all users, including those without disabilities.
  6. W3C/WAI: Accessibility Evaluation Reporting Template (new window)

Types Of Adaptable Interfaces

Accessibility solutions benefit people with and without disabilities and are becoming increasingly available in standard computer hardware, mobile devices, operating systems, web browsers, and other tools. People with disabilities access and navigate the Web in different ways, depending on their individual needs and preferences. Sometimes people configure standard software and hardware according to their needs, and sometimes people use specialized software or hardware that help them perform certain tasks.

  • Assistive Technologies: Software or hardware that people with disabilities use to improve interaction with the web. These include screen readers that read aloud web pages for people who cannot read text, and screen magnifiers for people with some types of low vision. Voice recognition software and selection switches for people who cannot use a keyboard or mouse.
  • Adaptive Strategies: Techniques that people with disabilities use to improve interaction with the web, such as increasing text size, reducing mouse speed, or turning on captions. Adaptive strategies include techniques with standard software, mainstream browsers, or with assistive technologies. A common adaptive strategy among users with low vision, no vision, cognitive and learning disabilities is to resize browser windows. Changes in layout can help in a number of ways. For example, a mobile layout on desktop might benefit users who wish to see less clutter on the page, remove distracting banner adds, and minimize the number of key tabs to get to content. For others, line lengths of text in articles and blog posts might be too long to read. Reducing screen width therefore helps reduce line length. There are also a number of accessibility extensions that can be added to browsers. Extensions can provide a number of features to help navigation such as color filters, visible focus indicators, Caret browsing and spatial navigation.
  • Keyboard Only: By people with cognitive, physical, and visual disabilities.
  • Touch Screen Only: By people with cognitive and physical disabilities.
  • Mouse and Keyboard: With software that compensates for hand tremor.
  • Voice recognition: Speech input and other hands-free interaction.

Some Additional Resources

There is no better way to understand the importance of accessibility and inclusive design than learning from actual users with disabilities.

Screen Reader Technology

Web Based User Agents

Everyone that is browsing the web has a user agent. This is the software that acts as the bridge between the user and the internet. When your browser loads a web page, it identifies itself as an agent when it retrieves the content you have requested, and the browser sends a host of information about the device and network that you are on. This is a set of data that allows web developers to customize the web experience. That is, the browser must understand the Application Protocol Interface (API) building blocks and command structure of the web page, so as to display meaningful information on the user device and allow the user to interact with the page elements. There are many different browsers, each using a specific API structure and thus delivering a different user experience on different devices. However, for the most part, the browser API is designed for visual users and Mouse interaction, but some browsers have accommodated blind users by providing an accessibility interface structure for screen readers.

In order to enjoy access to websites and other digital media, many people with visual impairments rely on assistive technologies such as braille displays and screen readers. A screen reader is a software application that converts the text displayed on a computer screen into synthesized speech. Text-to-speech capabilities are a vital component of AI Assistants, such as Apple’s Siri and Amazon’s Alexa, which talk with users by converting their replies into digital speech. Screen readers are a specific use case of text-to-speech technology that improve accessibility for people with visual disabilities. These days the market offers many options for screen readers. Some are intended for use with different operating systems and applications; some are free and open source; others are commercial software. Screen readers are able to look for and process any kind of text that is displayed on the screen of a computer or mobile device, including website content, icon labels, documents, spreadsheets, file menus and more. The software also reads the title of web pages and the alternative text captions for any images on the page. Once it identifies this text, the screen reader speaks it out loud at the user’s request, pausing at punctuation such as periods, commas, and exclamation marks. The user can adjust settings such as the speed and volume of the speech. To learn more see List Of Screen Readers At Wikipedia (new window)

Using The JAWS Screen Reader

The Freedom Scientific JAWS screen reader (new window) can be a powerful tool in helping to evaluate the usability of a website. JAWS, a Microsoft Windows based screen reader assistive technology, presents Web pages using the JAWS Virtual Cursor. This allows users to read and navigate a Web page as though it were a text document. Users press the ARROW keys to read line by line, word by word, character by character, and so on. JAWS also provides Navigation Quick Keys, which are alpha-numeric keys that move the Virtual Cursor to features of the page such as links, headings, and form controls. In addition, users can press the TAB key to move between focusable elements on the page. Using the ARROW keys or Navigation Quick Keys to change the position of the Virtual Cursor does not change the actual focus point in the application. This means that even if JAWS reads the text of a given link on a Web page for example, that link doesn’t necessarily have the keyboard focus. Conversely, pressing the TAB or SHIFT+TAB key to navigate moves the focus point and the Virtual Cursor follows the focus. The JAWS Navigation quick keys make it faster and easier to move around on a Web page and anywhere else the Virtual Cursor is active. These commands are all assigned to keys on the main part of the keyboard. A web page well structured with the HTML tags, make it possible for JAWS user to quickly navigate the page and gain an understanding of the page content.

    JAWS Cursor Modes:

  1. The PC Cursor is linked to the keyboard functions of Windows and applications. This is the cursor that is used when typing information, moving through options in dialog boxes, and selecting options or icons. As you type information, the PC Cursor follows along with each key you press. pressing NUM PAD PLUS activates the PC Cursor.
  2. The JAWS Cursor is linked to mouse pointer functions in Windows and other applications. It is used to read information the PC Cursor cannot read, such as toolbar information. The mouse follows along with the JAWS Cursor when it is moved. press NUM PAD MINUS to activate the JAWS Cursor.
  3. The Touch Cursor enables you to navigate the Windows environment using a touch screen, found on many tablet computers running Windows 8 or later. To control JAWS from a touch screen, you will use one or more fingers to perform various gestures directly on the screen, such as tapping, flicking, and swiping. press SHIFT+NUM PAD PLUS to activate the Touch Cursor.
  4. The virtual PC Cursor mimics the functions of the PC Cursor, but is activated by default when entering an HTML document, such as a Web page on the Internet. The virtual PC Cursor speaks the number of elements on the page when it first opens. You can use the ARROW keys to navigate and read the document, or use Navigation Quick Keys to move to specific elements, such as paragraphs, tables, or headings. Press Insert+z keys to toggle the Virtual PC Cursor on and off, or Insert+z twice quickly for all opened Window applications.

Using The NVDA Screen Reader

NVDA – NonVisual Desktop Access (new window) is a free, open source software, Microsoft Windows based assistive technology, screen reader which enables blind and vision impaired people to use computers. It reads the text on the screen in a computerized voice. You can control what is read to you by moving the cursor to the relevant area of text with a mouse or the arrows on your keyboard. NVDA can be installed on your computer, or a portable version can be use on an USB thumb drive. NVDA uses a virtual buffer concept common to all Windows screen readers on the market. NVDA takes the HTML of the page and converts it into a flat document with semantic information in the order the HTML appears in the browser source. As you navigate, NVDA will speak semantic information such as link, heading level 1 through heading level 6, button, images, and much more.

Using The Voiceover Screen Reader

VoiceOver (new window) is a screen reader program that comes on Mac computers, iPhones, iPads, and iPod touches. VoiceOver is not a standalone screen reader assistive technology. It is deeply integrated into the iOS operating system and all the built-in apps on iOS devices, and as developers update their apps to take advantage of the accessibility interfaces provided by Apple, their apps can start working with VoiceOver right away. It can also access most OS X native applications and functions. VoiceOver gives auditory descriptions of each onscreen element, and provides helpful hints along the way (whether you prefer using gestures, a keyboard, or a braille display), and it supports more than 30 languages, including multiple voice options.

When it comes to testing, there is no substitute for using your app on an iOS device with VoiceOver turned on, or better still asking VoiceOver users to test your app. You can toggle VoiceOver on and off quickly by setting it to the triple-click setting in the Accessibility Settings. When VoiceOver is turned on you will need to use a different set of gestures. After you have turned VoiceOver on, you will notice that many familiar gestures have different effects. For example, a single tap causes VoiceOver to speak the selected item and a double tap activates the selected item. When an element is selected, VoiceOver draws a black rounded rectangle around it, which is called the VoiceOver cursor.

Using The Windows Narrator Screen Reader

The Microsoft Windows Narrator screen reader (new window) is a screen-reading app built into the Windows operating system. Microsoft Narrator was written by Professor Paul Blenkhorn in 2000 and has been included with every version of Windows since that time. While Microsoft recommends that the visually impaired purchase a full-function screen reader for general computer use, Narrator is a useful piece of built-in accessibility software. Narrator is included with every copy of Microsoft Windows, providing a measure of access to Windows without the need to install additional software as long as the computer in use includes a sound card and speakers or headphones.

Narrator can assist a blind person in installing a full-function screen reader, assisting the user until their screen reader of choice is up and running. Because Narrator is a lightweight screen reader that requires minimal “hooks” into the operating system, Narrator can provide speech when a full-function screen reader might be unable to do so, such as during the process of updating hardware drivers. The Ease of Access settings in Windows are easy to discover, learn and use. Settings are grouped by ability (vision, hearing, and interaction), with the most frequently used settings listed first. To get directly to Narrator settings, press Windows logo key + Ctrl + N.

Additional Screen Reader Resources

Accessibility Training Resources

Accessibility Training Resources