Introduction and Background

Current ‘measures of success’

How well is your accessibility program meeting the needs of the intended target audience? Historically, two main measures have been taken, neither of which tells you how your accessibility program meets the needs of your intended target audience.

The first measure that has been taken is the ‘organizational maturity assessment’. This is designed to gather information on how well the accessibility program is operating. In order to find out whether accessibility has been addressed at any level, or if accessibility has been non-existent, a series of questions and data gathering exercises can give you data points such as “We are just starting out on x (Level 1); we have resources applied to y (Level 2); and we’re succeeding in z (Level 3).” The x, y, and z might be policy, website accessibility, and training. In practice, lots of measures of a program are combined to determine ‘how well we’re doing’. In the 2000’s US, the Department of Justice would infrequently gather such data by asking other government departments to assess their progress, and then publish these as survey results. In the following decade, the Office of Management and Budget (OMB) required departments to conduct maturity assessments and report back to OMB. Although this information is now collected twice a year, OMB does not publish the results. So, the US government supposedly knows how their accessibility programs are doing, but even though that sort of information is valuable to help manage and implement programs, it still doesn’t tell the department (or the public) how well those programs are meeting the needs of their target audience. The target audience is never addressed in any program maturity assessment metric. (For more on conducting this topic, see our article
How can I assess organizational maturity with respect to accessibility?)

The second measure, historically, has been the number of complaints from employees. It was very typical, when asked how a government accessibility program was going, to hear a response like “Terrific! We have had no complaints!”. Again with reference to the US Federal Government, a legal requirement exists to track and address complaints. However, the typical response shows that program mangers often mistakenly assume that a lack of complaints equates to satisfaction of end users. It is a mistaken assumption because the likelihood of complaints from the population as a whole is incredibly low, and the likelihood of complaints from people with disabilities is lower still. This makes logical sense when you consider that people with disabilities are the most under-represented marginalized group in society in terms of employment numbers: if you are lucky enough to secure employment, why make waves? So, again, the number of complaints you receive is no measure of how well your program is meeting the needs of your intended target audience. (For more in-depth discussion of complaints, see our companion guide
Beyond offering employees a complaint process: proactive measures to tackle accessibility issues)

The internal measures of program maturity and complaints don’t provide us with good measures of success. How do we know there are problems though? The easiest answer is to point to the plethora of published studies that have been made on how well government websites perform when tested for accessibility. The results are usually shocking in how bad they are.

Functional/Operable does not equal Usable

The field of usability (a.k.a. human factors and ergonomics) grew out of a need to address problems with technology that functioned well, but couldn’t be effectively used by people. Military aviation was a particularly fertile ground for research and development of usability techniques. Post WWII, the jet age meant more and more functions could be added to aircraft, and even though these new functional elements could be tested on the ground, pilots under stress when something went wrong would make mistakes in pressing the right button at the right time, or pulling a lever the wrong way to avoid a crash. As computing became more a part of everyday business, the same principles for making complex flight decks usable were applied to make software easier to use, reduce user mistakes, and providing means for users to correct mistakes.

Broadly speaking, engineers would design and build interfaces to be functional/operable, whereas usability professionals would evaluate and help refine those interfaces to be usable.

Functional/Operable by people with disabilities does not equal usable by people with disabilities

No one expects that military aircraft should be operated by people who are blind. However, these days we do expect that computer software and websites should be operated by people who are blind. We even demand this in legislation. For example, in the US, the Federal government requires that computer interfaces used by its employees, and the public, be operable by people with disabilities. The current functionality/operability requirements are given in the Web Content Accessibility Guidelines 2.0 (WCAG2.0).

It’s possible to legislate operability. It’s much harder (and practically non-existent) to legislate usability.

Even though usability isn’t legislated, we have compelling reasons to invest in testing the usability of our aircraft (so they’re less likely to crash), and to test the usability of our software (so users are less likely to send out unintended/erroneous messages to the public). But, more often than not, the required operability testing for accessibility isn’t accompanied by user testing with people with disabilities.

In our experience, we find that most government departments don’t have usability testing programs. However, many government departments do have accessibility testing programs (as ensuring operability by people with disabilities can be legislated). Even those government departments that do have usability testing programs do not regularly liaise with the accessibility testing programs. If you want to find out how well your accessibility program is meeting the needs of the intended target audience, you will need to include people with disabilities as testing participants in that program. Otherwise, create a new initiative to conduct usability testing by people with disabilities.

Do you know, for example, if your employees who cannot use a mouse are staying late at their desks to complete their work because it takes 5,000 more keystrokes than their colleagues who can use a mouse to complete their daily tasks? Do you know how often your blind employees have to lean over and bother their sighted colleagues to describe the images on their screen?  

Excuses, excuses

The opening quote from this guide listed Krug’s ‘top five plausible excuses’ (no time, no money, no expertise, no lab, no interpretation skills). In his popular and concise book, he guides people on how to overcome each of these excuses. Usability, as Krug points out, doesn’t have to be overly time-consuming, nor does it have to be expensive. The expertise to begin the process can be gained in a cheap paperback, or from many free tutorials on the web. You don’t need a lab. And when you get the results, you’ll be able to interpret them just fine:

“Usability testing has been around for a long time, and the basic idea is pretty simple: If you want to know whether your software or your Web site or you VCR remote control is easy enough to use, watch some people while they try to use it and note where they run into trouble. Then fix it, and test it again.”
—Steve Krug, 2000, Don’t Make Me Think. (p.143)

If you currently have an environment in which people can make excuses, they will. If you are struggling in a situation where staff are able to make excuses and you are experiencing frustration in getting things going, see our article on
How can I distribute the responsibility and accountability for accessibility?

What usability measurements can you gather?

In gauging success in terms of how well your accessibility program is meeting the needs of the intended target audience—there are several types of usability measurements you can gather:

• Informal usability testing. This can be as simple as sitting with someone at their desk and asking them to do tasks while telling you what they are finding and thinking as they go.
• Formal usability testing. This can use recordings, timing measures, and gathering targeted data. It can include comparisons between nondisabled and disabled participants on objective measures like speed, number of errors, task completion, and subjective measures such as user opinions.
• Focus groups. These are somewhat unfashionable these days, but gathering together groups of employees to hear what their opinions are on the ICT can be useful. Employee affinity groups may be useful for finding potential participants.
• Interviews. You are more likely to get uncensored opinions from direct interviews than you are from focus groups. However, that is only if the interviewees are assured of complete confidentiality and anonymity.
• Surveys. After you have collected data on a small scale with usability studies and interviews, you can verify the results and/or gauge the extent of problems by using those results to inform the design of surveys.

For more on user testing options, see our article
How do I ensure my products work for people with disabilities?

What other proactive measurements can you gather?

In addition to usability testing, there are also quality checks for operability that you can proactively gather. Say, for example, that you have a software application that passed in lab testing for accessibility functionality/operability. You could find users of that application in the field, and find out whether the software still passes the same operability tests.

The test tools and methods that you use may have to be modified for use in the field, but spot-checking ensures that what is in use is living up to what is specified at the time of purchase (or build).

For more on accessibility testing, see our article
What procedures should I use to test my ICT for accessibility?