It was a pleasure to work with you, as you and your team made this process a more pleasant experience for our team and the participants.
Karen Perry
Residents in the commonwealth of Virginia must take their car for a “safety inspection” each year. The test criteria is published by the state and it is fairly easy to determine in advance if your car is going to pass the inspection. If there are any issues, you fix them before the inspection. I make sure that all lights are functional, the tires all have proper tread, the windshield wipers are working, etc. For my car, there will be no “surprises.”
How many people reading this took the SAT, the MAT or the GRE without first doing any preparation or practice test taking? My guess would be a very small percentage.
The Office of the National Coordinator for Health Information Technology (ONC) requires that EHR vendors include summative usability test results in their Meaningful Use Stage 2 (and the 2015) certification submission.
See http://www.theusabilitypeople.com/ehr-usability-testing-onc-2015-certifi... for our take on Meaningful Use, MACRA, MIPS and the ONC 2015 Certification
Summative usability testing for safety-enhanced design involves recruiting targeted users as test participants (Doctors, Nurses, and other medical practitioners) and asking these users to complete a set of pre-defined tasks. An expert test facilitator conducts the testing via an established test protocol while the test sessions are recorded and later analyzed.
Just like a vehicle safety inspection, the SAT or GRE, summative evaluations are used to determine if the design meets specific measurable performance and/or satisfaction goals.
We have heard that statement quite a bit these days. Especially from EHR vendors that have contracted with us, some at the last minute, to conduct a summative evaluation of their EHR to meet the 2014/15 ONC Certification criteria.
The usability reports for ONC certified EHRs are published on the Certified Health IT Product List (CHPL). Why would EHR software vendors go through a formal test procedure without knowing or understanding the potential usability issues with their EHR system? Why would a company want to expose the usability issues in their system for potential review and/or exploitation by their competitors?
Here is an example of a published and somehow certified study https://www.icsalabs.com/sites/default/files/2014-EHRA415566-2014-0509-0... (See the Usability Report attachment within the PDF)
The “dirty laundry” issue rises from the fact that many of these vendors don’t have a user-centered design program and the required summative tests are often the first formal evaluation of their interface. We try to be gentle in our reports, but the data often speaks for itself.
Formative evaluations are a type of usability evaluation that helps to "form" the design for a product or service. Heuristic evaluations (aka Expert reviews), user interface inspections, thinking-aloud testing, pluralistic usability walkthrough, and cognitive walkthrough are some methods that can be used for formative evaluation. Formative evaluations are a cost effective way to find potential usability issues with an EHR. They involve evaluating the interface of the EHR during development, often iteratively, with the goal of detecting and eliminating usability problems.
Before running the summative usability evaluation for §170.314(g)(3) Safety-enhanced design we recommend that EHR vendors begin working with a usability expert to identify possible usability issues before they are exposed as “dirty laundry” in a formal report.
A Usability expert, that knows and understands the ONC established test protocol can examine the user interface and user interaction patterns of an EHR and provide a list of potential area of improvement. The reports would typically also contain actionable feedback that the vendor can take directly to their development team for implementation--before the summative test!
A few “quick” fixes to the interface of an EHR can go a long way to removing some of the “dirty laundry” that can provide fodder to your competitors.