We provide the summative assessment for EHR Safety-enhanced design (§170.314.g.3) (aka Usability) Testing for Meaningful Use and ONC 2015 Certification
For the 2015 Edition Certification (aka Meaningful Use Stage 3), The Office of the National Coordinator for Health Information Technology has made several enhancements to the usability or Safety-enhanced Design testing portion of the certification criteria. Note that the ONC 2015 Certification for Safety-enhanced Design (§170.314.g.3) final rule is now active policy!
Summative Usability testing for safety enhanced design (§170.314.g.3) involves recruiting targeted users as test participants (Doctors, Nurses, and other medical practitioners) and asking these users to complete a set of pre-defined tasks. An expert test facilitator conducts the testing via an established test protocol while the test sessions are recorded and later analyzed.
Usability testing cannot happen overnight, and it is important that you reach out early so that the required testing can be completed in time for your appointment with an ONC-ACB.
We recommend that you allow at least 4-5 weeks in your submission timeline for the preparation, conducting and reporting of a summative usability test for the ONC 2015 Safety-enhanced Design Certification.
In a public comment on the 2014 certification process, Drummond (one of the ONC ATCB's) stated that
"...the usability test effort is not a small undertaking for EHR vendors. In fact, it (is) essentially one of the most difficult criteria to complete."
User-Centered Design: Helping users become Effective, Efficient, and Satisfied
ONC states that a User-centered design processes must be applied to each capability an EHR technology includes that is specified in the following certification criteria: §170.314(a)(1); § 170.314(a)(2); § 170.314(a)(6); § 170.314(a)(7); § 170.314(a)(8); § 170.314(a)(16); §170.314(b)(3); and § 170.314(b)(4)
We recommend that EHR vendors follow the NISTIR 7741 guidelines and observe the ISO 9241-11 standard. ISO-9241 Part 11: (1998) pertains to the extent to which a product can be used by specified users to achieve specified goals with:
Effectiveness (Task completion by users),
Efficiency (Task on time) and
Satisfaction (responded by user in term of experience) in a specified context of use (users, tasks, equipment & environments).
Summative usability testing is an industry standard usability methodology (see http://bit.ly/13WYqTU ) During the testing a trained user experience professional works with the EHR vendor to create a series of representative tasks and then measures the amount of time it takes to complete the task (time on task), how many and what type of errors occur (error rates), and satisfaction with the task interaction (user satisfaction). The results —as measured by things like task times, assisted and un-assisted completion rates and standardized satisfaction scores --can be benchmarked against current industry standards. The ONC has recommended that at least 15 participants be evaluated in a summative evaluation
Summative testing for the 2015 certification must address the following 12 areas:
Section 170.315(a)(1) Computerized provider order entry – medications
Section 170.315(a)(2) Computerized provider order entry – laboratory
Section 170.315(a)(3) Computerized provider order entry – diagnostic imaging
Section 170.315(a)(4) Drug-drug, drug-allergy interaction checks
Section 170.315(a)(5) Demographics (New)
Section 170.315(a)(6) Problem list (New)
Section 170.315(a)(7) Medication list
Section 170.315(a)(8) Medication allergy list
Section 170.315(a)(9) Clinical decision support
Section 170.315(a)(14) Implantable device list (New)
Section 170.315(b)(2) Clinical information reconciliation and incorporation
Section 170.315(b)(3) Electronic prescribing
How should the Results Presented?
Results from the summative study should be created and presented to the ONC using the Customized Common Industry Format (CCIF) Template for EHR Usability Testing (NISTIR 7742 see http://1.usa.gov/gAZFDt ) These reports typically include an executive summary, an introduction, a method section, discussion of the results and any appendices
The following are required for submission to the ONC for the 2105 Certification:
- Name and version of the product
- Date and location of the test
- Test environment
- Description of the intended users
- Total number of participants
- Description of participants as follows:
- Professional experience
- Computer experience
- Product experience
- Description of the user tasks that were tested and association of each task to corresponding certification criteria
- List of the specific metrics captured during the testing
- Task Success (%)
- Task Failures (%)
- Task Standard Deviations (%)
- Task Performance Time
- User Satisfaction Rating (Scale with 1 as very difficult and 5 as very easy)
- Test results for each task using metrics listed above
- Results and data analysis narrative:
- Major test finding
- Areas for improvement
NISTIR 7742 Submission Requirements, New Requirements and Compliance Guidance
ONC now requires that Health IT developers include the specific information from the NISTIR 7742 “Customized Common Industry Format Template for Electronic Health Record Usability Testing. All of the data elements described in NISTIR 7742 and every section must be completed, including “major findings” and “areas for improvement.”
ONC identified the requirements for the submission of the demographic characteristics of test participants. These test participant characteristics must reflect the audience of current and future users of the EHR.
In accordance with NISTIR 7804, ONC requires that the test scenarios be based upon an analysis of critical use risks for patient safety, which can be mitigated or eliminated by improvements to the user interface design.
Health IT developers must now select an industry standard User-Centered design process because compliance with this certification criterion requires submission of the name, description, and citation (URL and/or publication citation) of the process that was selected. Examples of method(s) that could be employed for UCD, including ISO 9241-11, ISO 13407, ISO 16982, ISO/IEC 62366, ISO 9241-210 and NISTIR 7741.
In the event that a health IT developer selects a UCD process that was not an industry standard (i.e., not developed by a voluntary consensus standards organization), but is based on one or more industry standard processes, the developer may name the process(es) and provide an outline of the process in addition to a short description as well as an explanation of the reason(s) why use of any of the existing UCD standards was impractical.
We recommend that EHR vendors follow the NISTIR 7741 guidelines.
ONC will now also permit the submission of an alternative acceptable user satisfaction measure such as the (Single Ease-of-use Question, System Usability Scale, or Software Usability Measurement Inventory) to meet the requirements of this criterion. Stated another way, a health IT developer could meet the proposed NIST 7742 based approach for user satisfaction or provide documentation of an alternative acceptable user satisfaction measure.
Number of Test Participants
ONC has adopted a provision as part of this criterion that requires 10 participants per criterion/capability as a mandatory minimum for the purposes of testing and certification. ONC suggests that this number will balance any potential burden for health IT developers, including small developers.
However, they strongly encourage health IT developers to exceed the mandatory minimum in an effort to identify and resolve more problems.
Formative vs. Summative testing
ONC considered whether a standardized report of formative testing could be submitted as an alternative to the criteria for which summative testing would be required, if formative testing reflected a thorough process that has tested and improved the usability of a product. Ultimately ONC agreed with commenters that contend that formative testing should not be a substitute for summative testing for the purposes of this criterion.
Retesting and Certification
ONC-ACB determinations related to the ongoing applicability of the SED certification criterion to certified health IT for the purposes of inherited certified status (§ 170.550(h)), adaptations and other updates would be based on the extent of changes to user-interface aspects of one or more capabilities to which UCD had previously been applied.
In other words if you make a significant change to the user interface of your EHR, you may have to re-certify for Safety-enhanced Design
Our Chief Experience Officer, Bennett Lauber is a member of the ONC Health IT Policy Committee Implementation, Usability, and Safety Workgroup. He shares his experiences working with EHR vendors with Federal regulators to influence public policy on the usability of health IT.
The Usability People conduct and report formative and summative usability evaluations using the NISTIR 7742 Customized Common Industry Format Template for EHR Usability Testing—suitable for satisfying the Safety-enhanced Design criteria portion of your Meaningful Use stage 2 and the 2015 Edition Certification.