REAL WORLD TESTING PLAN TEMPLATE

BACKGROUND & INSTRUCTIONS

Under the ONC Health IT Certification Program (Program), health IT developers are required to conduct Real World Testing of their certified health IT (45 CFR 170.405). The Office of the National Coordinator for Health Information Technology (ONC) issues Real World Testing resources to clarify health IT developers’ responsibilities for conducting Real World Testing, to identify topics and specific elements of Real World Testing that ONC considers a priority, and to assist health IT developers in developing their Real-World Testing plans.

Health IT developers have maximum flexibility to develop innovative plans and measures for Real World Testing. As developers are planning how they will execute Real World Testing, they should consider the overall complexity of the workflows and use cases within the care settings in which they market their certified health IT to determine the approaches they will take. This Real-World Testing plan template was created to assist health IT developers in organizing the required information that must be submitted for each element in their Real-World Testing plan. While the use of this template is voluntary, health IT developers may find it useful in preparing their Real-World Testing plans. Health IT developers must submit one plan for each year of Real-World Testing (see resources listed below for specific timelines and due dates). ONC does not encourage updating plans outside the submission timeline and will not post updates on the Certified Health IT Product List (CHPL). If adjustments to approaches are made throughout Real World Testing, the health IT developer should reflect these adjustments in their Real-World Testing results report. ONC expects that the Real-World Testing results report will include a description of these types of changes, the reasons for them, and how intended outcomes were more efficiently met as a result. While every effort has been made to ensure the accuracy of restatements of 45 CFR Part 170, this template is not a legal document. The official program requirements are contained in the relevant laws and regulations. This resource should be read and understood in conjunction with the following companion resources, which describe in detail many of the Program requirements referenced in this resource.

Health IT developers should also review the following regulatory materials, which establish the core requirements and responsibilities for Real World Testing under the Program.

  • 21st Century Cures Act: Interoperability, Information Blocking, and the ONC Health IT Certification Program final rule, 85 FR 25642 (May 1, 2020) (ONC Cures Act Final Rule)

TEMPLATE INSTRUCTIONS

The following template is organized by elements required to be submitted in the Real World Testing plan. Each section provides a field for submitting responses and/or explanations for how the health IT developer will address each required element in their Real World Testing approach. These fields serve as a foundation of information required for developing a Real World Testing plan and can be expanded with additional rows or columns to address the specific needs of the Real World Testing plan being submitted.

GENERAL INFORMATION

Plan Report ID Number: [For ONC-Authorized Certification Body use only]

Developer Name: Netsmart

Product Name(s): gEHRiMed

Version Number(s): 4.0 4.1, 4.2

Certified Health IT: 170.315(b)(1), 170.315(b)(2), 170.315(b)(6), 170.315(e)(1), 170.315(f)(1), 170.315(b)(2), 170.315(g)(7), 170.315(g)(8), 170.315(g)(9)

Product List (CHPL) ID(s): 15.04.04.1532.gEHR.04.02.1.210420

Developer Real World Testing Page URL: https://www.ntst.com/-/media/pdfs/certifications/gmd-rwt-patient-portal.pdf

JUSTIFICATION FOR REAL WORLD TESTING APPROACH

Provide an explanation for the overall approach to Real World Testing, including an outline of the approach and how data will be used to demonstrate successful Real World Testingi.

All measures should reasonably align with the elements within a Real World Testing plan, the scope of the certification, the types of settings in which the certified health IT is marketed, and other factors relevant to the implementation of the certified Health IT Module(s). The justification should reflect how each element within the plan is relevant to the developer’s overall strategy for meeting the Real World Testing Condition and Maintenance of Certification requirements.

Note: A single Real World Testing plan may address multiple products and certification criteria for multiple care settings.

Use Case: Evaluating real-world use of patient engagement – view, download, and transmit to a third party We worked with a client that used the patient portal in the real world to evaluate real world use of the portal to view health IT, and download health data. We will monitor this use and functionality over the performance year to evaluate use and functionality as laid out in specifications and regulations for the following associated module:

170.315(e)(1): Patient engagement - View, download, and transmit to 3rd party.

STANDARDS UPDATES (INCLUDING STANDARDS VERSION ADVANCEMENT PROCESS (SVAP) AND UNITED STATES CORE DATA FOR INTEROPERABILITY (USCDI))

Both required and voluntary standards updates must be addressed in the Real World Testing plan. Real World Testing plans must include all certified health IT updated to newer versions of standards prior to August 31 of the year in which the updates were made.

Describe approach(es) for demonstrating conformance to all certification requirements using each standard to which the health IT is certified. List each version of a given standard separately. For each version of a standard submit the following:

  • Identify standard versions
  • Indicate what certification criteria in which product(s) has been updated
  • If reporting for multiple products, identify the certification criteria that were affected by the update for each of the associated products
  • CHPL ID for each Health IT Module
  • Method used for standard update (e.g., SVAP)
  • Date notification sent to ONC-ACB
  • If SVAP, date notification sent to customers
  • Measure used to demonstrate conformance with updated standard(s)
  • Which certification criteria were updated to USCDI and/or to which version of USCDI was the certification criteria updated?

Standard (and version)

Paragraph (e)(1)(i)
§ 170.204(a)(1) Web Content Accessibility Guidelines (WCAG) 2.0, Level A Conformance
§ 170.204(a)(2) WCAG 2.0, Level AA Conformance
§ 170.205(a)(4) HL7 Implementation Guide for CDA Release 2 Consolidation CDA Templates for Clinical Notes (US Realm), Draft Standard for Trial Use Release 2.1 C-CDA 2.1 with Errata, August 2015, June 2019 (with Errata)
§ 170.205(a)(5) Health Level 7 (HL7®) CDA R2 IG: C-CDA Templates for Clinical Notes R2.1 Companion Guide, Release 2, October 2019, IBR approved for § 170.205(a)(5)

Standards Version Advancement Process (SVAP) Version(s) Approved
Web Content Accessibility Guidelines (WCAG) 2.1, Level A Conformance
Web Content Accessibility Guidelines (WCAG) 2.1, Level AA Conformance

Paragraph (e)(1)(i)(A)
§ 170.213 United States Core Data for Interoperability (USCDI)
Laboratory test report(s):

  1. The information for a test report as specified all the data specified in 42 CFR 493.1291(c)(1) through (7);
    1. For positive patient identification, either the patient's name and identification number, or a unique patient identifier and identification number.
    2. The name and address of the laboratory location where the test was performed.
    3. The test report date.
    4. The test performed.
    5. The specimen source, when appropriate.
    6. The test result and, if applicable, the units of measurement or interpretation, or both.
    7. Any information regarding the condition and disposition of specimens that do not meet the laboratory's criteria for acceptability.
  2. The information related to reference intervals or normal values as specified in 42 CFR 493.1291(d) – Pertinent “reference intervals” or “normal” values, as determined by the laboratory performing the tests, must be available to the authorized person who ordered the tests and, if applicable, the individual responsible for using the test results.
  3. The information for corrected reports as specified in 42 CFR 493.1291(k)(2) – When errors in the reported patient test results are detected, the laboratory must do the following: Issue corrected reports promptly to the authorized person ordering the test and, if applicable, the individual using the test results.

Paragraph (e)(1)(i)(B) § 170.213 United States Core Data for Interoperability (USCDI)
§ 170.205(a)(4) HL7® Implementation Guide for CDA Release 2 Consolidation CDA Templates for Clinical Notes (US Realm), Draft Standard for Trial Use Release 2.1 C-CDA 2.1, August 2015, June 2019 (with Errata)
§ 170.205(a)(5) Health Level 7 (HL7®) CDA R2 IG: C-CDA Templates for Clinical Notes R2.1 Companion Guide, Release 2, October 2019, IBR approved for § 170.205(a)(5)
Laboratory test reports:

  1. The information for a test report as specified all the data specified in 42 CFR 493.1291(c)(1) through (7) –
    1. For positive patient identification, either the patient's name and identification number, or a unique patient identifier and identification number.
    2. The name and address of the laboratory location where the test was performed.
    3. The test report date.
    4. The test performed.
    5. Specimen source, when appropriate.
    6. The test result and, if applicable, the units of measurement or interpretation, or both.
    7. Any information regarding the condition and disposition of specimens that do not meet the laboratory's criteria for acceptability.
  2. The information related to reference intervals or normal values as specified in 42 CFR 493.1291(d) – Pertinent “reference intervals” or “normal” values, as determined by the laboratory performing the tests, must be available to the authorized person who ordered the tests and, if applicable, the individual responsible for using the test results.
  3. The information for corrected reports as specified in 42 CFR 493.1291(k)(2) – When errors in the reported patient test results are detected, the laboratory must do the following: Issue corrected reports promptly to the authorized person ordering the test and, if applicable, the individual using the test results.

Paragraph (e)(1)(i)(C)
Please refer to the standards required for § 170.315(d)(9) “trusted connection” for the encrypted method of electronic transmission.

Paragraph (e)(1)(ii)
§ 170.210(g) Synchronized clocks. The date and time recorded utilize a system clock that has been synchronized following (RFC 5905) Network Time Protocol Version 4 (incorporated by reference in § 170.299).

Updated certification criteria and associated product

No updates have been made.

Health IT Module CHPL ID

15.04.04.1532.gEHR.04.02.1.210420

Method used for standard update

N/A, updates have not been made

Date of ONC-ACB notification

04/20/2021

Date of customer notification (SVAP only)

n/a

Conformance measure

n/a

USCDI-updated certification criteria (and USCDI version)

n/a

MEASURES USED IN OVERALL APPROACH

Each plan must include at least one measurement/metric that addresses each applicable certification criterion in the Health IT Module’s scope of certification. Describe the method for measuring how the approach(es) chosen meet the intent and purpose of Real World Testing.

For each measurement/metric, describe the elements below:

  • Description of the measurement/metric
  • Associated certification criteria
  • Justification for selected measurement/metric
  • Care setting(s) that is addressed
  • Expected outcomes

DESCRIPTION OF MEASUREMENT/METRIC

Describe the measure(s) that will be used to support the overall approach to Real World Testing.

Measurement/Metric

Description

Number of ‘views’ by patients of their health data over an identified population denominator.

Patient Engagement – Patient engagement in their health data by viewing their data per standards related to 170.315(e)(1) will be reviewed from the IIS logs / database to determine usage over time for the identified denominator.

Number of ‘downloads’ by patients of their health data over an identified population denominator.

Patient Engagement – Patient engagement in their health data by downloading their data per standards related to 170.315(e)(1) will be reviewed from the database to determine usage over time for the identified denominator.

ASSOCIATED CERTIFICATION CRITERIA

List certification criteria associated with the measure and if updated to the 2015 Edition Cures Update criteria.

Measurement/Metric

Associated Certification Criteria

170.315(e)(1): Patient Engagement - View

  1. View. Patients (and their authorized representatives) must be able to use health IT to view, at a minimum, the following data:
    1. The data classes expressed in the standard in § 170.213 (which should be in their English (i.e., non-coded) representation if they associate with a vocabulary/code set), and in accordance with § 170.205(a)(4) and (a)(5), and paragraphs (e)(1)(i)(A)(3)(i) through (iii) of this section, or
    2. The Common Clinical Data Set in accordance with § 170.205(a)(4) and paragraphs (e)(1)(i)(A)(3)(i) through (iv) of this section for the period until December 31, 2022.
    3. The following data classes:
      1. Assessment and plan of treatment. In accordance with the “Assessment and Plan Section (V2)” of the standard specified in § 170.205(a)(4); or in accordance with the “Assessment Section (V2)” and “Plan of Treatment Section (V2)” of the standard specified in § 170.205(a)(4).
      2. Goals. In accordance with the “Goals Section” of the standard specified in § 170.205(a)(4).
      3. Health concerns. In accordance with the “Health Concerns Section” of the standard specified in § 170.205(a)(4).
      4. Unique device identifier(s) for a patient's implantable device(s). In accordance with the “Product Instance” in the “Procedure Activity Procedure Section” of the standards specified in § 170.205(a)(4).
    4. Ambulatory setting only. Provider's name and office contact information.
    5. Inpatient setting only. Admission and discharge dates and locations; discharge instructions; and reason(s) for hospitalization.
    6. Laboratory test report(s). Laboratory test report(s), including:
      1. The information for a test report as specified all the data specified in 42 CFR 493.1291(c)(1) through (7);
      2. The information related to reference intervals or normal values as specified in 42 CFR 493.1291(d); and
      3. The information for corrected reports as specified in 42 CFR 493.1291(k)(2).
    7. Diagnostic image report(s).

170.315(e)(1): Patient Engagement - Download

  1. Download.
    1. Patients (and their authorized representatives) must be able to use technology to download an ambulatory summary or inpatient summary (as applicable to the health IT setting for which certification is requested) in the following formats:
      1. Human readable format; and
      2. The format specified in accordance to the standard specified in § 170.205(a)(4) following the CCD document template.
    2. When downloaded according to the standard specified in § 170.205(a)(4) following the CCD document template, the ambulatory summary or inpatient summary must include, at a minimum, the following data (which, for the human readable version, should be in their English representation if they associate with a vocabulary/code set):
      1. Ambulatory setting only. All of the data specified in paragraph (e)(1)(i)(A)(1), (2), (4), and (5) of this section.
      2. Inpatient setting only. All of the data specified in paragraphs (e)(1)(i)(A)(1), and (3) through (5) of this section.
    3. Inpatient setting only. Patients (and their authorized representatives) must be able to download transition of care/referral summaries that were created as a result of a transition of care (pursuant to the capability expressed in the certification criterion specified in paragraph (b)(1) of this section).

JUSTIFICATION FOR SELECTED MEASUREMENT/METRIC

Provide an explanation for the measurement/metric selected to conduct Real World Testing.

Measurement/Metric

Justification

Number of ‘views’ by patients of their health data over an identified population denominator.

This metric allows testing of how patient engagement – viewing health information is used by clients in the real-world setting per the standards related to viewing patient data, displaying in the standard format, allowing client selection of assessment, plan, treatment, goals, health concerns, unique device identifiers, corrected reports if/where applicable, per 170.315(e)(1) standards related to viewing health data.

Number of ‘downloads’ by patients of their health data over an identified population denominator.

This metric allows testing of how patient engagement – downloading health information is being used by clients in the real-world setting. We will evaluate downloads and usage over time for our target population per the standards laid out in 170.315(e)(1) per standards related to downloading health data.

Transmissions

Transmissions is a functionality not used by our user base after evaluation. In consulting with our ONC-ACB, we found that as ‘view / download’ was used, additional test data would not be needed to be created for ‘transmit’ functionality.

Care Setting(s)

The expectation is that a developer’s Real World Testing plan will address each type of clinical setting in which their certified health IT is marketed. Health IT developers are not required to test their certified health IT in every setting in which it is marketed for use. Developers should address their choice of care and/or practice settings to test and provide a justification for the chosen approach.

Note: Health IT developers may bundle products by care setting, criteria, etc. and design one plan to address each, or they may submit any combination of multiple plans that collectively address their products and the care settings in which they are marketed

List each care setting which is covered by the measure and an explanation for why it is included

Care Setting

Justification

The user population for this functionality is in a post-acute, long term care setting.

This is the care setting in which Gehrimed electronic health record technology is used.

EXPECTED OUTCOMES

Health IT developers should detail how the approaches chosen will successfully demonstrate that the certified health IT:

(1) is compliant with the certification criteria, including the required technical standards and vocabulary codes sets;

(2) is exchanging electronic health information (EHI) in the care and practice settings for which it is marketed for use; and/or,

(3) EHI is received by and used in the certified health IT.

(from 85 FR 25766)

Not all of the expected outcomes listed above will be applicable to every certified Health IT Module, and health IT developers may add an additional description of how their measurement approach best addresses the ongoing interoperability functionality of their product(s). Health IT developers could also detail outcomes that should not result from their measurement approach if that better describes their efforts.

Within this section, health IT developers should also describe how the specific data collected from their Real World Testing measures demonstrate expected results. Expected outcomes and specific measures do not necessarily have to include performance targets or benchmarks, but health IT developers should provide context for why specific measures were selected and how the metrics demonstrate individual criterion functionality, EHI exchange, and/or use of EHI within certified health IT, as appropriate.

Measurement/Metric

Expected Outcomes

Number of ‘views’ by patients of their health data over an identified population denominator.

Expected validation of normal viewing of patient data over time. We will look at views over time and note/address any issues if applicable while monitoring. Validating view activity automatically verifies 170.205(a)(1)/(2) compliance as well as 107.205(a)(4)/(5) compliance based on how data view is setup, as well as CCDS / USCDI / HL7 standards as outlined in 170.213, 170.205(a)(4)/(5)

Number of ‘downloads’ by patients of their health data over an identified population denominator.

Expected validation of normal downloading of patient data in human readable format with the data they selected. We expect the number of download attempts to be congruent with downloads for a patient’s data. Downloading automatically validates functionality of associated certified criteria related to 170.205(a)(4)/(5)

SCHEDULE OF KEY MILESTONES

Include steps within the Real World Testing plan that establish milestones within the process. Include details on how and when the developer will implement measures and collect data. Key milestones should be relevant and directly related to expected outcomes discussed in the next section.

For each key milestone, describe when Real World Testing will begin in specific care settings and the date/timeframe during which data will be collected.

Key Milestone

Care Setting

Date/Timeframe

Identification of Send / Receive denominator group(s).

Long Term Post-Acute Care

1/15/2022

Set up of evaluation of Views, Downloads based upon IIS information.

Long Term Post-Acute Care

2/15/2022

Over a timeframe specified: analyze views, downloads via IIS / database information as/where applicable.

Long Term Post-Acute Care

2/30/2022

Automated review of view, download data with monthly review that data is valid.

Long Term Post-Acute Care

3/15/2022

Submission to ONC-ACB by determined date (TBD on ONC0-ACB Direction)

Long Term Post-Acute Care

~12/31/2022

ATTESTATION

The Real World Testing plan must include the following attestation signed by the health IT developer authorized representative.

Note: The plan must be approved by a health IT developer authorized representative capable of binding the health IT developer for execution of the plan and include the representative's contact information.ii

This Real World Testing plan is complete with all required elements, including measures that address all certification criteria and care settings. All information in this plan is up to date and fully addresses the health IT developer’s Real World Testing requirements.

Authorized Representative Name: Benjamin Britton

Authorized Representative Email: bbritton@ntst.com

Authorized Representative Phone: 970-658-6897

Authorized Representative Signature:

iCertified health IT continues to be compliant with the certification criteria, including the required technical standards and vocabulary codes sets; certified health IT is exchanging EHI in the care and practice settings for which it is marketed for use; and EHI is received by and used in the certified health IT. (85 FR 25766)

iihttps://www.federalregister.gov/d/2020-07419/p-3582