Under the ONC Health IT Certification Program (Program), health IT developers are required to conduct Real World Testing of their certified health IT (45 CFR 170.405). The Office of the National Coordinator for Health Information Technology (ONC) issues Real World Testing resources to clarify health IT developers’ responsibilities for conducting Real World Testing, to identify topics and specific elements of Real World Testing that ONC considers a priority, and to assist health IT developers in developing their Real-World Testing plans.
Health IT developers have maximum flexibility to develop innovative plans and measures for Real World Testing. As developers are planning how they will execute Real World Testing, they should consider the overall complexity of the workflows and use cases within the care settings in which they market their certified health IT to determine the approaches they will take. This Real-World Testing plan template was created to assist health IT developers in organizing the required information that must be submitted for each element in their Real-World Testing plan. While the use of this template is voluntary, health IT developers may find it useful in preparing their Real-World Testing plans. Health IT developers must submit one plan for each year of Real-World Testing (see resources listed below for specific timelines and due dates). ONC does not encourage updating plans outside the submission timeline and will not post updates on the Certified Health IT Product List (CHPL). If adjustments to approaches are made throughout Real World Testing, the health IT developer should reflect these adjustments in their Real-World Testing results report. ONC expects that the Real-World Testing results report will include a description of these types of changes, the reasons for them, and how intended outcomes were more efficiently met as a result. While every effort has been made to ensure the accuracy of restatements of 45 CFR Part 170, this template is not a legal document. The official program requirements are contained in the relevant laws and regulations. This resource should be read and understood in conjunction with the following companion resources, which describe in detail many of the Program requirements referenced in this resource.
Health IT developers should also review the following regulatory materials, which establish the core requirements and responsibilities for Real World Testing under the Program.
The following template is organized by elements required to be submitted in the Real World Testing plan. Each section provides a field for submitting responses and/or explanations for how the health IT developer will address each required element in their Real World Testing approach. These fields serve as a foundation of information required for developing a Real World Testing plan and can be expanded with additional rows or columns to address the specific needs of the Real World Testing plan being submitted.
Plan Report ID Number: [For ONC-Authorized Certification Body use only]
Developer Name: Netsmart
Product Name(s): gEHRiMed
Version Number(s): 4.0 4.1, 4.2
Certified Health IT: 170.315(b)(1), 170.315(b)(2), 170.315(b)(6), 170.315(e)(1), 170.315(f)(1), 170.315(b)(2), 170.315(g)(7), 170.315(g)(8), 170.315(g)(9)
Product List (CHPL) ID(s): 15.04.04.1532.gEHR.04.02.1.210420
Developer Real World Testing Page URL: https://www.ntst.com/-/media/pdfs/certifications/gmd-rwt-transitions.pdf
Provide an explanation for the overall approach to Real World Testing, including an outline of the approach and how data will be used to demonstrate successful Real World Testingi.
All measures should reasonably align with the elements within a Real World Testing plan, the scope of the certification, the types of settings in which the certified health IT is marketed, and other factors relevant to the implementation of the certified Health IT Module(s). The justification should reflect how each element within the plan is relevant to the developer’s overall strategy for meeting the Real World Testing Condition and Maintenance of Certification requirements.
Note: A single Real World Testing plan may address multiple products and certification criteria for multiple care settings.
Use Case: Evaluating real-world use of C-CDA, and C-CDA conformance, we worked with clients who are using these modules listed below in their current environment. We identified a client that used C-CDA functionality to coordinate transitions of care, reconcile and incorporate clinical information, and export data to collaborating providers as follow up communication for primary care providers and other patient-affiliated providers. The practice regularly used certified CCDA technology to send patient data for transitions of care, to reconcile and incorporate clinical information, and to export data where / when necessary for patient / provider use. The methodology used by clients in this fashion addresses the certified modules:
Both required and voluntary standards updates must be addressed in the Real World Testing plan. Real World Testing plans must include all certified health IT updated to newer versions of standards prior to August 31 of the year in which the updates were made.
Describe approach(es) for demonstrating conformance to all certification requirements using each standard to which the health IT is certified. List each version of a given standard separately. For each version of a standard submit the following:
Standard (and version)
Standards associated with 170.315(b)(1):
Paragraphs (b)(1)(i)(A) and (B)
§ 170.202(a)(2) Direct Project: ONC Applicability Statement for Secure Health Transport, Version 1.2 August 2015
§ 170.202(d) ONC Implementation Guide for Direct Edge Protocols, Version 1.1, June 25, 2014
Paragraph (b)(1)(i)(C)
§ 170.205(p)(1) IHE IT Infrastructure Technical Framework Volume 2b (ITI TF- 2b)
Paragraph (b)(1)(ii)
§ 170.205(a)(3) Health Level 7 (HL7®) Implementation Guide for CDA® Release 2: IHE Health Story Consolidation, DSTU Release 1.1 (US Realm) Draft Standard for Trial Use July 2012
§ 170.205(a)(4) HL7® Implementation Guide for CDA Release 2 Consolidation CDA Templates for Clinical Notes (US Realm), Draft Standard for Trial Use Release 2.1 C-CDA 2.1, August 2015, June 2019 (with Errata)
§ 170.205(a)(5) HL7® CDA R2 IG: C-CDA Templates for Clinical Notes R2.1 Companion Guide, Release 2, October 2019, IBR approved for § 170.205(a)(5).
Paragraphs (b)(1)(iii)(A)-(F)
§ 170.213 United States Core Data for Interoperability (USCDI)
§ 170.205(a)(3) Health Level 7 (HL7®) Implementation Guide for CDA® Release 2: IHE Health Story Consolidation, DSTU Release 1.1 (US Realm) Draft Standard for Trial Use July 2012
§ 170.205(a)(4) HL7 Implementation Guide for CDA Release 2 Consolidation CDA Templates for Clinical Notes (US Realm), Draft Standard for Trial Use Release 2.1 C-CDA 2.1 with Errata, August 2015, June 2019 (with Errata)
§ 170.205(a)(5) Health Level 7 (HL7®) CDA R2 IG: C-CDA Templates for Clinical Notes R2.1 Companion Guide, Release 2, October 2019, IBR approved for § 170.205(a)(5).
§ 170.207(a)(4) International Health Terminology Standards Development Organisation (IHTSDO) Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT® ) U.S. Edition, September 2019 Release
§ 170.207(i) Encounter diagnoses: The code set specified at 45 CFR 162.1002(c)(2) for the indicated conditions ICD-10-CM as maintained and distributed by HHS, for the following conditions:
Paragraph (b)(1)(iii)(G)
§ 170.205(a)(4) HL7® Implementation Guide for CDA Release 2 Consolidation CDA Templates for Clinical Notes (US Realm), Draft Standard for Trial Use Release 2.1 C-CDA 2.1, August 2015, June 2019 (with Errata)
§ 170.205(a)(5) HL7® CDA R2 IG: C-CDA Templates for Clinical Notes R2.1 Companion Guide, Release 2, October 2019, IBR approved for § 170.205(a)(5).
§ 170.207(n)(1) Birth sex must be coded in accordance with HL7 Version 3 Standard, Value Sets for AdministrativeGender and NullFlavor attributed as follows:
§ 170.207(q)(1) International Telecommunication Union E.123: Notation for national and international telephone numbers, e-mail addresses and web addresses and International Telecommunication Union E.164: The international public telecommunication numbering plan
Standards associated with 170.315(b)(2):
Paragraphs (b)(2)(i) and (ii)
§ 170.213 United States Core Data for Interoperability (USCDI)
§ 170.205(a)(3) Health Level 7 (HL7®) Implementation Guide for CDA® Release 2: IHE Health
Story Consolidation, DSTU Release 1.1 (US Realm) Draft Standard for Trial Use July 2012
§ 170.205(a)(4) HL7® Implementation Guide for CDA® Release 2: Consolidated CDA Templates
for Clinical Notes (US Realm), Draft Standard for Trial Use Release 2.1 August 2015, June 2019
(with Errata)
§ 170.205(a)(5) HL7® CDA R2 IG: C-CDA Templates for Clinical Notes R2.1 Companion Guide,
Release 2, October 2019, IBR approved for § 170.205(a)(5)
Paragraphs (b)(2)(iii)(B) – ( D)
§ 170.213 United States Core Data for Interoperability (USCDI)
Paragraph (b)(2)(iv)
§ 170.205(a)(4) HL7® Implementation Guide for CDA® Release 2: Consolidated CDA Templates
for Clinical Notes (US Realm), Draft Standard for Trial Use Release 2.1 August 2015, June 2019
(with Errata)
§ 170.205(a)(5) HL7® CDA R2 IG: C-CDA Templates for Clinical Notes R2.1 Companion Guide,
Release 2, October 2019, IBR approved for § 170.205(a)(5)
Standards associated with 170.315(b)(3):
Paragraph (b)(6)(ii)
§ 170.205(a)(4) HL7 Implementation Guide for CDA® Release 2: Consolidated CDA Templates
for Clinical Notes (US Realm), Draft Standard for Trial Use Release 2.1, August 2015
§ 170.207(a)(4) International Health Terminology Standards Development Organisation
(IHTSDO) Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT®), U.S. Edition,
September 2015 Release
§ 170.207(i) ICD-10-CM
Please refer to the Data Elements and Vocabularies applicable to the Common Clinical Data Set (CCDS) as outlined in the Common Clinical Data Set Reference Document
Updated certification criteria and associated product
No updates have been made.
Health IT Module CHPL ID
15.04.04.1532.gEHR.04.02.1.210420
Method used for standard update
N/A, updates have not been made
Date of ONC-ACB notification
04/20/2021
Date of customer notification (SVAP only)
n/a
Conformance measure
n/a
USCDI-updated certification criteria (and USCDI version)
n/a
Each plan must include at least one measurement/metric that addresses each applicable certification criterion in the Health IT Module’s scope of certification. Describe the method for measuring how the approach(es) chosen meet the intent and purpose of Real World Testing.
For each measurement/metric, describe the elements below:
Describe the measure(s) that will be used to support the overall approach to Real World Testing.
Measurement/Metric
Description
Number of CCDAs sent for a % population over a time period. Numerator: Total number of CCDAs sent; Denominator: Total population.
Care coordination – transitions of care will be evaluated by analyzing a current active population using the CCDA ‘send’ functionality over the assessed population. This addressed CCDA sending as well as CCDA creation.
Number of CCDAs received for a % population over a time period. Numerator: Total number of CCDAs Received; Denominator: Total population.
Care coordination – transitions of care will be evaluated by analyzing a current active population using the CCDA ‘receive’ functionality over the assessed population.
Number of CCDAs Displayed for a % population over a time period. Numerator: Total number of CCDAs Displayed; Denominator: Total population.
Display and reconciliation are congruent in our HealthIT, and depend on the user to determine what will be reconciled. We’ll observe the number of CCDAs displayed over time for our population, and subsequently incorporated.
Number of CCDAs Reconciled for a % population over a time period. Numerator: Total number of CCDAs Reconciled; Denominator: Total population.
We’ll observe reconciled CCDAs as a function of total number of CCDAs received to evaluate real world functionality of this module.
CCDA creation: Number of CCDAs created for the target population over time
This will allow us to evaluate CCDA creation for users in the real world over evaluated time.
List certification criteria associated with the measure and if updated to the 2015 Edition Cures Update criteria.
Measurement/Metric
Associated Certification Criteria
170.315(b)(1): Care Coordination – Transitions of Care
(i) Send and receive via edge protocol
(ii) Validate and display
(iii) Create.
170.315 (b)(2) Clinical information reconciliation and incorporation
(i) General requirements.
(ii) Correct patient
(iii) Reconciliation.
(iv) System verification.
170.315 (b)(6) Data export
(i) General requirements for export summary configuration.
Provide an explanation for the measurement/metric selected to conduct Real World Testing.
Measurement/Metric
Justification
CCDA send, receive, display, reconcile/incorporate, create
The metrics for CCDA functionality reflect current real-world evaluation and clinical use of CCDA functionality to coordinate care per the standards in 170.315(b)(1). We will be looking at IIS data to determine CCDAs sent/created, received/reconciled over our user base.
We’re further able to evaluate clinical information reconciliation and incorporation pursuant to 170.315(b)(2) by analyzing IIS data to evaluate CCDA data received/incorporated. Other items related to the standards occur in the backend automatically (patient matching, correct pt., system verification).
Lastly, we’ll assess creation and export of CCDAs pursuant to standards outlined in 170.315(b)(6) for user creation of CCDAs per general export summary requirements; this will also be done via IIS evaluation to analyze CCDAs sent for the evaluated population.
The expectation is that a developer’s Real World Testing plan will address each type of clinical setting in which their certified health IT is marketed. Health IT developers are not required to test their certified health IT in every setting in which it is marketed for use. Developers should address their choice of care and/or practice settings to test and provide a justification for the chosen approach.
Note: Health IT developers may bundle products by care setting, criteria, etc. and design one plan to address each, or they may submit any combination of multiple plans that collectively address their products and the care settings in which they are marketed
List each care setting which is covered by the measure and an explanation for why it is included
Care Setting
Justification
The user population for this functionality is in a post-acute, long term care setting.
This is the care setting in which Gehrimed electronic health record technology is used; this is the care setting where these modules will be evaluated.
Health IT developers should detail how the approaches chosen will successfully demonstrate that the certified health IT:
(1) is compliant with the certification criteria, including the required technical standards and vocabulary codes sets;
(2) is exchanging electronic health information (EHI) in the care and practice settings for which it is marketed for use; and/or,
(3) EHI is received by and used in the certified health IT.
(from 85 FR 25766)
Not all of the expected outcomes listed above will be applicable to every certified Health IT Module, and health IT developers may add an additional description of how their measurement approach best addresses the ongoing interoperability functionality of their product(s). Health IT developers could also detail outcomes that should not result from their measurement approach if that better describes their efforts.
Within this section, health IT developers should also describe how the specific data collected from their Real World Testing measures demonstrate expected results. Expected outcomes and specific measures do not necessarily have to include performance targets or benchmarks, but health IT developers should provide context for why specific measures were selected and how the metrics demonstrate individual criterion functionality, EHI exchange, and/or use of EHI within certified health IT, as appropriate.
Measurement/Metric
Description
Number of CCDAs sent for a % population over a time period. Numerator: Total number of CCDAs sent; Denominator: Total population.
Based on database evaluation, we expect to see CCDAs sent without error for the population assessed, over time; Several items happen automatically in the backend as a result of successful CCDA send, pursuant to the standards set for the certified modules including sending via Direct Edge Protocols, HL7 compliance, USCDI compliance depending on the certified modality used, and the manner used.
Number of CCDAs received for a % population over a time period. Numerator: Total number of CCDAs Received; Denominator: Total population.
Assessing IIS logs, we expect to see CCDAs received are incorporated for the identified denominator population. Receipt / incorporation occurs in at the same time. Several items happen automatically in the backend as a result of successful CCDA receive, pursuant to the standards set for the certified modules including sending via Direct Edge Protocols, HL7 compliance, USCDI compliance depending on the certified modality used, and the manner used.
Number of CCDAs Displayed for a % population over a time period. Numerator: Total number of CCDAs Displayed; Denominator: Total population.
We expect this to be close in number to the number of CCDAs received. Upon evaluation of database logs received, we expect incorporation / reconciliation for CCDAs over the denominator population over the timeframe evaluated. When CCDAs are received, they are displayed for incorporation simultaneously. Several items happen automatically in the backend as a result of successful CCDA display, pursuant to the standards set for the certified modules including sending via Direct Edge Protocols, HL7 9 compliance, USCDI compliance depending on the certified modality used, and the manner used.
Number of CCDAs Reconciled for a % population over a time period. Numerator: Total number of CCDAs Reconciled; Denominator: Total population.
This is expected to be congruent with the number of CCDAs received / displayed as this functionality is congruent in the process of receiving, reviewing, reconcile. We expect to see a similar number for CCDAs received over the denominator population over time. Several items happen automatically in the backend as a result of successful CCDA reconcile, pursuant to the standards set for the certified modules including sending via Direct Edge Protocols, HL7 compliance, USCDI compliance depending on the certified modality used, and the manner used.
CCDA creation: Number of CCDAs created for the target population over time
We expect this to be close to the number of CCDAs sent for care coordination, transitions of care, other provider information as the functionality for creation is generally tied with CCDA send. This will be evaluated via database logs for the identified population over time. Several items happen automatically in the backend as a result of successful CCDA creation, pursuant to the standards set for the certified modules including sending via Direct Edge Protocols, HL7 compliance, USCDI compliance depending on the certified modality used, and the manner used.
Include steps within the Real World Testing plan that establish milestones within the process. Include details on how and when the developer will implement measures and collect data. Key milestones should be relevant and directly related to expected outcomes discussed in the next section.
For each key milestone, describe when Real World Testing will begin in specific care settings and the date/timeframe during which data will be collected.
Key Milestone
Care Setting
Date/Timeframe
Identification of methodologies to evaluate planned data
Long Term Post-Acute Care
12/31/2021
Begin evaluation / data collection of metrics
Long Term Post-Acute Care
1/15/2022
Initial evaluation / analysis of data received as a check
Long Term Post-Acute Care
1/30/2022
Based upon initial data check (1/30/2022), begin monthly evaluation of metrics
Long Term Post-Acute Care
2/15/2022
Submission to ONC-ACB by determined date (TBD on ONC0-ACB Direction)
Long Term Post-Acute Care
~12/31/2022
The Real World Testing plan must include the following attestation signed by the health IT developer authorized representative.
Note: The plan must be approved by a health IT developer authorized representative capable of binding the health IT developer for execution of the plan and include the representative's contact information.ii
This Real World Testing plan is complete with all required elements, including measures that address all certification criteria and care settings. All information in this plan is up to date and fully addresses the health IT developer’s Real World Testing requirements.
Authorized Representative Name: Benjamin Britton
Authorized Representative Email: bbritton@ntst.com
Authorized Representative Phone: 970-658-6897
Authorized Representative Signature:
iCertified health IT continues to be compliant with the certification criteria, including the required technical standards and vocabulary codes sets; certified health IT is exchanging EHI in the care and practice settings for which it is marketed for use; and EHI is received by and used in the certified health IT. (85 FR 25766)