STS National Database Audit Policy

Purpose 

The purpose of an independent audit is two-fold. First, an audit assures stakeholders (Database participants, patients, researchers, consumers, regulators, payers, and industry) that the clinical data in the five STS National Database components—Adult Cardiac Surgery, Congenital Heart Surgery, General Thoracic Surgery, Intermacs, and Pedimacs—are consistent, accurate, and complete. Second, the audit process is designed as an educational opportunity to complement the participant’s internal quality controls established to maximize the integrity of the data collected. 

Policy

1. Participation agreements in the Adult Cardiac Surgery Database (section 1.2.1 or 1.3.1), Congenital Heart Surgery Database (section 1.2.1), General Thoracic Surgery Database (section 1.2.1), Intermacs Database (section 1.2.4) and Pedimacs Database (section 1.2.4), require participants to submit accurate and complete data and provide cooperation and assistance with any independent audit.  

“Participant hereby warrants that all data submitted for inclusion in the Database will be accurate and complete, and acknowledges that such data may be subject to independent audit in accordance with STS Policies. Participant will use its best efforts to address any data or related deficiencies identified by the independent data warehouse service provider, and agrees to cooperate with and assist STS and its designees in connection with the performance of any independent audit.”

2. STS responsibilities: 

(a) Contract with an external independent organization (audit company) to provide secure, HIPAA-compliant auditing services
(b) Notify sites selected for STS audit participation
(c) Provide education to the audit company regarding data definitions 
(d) Explain the audit process and expectations to sites
(e) Send summary results to sites
(f) Provide oversight in review and interpretation of the audit results, including identification of data quality outliers and provision of remediation assistance, as deemed appropriate by the associated workforce or task force.
(g) Provide access to Database-specific educational resources to audited sites, especially those with results that do not meet the expectation of STS (please see section 7). These resources include: Optional attendance to Advances in Quality & Outcomes: A Data Managers Meeting (AQO), Database-specific training manuals, educational webinars, STS National Database Data Manager Collaborative Program, National Database News, and the FAQ inbox. 

3. Participant responsibilities: 

(a) Cooperate with external audit organization to schedule the audit 
(b) Provide records (at no charge) or access to records in a timely fashion 
(c) Provide records in a HIPAA-compliant format
(d) Participate in post-audit review and educational opportunities

4. Each of the Database task forces is responsible for identifying audit variables.

5. Audit site selection:

(a) Each year approximately 10% of sites in each Database will be audited
(b) To be included in the audit pool, a site must a. be an active Database participant for all 12 months of the audited time period and b. capture the specified minimum number of procedures/admissions during the audit time period.

(c) All US and Canadian sites are included in the audit pool except:
     1.    Sites that have been identified as not meeting expectations during a recent audit (Please see section 7).
       2.    Sites that have successfully completed an STS audit within the past 3 years.
(d) STS reserves the right to utilize various methodologies, including random and targeted selection.  
      1.    Targeted audits may be scheduled if requested by a participant or deemed appropriate by STS.  

6. Audit summary results: 
A final Summary Report will be emailed by STS to each site. The rate of agreement and data completeness will be considered when evaluating each site and identifying sites that do not meet the STS expectations (please see section 7).  
    
7. Data quality
(a) Definitions:   
     1.    Data Accuracy (Data Agreement) – rate of agreement between data submitted by the participant and data in the medical record as adjudicated by the auditors 
      2.    Data Completeness/Missing Data – data fields not completed or, for certain critical variables such as those related to Operative Mortality, entered as Unknown
      3.    Data Completeness/Missing Records – discrepancy between the case log and cases entered into the Database
(b) Data accuracy will be calculated three ways:
     1.    Variable accuracy (agreement) will be measured for each audited variable. The denominator will equal the total number of records audited. The numerator will be the number of records where that variable was answered completely and correctly. This will be reported back as a percentage to sites. 
     2.    Section accuracy (agreement) will be measured for each section (i.e., demographics, risk factors, complications, discharge, mortality, etc.). The denominator will equal the total number of variables within the corresponding section where the variable was available or should have been available to abstract, times the total number of records abstracted. The numerator will equal the total number of variables within the specific section that were abstracted completely and correctly times the total number of records. 
                a.    For example, if the demographics section is audited on 20 records and there are three variables within the demographic section and an answer was expected for each variable on all records, then the denominator will be 60. If the site had one abstraction error within the demographic section for each record for a total of 20 total errors, then the result would be 20/60 or 66.6% accuracy for the demographic section.
     3.    Overall site accuracy (agreement) will be measured for each site as an overall agreement rate. The denominator will equal the total number of records audited times the total number of variables that were available or should have been available for abstraction of each record. The numerator will equal the total number of variables that were abstracted completely and correctly times the total number of records audited at the site.
     4.    Each agreement rate (variable, section, and overall) will also be calculated using the entire cohort to achieve the overall STS Database-specific rates.  
(c) For Congenital, the rate of agreement and rate of completeness are calculated using the following methods. 
      1.    Rate of Agreement = (Match + Acceptable Variance) / (Match + Acceptable Variance + Unacceptable Variance)
       2.    Rate of Completeness = (Match + Acceptable Variance + Unacceptable Variance) / (Match + Acceptable Variance + Unacceptable Variance + Missing Data) 
                           i.    Acceptable variance is defined as differing nomenclature among providers. See Appendix A for the formal definition of Acceptable Variance: Primary Diagnosis and Primary Procedure. 
(d) For all sections (excluding the Complications or Mortality variables listed in Appendix B, C and D) and overall site accuracy (agreement) rates, the following grading system will be provided to sites.
1.    98.0% or greater is defined as a site that exceeds expectations. 
2.    90.0% to 97.9% is defined as a site that meets expectations. 
3.    89.9% or less is defined as a site that does not meet expectations.
(e) For the Complications section, the following grading system will be provided to sites. See Appendix B for a breakdown of the complications section per Database. 
1.    98.0% or greater is defined as a site that meets expectations.
2.    90.0 to 97.9% is defined as a site that does not meet expectations.
3.    A site achieving 89.9% or less for accuracy on the Complications section will require a re-audit.
(f) For the Mortality section, the following grading system will be provided to sites. See Appendix C for a breakdown of the mortality section per Database. 
1.    98.0% or greater is defined as a site that meets expectations.
2.    90.0 to 97.9% is defined as a site that does not meet expectations.
3.    A site achieving 89.9% or less for accuracy on the Mortality section will require a re-audit.
(g) For the Mortality section, verification of post-procedure status is required to be submitted to the audit company. This includes any tracking logs, EHR, or other documentation to support coding of those variables listed in Appendix D. 
1.    100.0% accuracy is defined as meeting expectations.
2.    99.9% or less for accuracy is defined as not meeting expectations and will require re-audit.
(h) Sites identified as not meeting expectations for overall site accuracy (89.9%) or sites having 89.9% or less for accuracy on the Complications section and Mortality section or 99.9% or less on the Mortality verification of post-procedure status variables will be required to undergo a follow-up audit within one to two years.
(i) Sites identified as continuously not meeting expectations (with a lower-than 90.0% agreement rate for the Complications and Mortality sections, or lower-than 100.0% for the Mortality verification of post-procedure status variables) following the second audit jeopardize their participation in the STS National Database. A participant will not receive a star rating (except in the instance of CHSD participants, who do not receive star ratings), will not be included in the aggregate data, and will not receive a risk-adjusted report of data. The site will also be denied the option to publicly reporting their data. 

Appendix A

Definition of Acceptable Variance: Primary Diagnosis and Primary Procedure

1. Surgeon auditor deems the participant’s choice of primary diagnosis or procedure to be materially interchangeable with that of the auditor (primum ASD vs. partial AVSD).
2. Surgeon auditor’s choice for primary diagnosis or procedure is on the list of all diagnoses or procedures submitted by the participant.
3. Participant chooses to submit an “atomic” vs. “molecular” procedural description.

(a) Algorithmically generated “molecular” procedure is on the “Congenital Combo Code” list.
(b) “Atomic vs. “molecular” procedural description does not impact the STAT score of the Index Procedure.

Appendix B

See below the audited variables that are included in the complications section (Adult Cardiac, General Thoracic, and Congenital Heart) and adverse events section (Intermacs and Pedimacs).

Appendix C

See below the audited variables that are included in the mortality section (Adult Cardiac, General Thoracic, and Congenital Heart) and the follow-up section (Intermacs and Pedimacs).

Appendix D

See below the post-procedure verification variables for each Database.