[Skip to Navigation]
Sign In
Figure 1.  Unadjusted Outcome Rates Among All Hospitalized Adults and Hospitalized Adults Aged 65 Years or Older
Unadjusted Outcome Rates Among All Hospitalized Adults and Hospitalized Adults Aged 65 Years or Older

The trends for unadjusted outcomes were either not statistically significant or, if significant, trivial (eg, the trend for 30-day postdischarge mortality in patients aged 65 years or older was −0.002% [P = .007]). Relative constancy of unadjusted rates was similar for various patient subsets (eg, those with community-acquired pneumonia; see eAppendix 2 in the Supplement for these additional figures). The study period comprised the baseline period (June 2010 to May 2011), beginning of Hospital Readmissions Reduction Program penalty phase (October 2012), and end of study period (December 2017). This latter period is split into 2 periods based on the study by Gupta et al4 (October 2012 to December 2014).

Figure 2.  Changes in Hospitalization Type and Hospitalization Rate Over Time
Changes in Hospitalization Type and Hospitalization Rate Over Time

The study period comprised the baseline period (June 2010 to May 2011), beginning of Hospital Readmissions Reduction Program penalty phase (October 2012), and end of study period (December 2017). This latter period is split into 2 periods based on the study by Gupta et al4 (October 2012 to December 2014). CMS 2 MN indicates the promulgation date of the Centers for Medicare & Medicaid Services’ 2-midnight rule; Inp ≥24 h/<24 h, 24 hours or more/less inpatient length of stay; KFHP, Kaiser Foundation Health Plan; Obs, observation; and Pub, meeting public reporting specifications.

Figure 3.  Adjusted and Unadjusted Rates of 30-Day Nonelective Rehospitalization and 30-Day Postdischarge Mortality
Adjusted and Unadjusted Rates of 30-Day Nonelective Rehospitalization and 30-Day Postdischarge Mortality

By the final month of the study, the observed to expected ratio for 30-day nonelective rehospitalization was 0.90 (95% CI, 0.85-0.95) and for 30-day postdischarge mortality was 0.87 (95% CI, 0.83-0.92). eAppendix 4 in the Supplement provides graphics for inpatient mortality (final observed to expected ratio, 0.79; 95% CI, 0.73-0.84), 30-day mortality (0.86; 95% CI, 0.82-0.89), and 30-day composite outcome (nonelective rehospitalization or death within 30 days of discharge, 0.90; 95% CI, 0.86-0.94). The study period comprised the baseline period (June 2010 to May 2011), beginning of Hospital Readmissions Reduction Program (HRRP) penalty phase (October 2012), and end of study period (December 2017). This latter period is split into 2 periods based on the study by Gupta et al4 (October 2012 to December 2014).

Table 1.  Cohort Characteristicsa
Cohort Characteristicsa
Table 2.  Association of Denominator Definition With Outcome Capture
Association of Denominator Definition With Outcome Capture
1.
Jencks  SF, Williams  MV, Coleman  EA.  Rehospitalizations among patients in the Medicare fee-for-service program.  N Engl J Med. 2009;360(14):1418-1428. doi:10.1056/NEJMsa0803563PubMedGoogle ScholarCrossref
2.
Centers for Medicare & Medicaid Services. Hospital Readmissions Reduction Program (HRRP). https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/AcuteInpatientPPS/Readmissions-Reduction-Program.html. Accessed March 27, 2019.
3.
Centers for Medicare & Medicaid Services. Health insurance exchange quality ratings system 101. https://www.cms.gov/newsroom/fact-sheets/health-insurance-exchange-quality-ratings-system-101. Published August 15, 2019. Accessed September 23, 2019.
4.
Gupta  A, Allen  LA, Bhatt  DL,  et al.  Association of the hospital readmissions reduction program implementation with readmission and mortality outcomes in heart failure.  JAMA Cardiol. 2018;3(1):44-53. doi:10.1001/jamacardio.2017.4265PubMedGoogle ScholarCrossref
5.
Khera  R, Dharmarajan  K, Wang  Y,  et al.  Association of the Hospital Readmissions Reduction Program with mortality during and after hospitalization for acute myocardial infarction, heart failure, and pneumonia.  JAMA Netw Open. 2018;1(5):e182777-e182777. doi:10.1001/jamanetworkopen.2018.2777PubMedGoogle ScholarCrossref
6.
Centers for Medicare & Medicaid Services. Fact sheet: two-midnight rule. https://www.cms.gov/newsroom/fact-sheets/fact-sheet-two-midnight-rule-0. Published October 30, 2015. Accessed April 17, 2019.
7.
Society of Hospital Medicine Public Policy Committee. The hospital observation care problem: perspectives and solutions from the Society of Hospital Medicine. https://www.hospitalmedicine.org/globalassets/policy-and-advocacy/advocacy-pdf/shms-observation-white-paper-2017. Published September 2017. Accessed October 30, 2017.
8.
Nuckols  TK, Fingar  KR, Barrett  M, Steiner  CA, Stocks  C, Owens  PL.  The shifting landscape in utilization of inpatient, observation, and emergency department services across payers.  J Hosp Med. 2017;12(6):443-446. doi:10.12788/jhm.2751PubMedGoogle ScholarCrossref
9.
Sabbatini  AK, Hsia  RY.  It’s time for a strategic approach to observation care.  J Hosp Med. 2017;12(6):479-480. doi:10.12788/jhm.2760PubMedGoogle ScholarCrossref
10.
Sabbatini  AK, Wright  B, Hall  MK, Basu  A.  The cost of observation care for commercially insured patients visiting the emergency department.  Am J Emerg Med. 2018;36(9):1591-1596. doi:10.1016/j.ajem.2018.01.040PubMedGoogle ScholarCrossref
11.
Sabbatini  AK, Wright  B.  Excluding observation stays from readmission rates—what quality measures are missing.  N Engl J Med. 2018;378(22):2062-2065. doi:10.1056/NEJMp1800732PubMedGoogle ScholarCrossref
12.
Hayward  RA.  Access to clinically-detailed patient information: a fundamental element for improving the efficiency and quality of healthcare.  Med Care. 2008;46(3):229-231. doi:10.1097/MLR.0b013e318167579cPubMedGoogle ScholarCrossref
13.
Roberts  ET, Zaslavsky  AM, Barnett  ML, Landon  BE, Ding  L, McWilliams  JM.  Assessment of the effect of adjustment for patient characteristics on hospital readmission rates: implications for pay for performance.  JAMA Intern Med. 2018;178(11):1498-1507. doi:10.1001/jamainternmed.2018.4481PubMedGoogle ScholarCrossref
14.
Epstein  AM, Jha  AK, Orav  EJ.  The relationship between hospital admission rates and rehospitalizations.  N Engl J Med. 2011;365(24):2287-2295. doi:10.1056/NEJMsa1101942PubMedGoogle ScholarCrossref
15.
Dharmarajan  K, Qin  L, Lin  Z,  et al.  Declining admission rates and thirty-day readmission rates positively associated even though patients grew sicker over time.  Health Aff (Millwood). 2016;35(7):1294-1302. doi:10.1377/hlthaff.2015.1614PubMedGoogle ScholarCrossref
16.
Escobar  GJ, Greene  JD, Scheirer  P, Gardner  MN, Draper  D, Kipnis  P.  Risk-adjusting hospital inpatient mortality using automated inpatient, outpatient, and laboratory databases.  Med Care. 2008;46(3):232-239. doi:10.1097/MLR.0b013e3181589bb6PubMedGoogle ScholarCrossref
17.
Escobar  GJ, Greene  JD, Gardner  MN, Marelich  GP, Quick  B, Kipnis  P.  Intra-hospital transfers to a higher level of care: contribution to total hospital and intensive care unit (ICU) mortality and length of stay (LOS).  J Hosp Med. 2011;6(2):74-80. doi:10.1002/jhm.817PubMedGoogle ScholarCrossref
18.
Liu  V, Kipnis  P, Rizk  NW, Escobar  GJ.  Adverse outcomes associated with delayed intensive care unit transfers in an integrated healthcare system.  J Hosp Med. 2012;7(3):224-230. doi:10.1002/jhm.964PubMedGoogle ScholarCrossref
19.
Escobar  GJ, Gardner  MN, Greene  JD, Draper  D, Kipnis  P.  Risk-adjusting hospital mortality using a comprehensive electronic record in an integrated health care delivery system.  Med Care. 2013;51(5):446-453. doi:10.1097/MLR.0b013e3182881c8ePubMedGoogle ScholarCrossref
20.
Escobar  GJ, Ragins  A, Scheirer  P, Liu  V, Robles  J, Kipnis  P.  Nonelective rehospitalizations and postdischarge mortality: predictive models suitable for use in real time.  Med Care. 2015;53(11):916-923. doi:10.1097/MLR.0000000000000435PubMedGoogle ScholarCrossref
21.
Agency for Healthcare Research and Quality. Quality Indicators Software Instructions, SAS Version 4.5. https://www.qualityindicators.ahrq.gov/Downloads/Software/SAS/V45/Software_Instructions_SAS_V4.5.pdf. Published May 2013. Accessed December 5, 2014.
22.
National Committee for Quality Assurance.  Technical Specifications. Vol 2. Washington, DC: National Committee for Quality Assurance; 2018.
23.
National Committee for Quality Assurance. HEDIS®1 2018 volume 2: technical update, 2019. https://www.ncqa.org/wp-content/uploads/2018/10/HEDIS-2019-Volume-2-Technical-Update.pdf. Accessed March 27, 2019.
24.
Escobar  GJ, Dellinger  RP.  Early detection, prevention, and mitigation of critical illness outside intensive care settings.  J Hosp Med. 2016;11(suppl 1):S5-S10. doi:10.1002/jhm.2653PubMedGoogle ScholarCrossref
25.
Escobar  GJ, Turk  BJ, Ragins  A,  et al.  Piloting electronic medical record-based early detection of inpatient deterioration in community hospitals.  J Hosp Med. 2016;11(suppl 1):S18-S24. doi:10.1002/jhm.2652PubMedGoogle ScholarCrossref
26.
Kipnis  P, Turk  BJ, Wulf  DA,  et al.  Development and validation of an electronic medical record-based alert score for detection of inpatient deterioration outside the ICU.  J Biomed Inform. 2016;64:10-19. doi:10.1016/j.jbi.2016.09.013PubMedGoogle ScholarCrossref
27.
Deyo  RA, Cherkin  DC, Ciol  MA.  Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases.  J Clin Epidemiol. 1992;45(6):613-619. doi:10.1016/0895-4356(92)90133-8PubMedGoogle ScholarCrossref
28.
Ladiray  D, Quenneville  B. Outline of the X-11 method. In:  Seasonal Adjustment with the X-11 Method: Lecture Notes in Statistics. Vol 158. New York, NY: Springer; 2001. doi:10.1007/978-1-4613-0175-2_3
29.
Box  GEP, Jenkins  GM, Reinsel  GC, Ljung  GM.  Time Series Analysis: Forecasting and Control. 5th ed. Hoboken, NJ: Wiley; 2015.
30.
Churpek  MM, Yuen  TC, Winslow  C,  et al.  Multicenter development and validation of a risk stratification tool for ward patients.  Am J Respir Crit Care Med. 2014;190(6):649-655. doi:10.1164/rccm.201406-1022OCPubMedGoogle ScholarCrossref
31.
Rothman  MJ, Rothman  SI, Beals  J  IV.  Development and validation of a continuous measure of patient condition using the electronic medical record.  J Biomed Inform. 2013;46(5):837-848. doi:10.1016/j.jbi.2013.06.011PubMedGoogle ScholarCrossref
32.
Venkatesh  AK, Wang  C, Ross  JS,  et al.  Hospital use of observation stays: cross-sectional study of the impact on readmission rates.  Med Care. 2016;54(12):1070-1077. doi:10.1097/MLR.0000000000000601PubMedGoogle ScholarCrossref
33.
Holloway  RG, Quill  TE.  Mortality as a measure of quality: implications for palliative and end-of-life care.  JAMA. 2007;298(7):802-804. doi:10.1001/jama.298.7.802PubMedGoogle ScholarCrossref
34.
Joynt Maddox  KE, Orav  EJ, Zheng  J, Epstein  AM.  Evaluation of Medicare’s bundled payments initiative for medical conditions.  N Engl J Med. 2018;379(3):260-269. doi:10.1056/NEJMsa1801569PubMedGoogle ScholarCrossref
35.
Wright  B, O’Shea  AM, Ayyagari  P, Ugwi  PG, Kaboli  P, Vaughan Sarrazin  M.  Observation rates at veterans’ hospitals more than doubled during 2005–13, similar to Medicare trends.  Health Aff (Millwood). 2015;34(10):1730-1737. doi:10.1377/hlthaff.2014.1474PubMedGoogle ScholarCrossref
36.
Lind  KD, Noel-Miller  CM, Sangaralingham  LR,  et al.  Increasing trends in the use of hospital observation services for older Medicare Advantage and privately insured patients.  Med Care Res Rev. 2019;76(2):229-239. doi:10.1177/1077558717718026PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    Health Policy
    December 4, 2019

    Tasas de rehospitalización de varios años y desenlaces clínicos hospitalarios en un sistema integrado de atención médica

    Author Affiliations
    • 1Systems Research Initiative, Kaiser Permanente Division of Research, Oakland, California
    • 2Intensive Care Unit, Kaiser Permanente Medical Center, Santa Clara, California
    • 3TPMG Consulting Services, Oakland, California
    JAMA Netw Open. 2019;2(12):e1916769. doi:10.1001/jamanetworkopen.2019.16769
    Puntos claveEnglish 中文 (chinese)

    Pregunta  ¿Cómo cambiaron los desenlaces clínicos de hospitalización, incluida la rehospitalización, en un sistema integrado de atención médica entre 2010 y 2017?

    Conclusiones  En este estudio de cohortes de 1 384 025 hospitalizaciones entre 679 831 pacientes en un sistema integrado de atención médica, la tasa de hospitalización, las tasas ajustadas de mortalidad hospitalaria (de 30 días y 30 días después del alta) y las tasas de rehospitalización no optativa disminuyeron a pesar del empeoramiento de la mezcla de casos. La proporción de hospitalizaciones sujetas a informes públicos también disminuyó, principalmente debido a los aumentos en las hospitalizaciones solo para observación.

    Significado  Las conclusiones de este estudio sugieren que las tasas de hospitalizaciones, rehospitalizaciones y mortalidad pueden reducirse simultáneamente, pero es poco probable que una sola medida describa con precisión estos cambios.

    Abstract

    Importance  Since the introduction of the rehospitalization rate as a quality measure, multiple changes have taken place in the US health care delivery system. Interpreting rehospitalization rates without taking a global view of these changes and new data elements from comprehensive electronic medical records yields a limited assessment of the quality of care.

    Objective  To examine hospitalization outcomes from a broad perspective, including the implications of numerator and denominator definitions, all adult patients with all diagnoses, and detailed clinical data.

    Design, Setting, and Participants  This cohort study obtained data from 21 hospitals in Kaiser Permanente Northern California (KPNC), an integrated health care delivery system that serves patients with Medicare Advantage plans, Medicaid, and/or Kaiser Foundation Health Plan. The KPNC electronic medical record system was used to capture hospitalization data for adult patients who were 18 years of age or older; discharged from June 1, 2010, through December 31, 2017; and hospitalized for reasons other than childbirth. Hospital stays for transferred patients were linked using public and internal sources.

    Exposures  Hospitalization type (inpatient, for observation only), comorbidity burden, acute physiology score, and care directives.

    Main Outcomes and Measures  Mortality (inpatient, 30-day, and 30-day postdischarge), nonelective rehospitalization, and discharge disposition (home, home with home health assistance, regular skilled nursing facility, or custodial skilled nursing facility).

    Results  In total, 1 384 025 hospitalizations were identified, of which 1 155 034 (83.5%) were inpatient and 228 991 (16.5%) were for observation only. These hospitalizations involved 679 831 patients (mean [SD] age, 61.4 [18.1] years; 362 582 female [53.3%]). The number of for-observation-only hospitalizations increased from 16 497 (9.4%) in the first year of the study to 120 215 (20.5%) in the last period of the study, whereas inpatient hospitalizations with length of stay less than 24 hours decreased by 33% (from 12 008 [6.9%] to 27 108 [4.6%]). Illness burden measured using administrative data or acute physiology score increased significantly. The proportion of patients with a Comorbidity Point Score of 65 or higher increased from 20.5% (range across hospitals, 18.4%-26.4%) to 28.8% (range, 22.3%-33.0%), as did the proportion with a Charlson Comorbidity Index score of 4 or higher, which increased from 28.8% (range, 24.6%-35.0%) to 38.4% (range, 31.9%-43.4%). The proportion of patients at or near critical illness (Laboratory-based Acute Physiology Score [LAPS2] ≥110) increased by 21.4% (10.3% [range across hospitals, 7.4%-14.7%] to 12.5% [range across hospitals, 8.3%-16.6%]; P < .001), reflecting a steady increase of 0.07 (95% CI, 0.04-0.10) LAPS2 points per month. Unadjusted inpatient mortality in the first year of the study was 2.78% and in the last year was 2.71%; the corresponding numbers for 30-day mortality were 5.88% and 6.15%, for 30-day postdischarge mortality were 3.94% and 4.22%, and for nonelective rehospitalization were 12.00% and 12.81%, respectively. All outcomes improved after risk adjustment. Compared with the first month, the final observed to expected ratio was 0.79 (95% CI, 0.73-0.84) for inpatient mortality, 0.86 (95% CI, 0.82-0.89) for 30-day mortality, 0.90 (95% CI, 0.85-0.95) for 30-day nonelective rehospitalization, and 0.87 (95% CI, 0.83-0.92) for 30-day postdischarge mortality. The proportion of nonelective rehospitalizations meeting public reporting criteria decreased substantially over the study period (from 58.0% in 2010-2011 to 45.2% in 2017); most of this decrease was associated with the exclusion of observation stays.

    Conclusions and Relevance  This study found that in this integrated system, the hospitalization rate decreased and risk-adjusted hospital outcomes improved steadily over the 7.5-year study period despite worsening case mix. The comprehensive results suggest that future assessments of care quality should consider the implications of numerator and denominator definitions, display multiple metrics concurrently, and include all hospitalization types and detailed data.

    Introduction

    After the 2009 publication of the article by Jencks et al,1 the Centers for Medicare & Medicaid Services (CMS) introduced the rehospitalization metric as a publicly reported quality measure, which was followed by the Hospital Readmissions Reduction Program (HRRP).2 The HRRP imposed penalties (beginning on October 1, 2012) to create incentives for improvement in care coordination among fee-for-service Medicare members. Among health systems that provide care for Medicare Advantage members, the HRRP sanctions play a smaller role. However, health plans have strong incentives to decrease readmissions as part of the CMS Five-Star Quality Rating System.3 Hospitals with excess readmissions can experience a decrease in their rating that is, in turn, associated with substantial reductions in reimbursement and their ability to enroll new members. The HRRP and the Five-Star Quality Rating System were instituted at the same time. Consequently, considerable research and regulations have focused on rehospitalization for community-acquired pneumonia, acute myocardial infarction, and congestive heart failure, the 3 original targeted conditions that remain the program’s centerpiece.

    The consensus in the United States is that rehospitalization rates have decreased, which has raised 3 major quality concerns. First, are decreases in 30-day rehospitalization rates accompanied by increases in inpatient, 30-day, and/or 30-day postdischarge mortality rates?4,5 Second, is the exclusion of patients admitted for observation only (a CMS mandate, whose value has been questioned,6,7 based on a physician’s assessment to keep a patient in the hospital for 2 midnights) falsely lowering the true rehospitalization rates and precluding adequate quality assessment?8-11 Third, in an era in which the hospitals and health care systems in the United States have transitioned to electronic medical records (EMRs) that can capture detailed clinical data, why does public reporting still rely on administrative data, despite calls for better risk adjustment?12,13

    One issue that has received limited attention is the population hospitalization rate. Using 2008 Medicare data on patients with community-acquired pneumonia, acute myocardial infarction, and congestive heart failure, Epstein et al14 highlighted the strong association between population hospitalization rates and rehospitalization rates. Subsequently, using Medicare data from 2008 to 2013, Dharmarajan et al15 showed a parallel reduction in both hospitalization and 30-day rehospitalization rates despite patients’ worsening comorbidity burden. This decrease was not associated with increases in a composite outcome (nonelective rehospitalization or death within 30 days of discharge).

    A potential problem with the current literature on rehospitalization is that it can lead to a fragmented view, given that each study addresses discrete issues. In this cohort study, we took a broad perspective to examine hospitalization and rehospitalization. Using clinically intuitive definitions, we considered how multiple factors described in recent literature have played out in 1 integrated health care delivery system. We sought to identify and understand several factors associated with the rehospitalization metric, including outcome definitions, illness burden measures, and criteria to define denominators and numerators. Recent literature suggests that several of these factors could be trending in different directions, making it difficult to discern the association between the rehospitalization metric and actual quality of care.

    Using detailed clinical data from a contemporary cohort in a health system that maximizes the use of EMRs, we expanded on recent research by examining outcomes in all adults. We evaluated the changes in hospitalization and metric definitions in the period before and after the implementation of the HRRP penalty phase. We reported on both rehospitalization and survival before and after discharge and considered illness burden (including acute physiology, not just comorbidities), the population hospitalization rate, discharge disposition, and changes in these factors over time. The study setting was Kaiser Permanente Northern California (KPNC), an integrated health care delivery system with 21 hospitals that serve patients with Medicare Advantage, Medicaid, and Kaiser Foundation Health Plans. Robust KPNC database systems provided us with an opportunity to examine hospitalization outcomes more broadly and to quantify temporal trends for factors that have not been extensively described in the existing literature.

    Methods

    This cohort study was approved by the KPNC Institutional Review Board for the Protection of Human Subjects, which waived the requirement for informed consent from participants given that this study involved only data. The study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline.

    Under a mutual exclusivity agreement, 9500 salaried physicians of The Permanente Medical Group provide care for 4.3 million members of the Kaiser Foundation Health Plan (KFHP) at facilities owned by Kaiser Foundation Hospitals. Deployment of a systemwide EMR (Epic) was completed in 2010.

    Within KPNC, a 21-hospital system that has been described elsewhere,16-20 and using its EMR system, we identified hospital stays for patients who met these criteria: hospital discharge from June 1, 2010, through December 31, 2017 (excluding nonovernight 1-day surgical procedures); age 18 years or older at hospitalization; and hospitalization that was not for childbirth (stays for postdelivery complications were included). We linked hospital stays for transferred patients and identified all deaths using public and internal sources.16-20 We assigned inpatient and 30-day mortality rates to the admitting hospital and 30-day rehospitalization and mortality rates to the discharging hospital. Using the data systems and study methods, we classified all hospitalizations as an index hospitalization, a rehospitalization, or both.

    Given KPNC practice standards that discourage direct admission to the hospital from the outpatient clinic setting, we considered rehospitalizations as nonelective if they began in the emergency department, if the principal diagnosis was an ambulatory care–sensitive condition,21 or if they began in an outpatient clinic and the patient had elevated severity of illness (mortality risk of 7.2% based on acute physiology score alone, described later).20 Hospitalizations that did not meet the nonelective criteria were also considered elective. We examined hospitalized patients’ KFHP membership data to ascertain whether a discharge satisfied the criteria of the National Committee for Quality Assurance’s Healthcare Effectiveness Data and Information Set (HEDIS). These HEDIS criteria stipulate continuous health plan membership in the 12 months before and the 30 days after hospital discharge, with a maximum gap in coverage of 45 days in the preceding 12 months.22,23 In addition, we captured whether a hospitalization was inpatient or for observation only. If a hospitalization began as for observation only but transitioned to inpatient status, we classified it as inpatient. To calculate hospitalization rates among KFHP members, we obtained monthly membership counts from internal reports.

    The dependent variables were hospitalization, inpatient mortality, 30-day mortality, nonelective rehospitalization within 30 days of hospital discharge, death within 30 days of hospital discharge, and a composite outcome (nonelective rehospitalization and/or death within 30 days of hospital discharge).

    We also captured KFHP membership, long-term comorbidity burden, severity of illness, and code status.19 At KPNC, each month, all adults with a medical record number are assigned a Comorbidity Point Score, version 2 (COPS2), which is based on CMS Hierarchical Condition Categories (score range, 0-1014 [scores above 300 are rare], with higher scores indicating increased mortality risk). Based on the experience with KPNC’s early warning system, now operational in all 21 hospitals,24-26 inpatients with a COPS2 of 65 or higher are evaluated by palliative care teams (a score above this threshold is associated with a high risk of in-hospital deterioration). For comparison, we assigned each hospitalization a Charlson Comorbidity Index score (range, 0-24 [scores above 5 are uncommon], with higher scores indicating increased mortality risk), using the methods of Deyo et al.27 At KPNC, patients are assigned a Laboratory-based Acute Physiology Score, version 2 (LAPS2; range: 0-414 [scores above 200 are uncommon] on admission and every hour after hospitalization, with higher scores indicating worsening instability), including a score assigned at 0800 on the discharge day (LAPS2dc). For example, in July 2018, the median hourly LAPS2 among all patients in the intensive care unit was 110, whereas the median ward score was 52. It is not possible to admit a patient to KPNC hospitals without specifying code status, which can be subsequently updated. We classified each patient’s care directive as full code or not (which included partial code, do not resuscitate, and comfort care only).19

    We also captured age at hospitalization, sex, hospitalization venue (emergency department or not), total index hospital length of stay (LOS), whether a patient experienced any overnight hospitalization in the first 7 days and separately in the 8 to 30 days before the index hospitalization,20 discharge disposition (home, regular or custodial skilled nursing facility [SNF], and home health services), and referral to hospice. We combined the Healthcare Cost Utilization Project’s single-level Clinical Classification Software diagnosis categories to categorize all International Classification of Diseases, Ninth Revision and International Statistical Classification of Diseases and Related Health Problems, Tenth Revision (ICD-10), hospitalization or principal diagnosis codes into 30 groups called Primary Conditions.19,20

    We used 3 cohort definitions: all (ALL), inpatient (INP), and subject to public reporting (PUB). The ALL cohort included all hospitalizations or discharges in the denominator. The INP cohort included only inpatient hospitalizations in the denominator. The PUB cohort approximated the HEDIS construct used in the CMS Five-Star Quality Rating System,3 which does not include mortality as an outcome. The PUB cohort included only discharges in which patients met the HEDIS membership requirements and was restricted to inpatient hospitalizations in both the numerator and denominator. Additionally, the PUB cohort excluded inpatient stays of less than 24 hours and for-observation-only hospitalizations, inpatient rehospitalizations without a preceding inpatient hospitalization within 30 days (ie, rehospitalizations following for-observation-only hospitalization), and inpatient and postdischarge deaths unless they occurred as part of an eligible inpatient rehospitalization.

    Statistical Analysis

    For all risk-adjusted analyses, we developed statistical models using data from the first year (June 1, 2010, to May 31, 2011) as a baseline. We used logistic regression with variables based on models that were developed and tested earlier and described in previous reports16,19,20: age, sex, COPS2, admission LAPS2, index discharge LOS, previous hospitalizations, discharge code status, diagnosis (Primary Condition), and KFHP membership status at the time of discharge. For the first 2 cohorts (ALL and INP), we developed risk adjustment models for all outcomes. For the PUB cohort, for which mortality was not a reported outcome, we only modeled rehospitalization. We used these models to estimate the probability of observing the outcome (expected) and the observed to expected ratios for multiple subgroups of patients and across time.

    For the graphical displays of patient outcomes and observed to expected ratios, we used a 3-month moving average technique and estimated the monthly observed and expected outcomes as a mean of the past, present, and future months (thus, the May 2015 data point was based on index hospitalizations from April 1, 2015, through June 30, 2015). We deseasonalized moving averages using the X-11 procedure.28 We assessed monthly trends through regression models with an autocorrelated error structure29 and differences in proportions with the χ2 test using 2-tailed, unpaired tests. Two-sided P < .05 was considered statistically significant. All statistical calculations and plots were performed with SAS, version 9.4 (SAS Institute Inc).

    For reference purposes in the graphical displays, we divided the period before the implementation of the HRRP penalties into 2 epochs: (1) reference year for multivariate analyses (June 1, 2010, to May 31, 2011) and (2) time until the beginning of the HRRP penalty phase (June 1, 2011, to September 30, 2012). We divided the period after the implementation of HRRP penalties into 2 epochs: (1) period that matches the dates used by Gupta et al4 (October 1, 2012, to December 31, 2014) and (2) the remaining years (January 1, 2015, to December 31, 2017).

    Results

    We identified 1 384 025 hospitalizations among 679 831 patients (mean [SD] age, 61.4 [18.1] years; 362 582 female [53.3%]). Of the total hospitalizations, 1 155 034 (83.5%) were inpatient and 228 991 (16.5%) were for observation only. Table 1 summarizes cohort characteristics before and after the implementation of the HRRP statutory financial penalties. The number of for-observation-only hospitalizations increased from 16 497 (9.4%) in the first year of the study (June 1, 2010, to May 31, 2011) to 120 215 (20.5%) in the last study period after penalty implementation (January 1, 2015, to December 31, 2017), whereas inpatient hospitalizations with LOS of less than 24 hours decreased by 33% (from 12 008 [6.9%] to 27 108 [4.6%]) between these periods. These changes were more pronounced among patients with acute myocardial infarction, community-acquired pneumonia, and congestive heart failure, in whom the absolute numbers also increased (eAppendix 1 in the Supplement). The proportion of hospitalizations in which patients met the strict HEDIS membership definition decreased slightly (81.8% to 78.9%). In contrast, the proportion of hospitalizations that met the public reporting definition (which excluded for-observation-only and short inpatient LOS) decreased by 14.5% from 68.6% (range across hospitals, 59.7%-76.2%) to 58.7% (range across hospitals, 40.0%-68.9%).

    The overall comorbidity burden, as measured by the COPS2, increased steadily by 0.16 (95% CI, 0.13-0.20) points per month. Table 1 shows that the proportion of patients with a COPS2 of 65 or higher increased from 20.5% (range across hospitals, 18.4%-26.4%) to 28.8% (range across hospitals, 22.3%-33.0%; P < .001), as did the proportion with a Charlson Comorbidity Index score of 4 or higher, which increased from 28.8% (range across hospitals, 24.6%-35.0%) to 38.4% (range across hospitals, 31.9%-43.4%; P < .001). Acute physiology scores also increased steadily each month, as did the proportion of patients at or near critical illness (LAPS2 ≥110), which increased by 21.4% (from 10.3% [range across hospitals, 7.4%-14.7%] to 12.5% [range across hospitals, 8.3%-16.6%], reflecting a steady increase of 0.07 [95% CI, 0.04-0.10] LAPS2 points per month; P < .001), although this increase was not statistically significant among patients with pneumonia (where the proportion fell from 19.1% to 17.5%; P = .10) or congestive heart failure (20.0% to 19.8%; P = .23) (eAppendix 1 in the Supplement). The proportion of patients with acute myocardial infarction at or near critical illness decreased from 14.5% (range across hospitals, 5.4%-24.0%) to 12.6% (range across hospitals, 8.0%-37.5%; P < .001) (eAppendix 1 in the Supplement).

    Despite increased acuity and comorbidity, hospital LOS remained stable, as did the proportion of patients discharged alive as full code. The proportion of patients discharged with home health services increased from 12.3% (range across hospitals, 7.1%-18.5%) to 17.8% (range across hospitals, 6.4%-31.1%]). Discharges to other venues decreased: regular discharge home, from 75.9% (range across hospitals, 71.6%-85.0%) to 71.8% (range across hospitals, 53.1%-87.5%); regular SNF, from 10.2% (range across hospitals, 6.8%-15.9%) to 9.3% (range across hospitals, 5.3%-13.6%); and custodial SNF, from 1.6% (range across hospitals, 0.8%-2.6%) to 1.2% (range across hospitals, 0.5%-2.3%). Table 1 also shows that, with the exception of nonelective rehospitalizations that were for observation only (which more than doubled from 1.2% [range across hospitals, 0.7%-1.8%] to 2.9% [range across hospitals, 1.7%-5.3%]), unadjusted outcomes remained constant over the study period.

    Figure 1 shows unadjusted rates for 5 outcomes among all hospitalizations (left panel) and hospitalizations for patients aged 65 years or older (right panel). Among all patients, inpatient mortality (first year, 2.78%; last year, 2.71%), 30-day mortality (first year, 5.88%; last year, 6.15%), 30-day postdischarge mortality (first year, 3.94%; last year, 4.22%), nonelective rehospitalization (first year, 12.00%; last year, 12.81%), and composite outcome (first year, 14.77%; last year, 15.76%) rates remained relatively constant over the study period. Among patients aged 65 years or older, inpatient mortality (first year, 4.05%; last year, 3.69), 30-day mortality (first year, 9.00%, last year, 8.84%); 30-day postdischarge mortality (first year, 6.25%, last year, 6.32%); nonelective rehospitalization (first year, 14.23%, last year, 14.64%), and composite outcome (first year, 18.71%, last year, 19.13%) rates also remained relatively constant over the study period.

    Figure 2 displays time trends for hospitalization type and hospitalization rate. The proportion of patients with short (which went from 6.9% in the first year to 6.1% in the last year) or observation (which went from 9.4% in the first year to 20.4% in the last year) hospitalizations increased (left panel), but this increase was not associated with increased hospitalization rates (right panel). These results were similar among patients of all ages (eAppendix 3 in the Supplement). The population hospitalization rate decreased by 13% among all adults (from 5.5 to 4.8 discharges per thousand KFHP members; P < .001), by 20% (from 3.0 to 2.4 per thousand KFHP members; P < .001) for patients younger than 65 years, and by 13% (17.2 to 14.9 per thousand KFHP members; P < .001) for patients aged 65 years or older.

    Figure 3 contrasts the unadjusted rate with the observed to expected ratio for nonelective rehospitalization (left panel) and for 30-day postdischarge mortality (right panel). Compared with the first year, the final observed to expected ratio was 0.90 (95% CI, 0.85-0.95) for 30-day nonelective rehospitalization and 0.87 (95% CI, 0.83-0.92) for 30-day postdischarge mortality. Similar decreases in adjusted outcome rates were seen for inpatient mortality (final observed rate was 0.79 [95% CI, 0.73-0.84]), 30-day mortality (0.86 [95% CI, 0.82-0.89]), and 30-day composite outcome (0.90 [95% CI, 0.86-0.94]) (eAppendix 2 in the Supplement). These reductions suggest that the system was able to maintain the same observed outcome rates despite worsening case mix.

    Table 2 and eAppendix 5 in the Supplement contrast data from the first to the last years of the study, highlighting the association between denominator definitions and the proportions of all hospitalizations and rehospitalizations included. For example, the restricted inpatient denominator captured 90.6% of all hospitalizations in the first year of the study; by 2017, this proportion decreased to only 79.6%. Decreased capture was similar among patients with low COPS2 scores (90.3% vs 79.4%) or LAPS2 scores (90.0% vs 78.3%). This decrease was most pronounced in the restricted public reporting denominator. For example, in the 2010 to 2011 period, only 58.0% of nonelective rehospitalizations were captured, and this proportion decreased to 45.2% by 2017. In addition, Table 2 shows that, because the public reporting metric excludes out-of-hospital deaths, by 2017 only 2.2% of inpatient deaths, 3.4% of 30-day deaths, and 5.0% of 30-day postdischarge deaths were captured by public reporting.

    Discussion

    Using data from a large contemporary cohort of all adult patients in the integrated KPNC system, this cohort study quantified improvements in 30-day hospital outcomes. Although hospitalized patients appeared to be sicker on admission over time, as evidenced by traditional measures such as the Charlson Comorbidity Index score, automated long-term scores (COPS2), and automated acute physiology scores (LAPS2), the outcomes improved for these patients. Hospitalization, rehospitalization, and mortality rates decreased simultaneously among all adults, not just those eligible for Medicare, Medicaid, or Kaiser Foundation Health Plans.

    These results complement and expand on the work of Epstein et al14 and Dharmarajan et al,15 highlighting that reducing hospitalization, rehospitalization, and mortality rates simultaneously is possible. Further evidence of the changing nature of hospitalization was the increasing proportion of patients discharged with home health services, which is most likely associated with changes in practice that emphasize home care. The proportion of hospitalizations subject to public reporting decreased substantially over the study period. Most of this decrease was associated with the increased rate of for-observation-only hospitalization.

    In addition to highlighting the gap between what public reporting metrics capture and what is happening in KPNC hospitals, the results of this study also illustrate the growing gap between data used by regulatory entities and data that are becoming widely used outside the research and regulatory settings. Although KPNC may be unusual in assigning automated severity of illness,19 inpatient deterioration risk,24-26 and rehospitalization risk20 scores to all of its adult patients, increasing numbers of hospitals are deploying automated early warning systems such as the electronic Cardiac Arrest Triage score30 or the Rothman Index,31 which demonstrates the increased availability of data that could have substantial implications for hospital rankings.12,13 Because most studies do not have data on severity of illness, comparing our findings is difficult. One notable exception was the study by Gupta et al,4 which found increased mortality where we did not but did report some admission physiological data (which, in contrast to the data in this study, remained stable before and after the HRRP penalties were implemented).

    Current reporting schema offer an incomplete, and certainly not patient-centered, perspective on the quality of hospital and postdischarge care. Because of the financial penalties associated with the rehospitalization metric, hospitals devote enormous resources to preventing rehospitalizations. In the absence of composite views displaying multiple metrics and balancing measures, however, it is difficult to assess whether these efforts are the most appropriate way to improve quality or whether higher scores in a single metric are necessarily associated with improved quality.

    These findings also suggest potentially fruitful areas for future research. Given that regulations encourage the use of 2 types of hospitalization (inpatient and for observation only), expanding on the work of researchers, such as Nuckols et al8 and Sabbatini et al,9 becomes important to better understand the characteristics of observation hospitalizations, including their difference from inpatient hospitalizations. Researchers must study the consequences of this shift on patients, and not just analyze its implications for hospital rankings, which may be minimal.32 Adverse consequences to patients could include increased out-of-pocket costs and differential access to care coordination. Future research should also include more granular analysis of the details of the discharge process and hospital-outpatient information transfer and referral. Another area that deserves continued attention is examination of the association between hospitalization and rehospitalization rates, which could vary dramatically depending on what incentives and quality safeguards are in place. Further research is also needed into the association between rehospitalization rates and inpatient, 30-day, 30-day postdischarge, and out-of-hospital mortality rates. At the least, public reporting should incorporate 2-dimensional plots that permit visualization of which hospitals are doing well on 2 metrics simultaneously.

    In an era in which an increasing proportion of patients have multimorbidity, we must address 2 challenges that face the medical profession. The first is the limited value of using mortality as a quality measure.33 The second is the need for prevention of hospitalization, including renewed efforts to find alternatives to hospitalization; after all, the best way to avoid rehospitalization is to not be hospitalized in the first place.

    Limitations

    This study has several limitations to the generalizability of the findings. First, the KPNC cohort consists of an insured population receiving care from a system with an unusually high degree of integration. This integration is manifest in the availability of services aimed at preventing hospitalization, such as chronic condition management programs, a central call center, and electronic patient portals. Furthermore, as a capitated system, KPNC does not receive incentives to hospitalize patients. This integration and incentive structure may explain KPNC’s ability to decrease all of the outcome measures simultaneously. This finding contrasts with the results in a recent study by Joynt Maddox et al,34 who reported that participation in the CMS Innovation Bundled Payments for Care Improvement initiative was not associated with decreases in rehospitalization or mortality among 492 participating hospitals.

    Second, KPNC’s for-observation-only hospitalization rate was relatively high, although some investigators have reported similar rates at individual hospitals.35,36 Patients in the present study appeared to be sicker over time, as evident from both diagnosis-based measures (Charlson Comorbidity Index score, COPS2) and acute physiology scores (LAPS2). Dharmarajan et al15 have also reported increasing comorbidity burden among hospitalized patients, but they did not have access to severity-of-illness data. Given that the use of physiological severity-of-illness scores outside the intensive care unit is not common in other health systems, we were not able to quantify the full extent of this trend and how generalizable it is.

    Conclusions

    This cohort study showed that hospitalizations, rehospitalizations, and mortality can be decreased simultaneously. A single measure is unlikely to accurately describe these changes. New data elements available from contemporary EMRs, such as severity-of-illness scores and patient care directives, should become part of public reporting.

    Back to top
    Article Information

    Accepted for Publication: October 14, 2019.

    Published: December 4, 2019. doi:10.1001/jamanetworkopen.2019.16769

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2019 Escobar GJ et al. JAMA Network Open.

    Corresponding Author: Gabriel J. Escobar, MD, Systems Research Initiative, Kaiser Permanente Division of Research, 2000 Broadway Ave, Oakland, CA 94612 (Gabriel.Escobar@kp.org).

    Author Contributions: Drs Escobar and Kipnis had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Escobar, Liu, Kipnis.

    Acquisition, analysis, or interpretation of data: All authors.

    Drafting of the manuscript: Escobar, Greene, Liu.

    Critical revision of the manuscript for important intellectual content: Escobar, Plimier, Liu, Kipnis.

    Statistical analysis: Plimier, Greene, Kipnis.

    Obtained funding: Escobar.

    Administrative, technical, or material support: Escobar.

    Supervision: Escobar, Liu.

    Conflict of Interest Disclosures: None reported.

    Funding/Support: This study was funded by The Permanente Medical Group Inc and Kaiser Foundation Hospitals Inc. Dr Liu was supported by grants K23GM112018 and R35GM128672 from the NIH.

    Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Additional Contributions: The following individuals from Kaiser Permanente Northern California provided additional assistance: Tracy Lieu, MD, MPH; Stephen Parodi, MD; Yi-Fen Chen, MD; and Diane Brown, RN, PhD, reviewed the manuscript; Andrew Amster, MSPH, and Tricia Reichert, BA, provided assistance with Centers for Medicare & Medicaid Services and HEDIS (Healthcare Effectiveness Data and Information Set) regulations. Philip Madvig, MD; Michelle Caughey, MD; and Christine Robisch, BA, MA, provided administrative support. Kathleen Daly, BS, provided assistance with formatting the manuscript. These individuals received no additional compensation, outside of their usual salary, for their contributions.

    References
    1.
    Jencks  SF, Williams  MV, Coleman  EA.  Rehospitalizations among patients in the Medicare fee-for-service program.  N Engl J Med. 2009;360(14):1418-1428. doi:10.1056/NEJMsa0803563PubMedGoogle ScholarCrossref
    2.
    Centers for Medicare & Medicaid Services. Hospital Readmissions Reduction Program (HRRP). https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/AcuteInpatientPPS/Readmissions-Reduction-Program.html. Accessed March 27, 2019.
    3.
    Centers for Medicare & Medicaid Services. Health insurance exchange quality ratings system 101. https://www.cms.gov/newsroom/fact-sheets/health-insurance-exchange-quality-ratings-system-101. Published August 15, 2019. Accessed September 23, 2019.
    4.
    Gupta  A, Allen  LA, Bhatt  DL,  et al.  Association of the hospital readmissions reduction program implementation with readmission and mortality outcomes in heart failure.  JAMA Cardiol. 2018;3(1):44-53. doi:10.1001/jamacardio.2017.4265PubMedGoogle ScholarCrossref
    5.
    Khera  R, Dharmarajan  K, Wang  Y,  et al.  Association of the Hospital Readmissions Reduction Program with mortality during and after hospitalization for acute myocardial infarction, heart failure, and pneumonia.  JAMA Netw Open. 2018;1(5):e182777-e182777. doi:10.1001/jamanetworkopen.2018.2777PubMedGoogle ScholarCrossref
    6.
    Centers for Medicare & Medicaid Services. Fact sheet: two-midnight rule. https://www.cms.gov/newsroom/fact-sheets/fact-sheet-two-midnight-rule-0. Published October 30, 2015. Accessed April 17, 2019.
    7.
    Society of Hospital Medicine Public Policy Committee. The hospital observation care problem: perspectives and solutions from the Society of Hospital Medicine. https://www.hospitalmedicine.org/globalassets/policy-and-advocacy/advocacy-pdf/shms-observation-white-paper-2017. Published September 2017. Accessed October 30, 2017.
    8.
    Nuckols  TK, Fingar  KR, Barrett  M, Steiner  CA, Stocks  C, Owens  PL.  The shifting landscape in utilization of inpatient, observation, and emergency department services across payers.  J Hosp Med. 2017;12(6):443-446. doi:10.12788/jhm.2751PubMedGoogle ScholarCrossref
    9.
    Sabbatini  AK, Hsia  RY.  It’s time for a strategic approach to observation care.  J Hosp Med. 2017;12(6):479-480. doi:10.12788/jhm.2760PubMedGoogle ScholarCrossref
    10.
    Sabbatini  AK, Wright  B, Hall  MK, Basu  A.  The cost of observation care for commercially insured patients visiting the emergency department.  Am J Emerg Med. 2018;36(9):1591-1596. doi:10.1016/j.ajem.2018.01.040PubMedGoogle ScholarCrossref
    11.
    Sabbatini  AK, Wright  B.  Excluding observation stays from readmission rates—what quality measures are missing.  N Engl J Med. 2018;378(22):2062-2065. doi:10.1056/NEJMp1800732PubMedGoogle ScholarCrossref
    12.
    Hayward  RA.  Access to clinically-detailed patient information: a fundamental element for improving the efficiency and quality of healthcare.  Med Care. 2008;46(3):229-231. doi:10.1097/MLR.0b013e318167579cPubMedGoogle ScholarCrossref
    13.
    Roberts  ET, Zaslavsky  AM, Barnett  ML, Landon  BE, Ding  L, McWilliams  JM.  Assessment of the effect of adjustment for patient characteristics on hospital readmission rates: implications for pay for performance.  JAMA Intern Med. 2018;178(11):1498-1507. doi:10.1001/jamainternmed.2018.4481PubMedGoogle ScholarCrossref
    14.
    Epstein  AM, Jha  AK, Orav  EJ.  The relationship between hospital admission rates and rehospitalizations.  N Engl J Med. 2011;365(24):2287-2295. doi:10.1056/NEJMsa1101942PubMedGoogle ScholarCrossref
    15.
    Dharmarajan  K, Qin  L, Lin  Z,  et al.  Declining admission rates and thirty-day readmission rates positively associated even though patients grew sicker over time.  Health Aff (Millwood). 2016;35(7):1294-1302. doi:10.1377/hlthaff.2015.1614PubMedGoogle ScholarCrossref
    16.
    Escobar  GJ, Greene  JD, Scheirer  P, Gardner  MN, Draper  D, Kipnis  P.  Risk-adjusting hospital inpatient mortality using automated inpatient, outpatient, and laboratory databases.  Med Care. 2008;46(3):232-239. doi:10.1097/MLR.0b013e3181589bb6PubMedGoogle ScholarCrossref
    17.
    Escobar  GJ, Greene  JD, Gardner  MN, Marelich  GP, Quick  B, Kipnis  P.  Intra-hospital transfers to a higher level of care: contribution to total hospital and intensive care unit (ICU) mortality and length of stay (LOS).  J Hosp Med. 2011;6(2):74-80. doi:10.1002/jhm.817PubMedGoogle ScholarCrossref
    18.
    Liu  V, Kipnis  P, Rizk  NW, Escobar  GJ.  Adverse outcomes associated with delayed intensive care unit transfers in an integrated healthcare system.  J Hosp Med. 2012;7(3):224-230. doi:10.1002/jhm.964PubMedGoogle ScholarCrossref
    19.
    Escobar  GJ, Gardner  MN, Greene  JD, Draper  D, Kipnis  P.  Risk-adjusting hospital mortality using a comprehensive electronic record in an integrated health care delivery system.  Med Care. 2013;51(5):446-453. doi:10.1097/MLR.0b013e3182881c8ePubMedGoogle ScholarCrossref
    20.
    Escobar  GJ, Ragins  A, Scheirer  P, Liu  V, Robles  J, Kipnis  P.  Nonelective rehospitalizations and postdischarge mortality: predictive models suitable for use in real time.  Med Care. 2015;53(11):916-923. doi:10.1097/MLR.0000000000000435PubMedGoogle ScholarCrossref
    21.
    Agency for Healthcare Research and Quality. Quality Indicators Software Instructions, SAS Version 4.5. https://www.qualityindicators.ahrq.gov/Downloads/Software/SAS/V45/Software_Instructions_SAS_V4.5.pdf. Published May 2013. Accessed December 5, 2014.
    22.
    National Committee for Quality Assurance.  Technical Specifications. Vol 2. Washington, DC: National Committee for Quality Assurance; 2018.
    23.
    National Committee for Quality Assurance. HEDIS®1 2018 volume 2: technical update, 2019. https://www.ncqa.org/wp-content/uploads/2018/10/HEDIS-2019-Volume-2-Technical-Update.pdf. Accessed March 27, 2019.
    24.
    Escobar  GJ, Dellinger  RP.  Early detection, prevention, and mitigation of critical illness outside intensive care settings.  J Hosp Med. 2016;11(suppl 1):S5-S10. doi:10.1002/jhm.2653PubMedGoogle ScholarCrossref
    25.
    Escobar  GJ, Turk  BJ, Ragins  A,  et al.  Piloting electronic medical record-based early detection of inpatient deterioration in community hospitals.  J Hosp Med. 2016;11(suppl 1):S18-S24. doi:10.1002/jhm.2652PubMedGoogle ScholarCrossref
    26.
    Kipnis  P, Turk  BJ, Wulf  DA,  et al.  Development and validation of an electronic medical record-based alert score for detection of inpatient deterioration outside the ICU.  J Biomed Inform. 2016;64:10-19. doi:10.1016/j.jbi.2016.09.013PubMedGoogle ScholarCrossref
    27.
    Deyo  RA, Cherkin  DC, Ciol  MA.  Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases.  J Clin Epidemiol. 1992;45(6):613-619. doi:10.1016/0895-4356(92)90133-8PubMedGoogle ScholarCrossref
    28.
    Ladiray  D, Quenneville  B. Outline of the X-11 method. In:  Seasonal Adjustment with the X-11 Method: Lecture Notes in Statistics. Vol 158. New York, NY: Springer; 2001. doi:10.1007/978-1-4613-0175-2_3
    29.
    Box  GEP, Jenkins  GM, Reinsel  GC, Ljung  GM.  Time Series Analysis: Forecasting and Control. 5th ed. Hoboken, NJ: Wiley; 2015.
    30.
    Churpek  MM, Yuen  TC, Winslow  C,  et al.  Multicenter development and validation of a risk stratification tool for ward patients.  Am J Respir Crit Care Med. 2014;190(6):649-655. doi:10.1164/rccm.201406-1022OCPubMedGoogle ScholarCrossref
    31.
    Rothman  MJ, Rothman  SI, Beals  J  IV.  Development and validation of a continuous measure of patient condition using the electronic medical record.  J Biomed Inform. 2013;46(5):837-848. doi:10.1016/j.jbi.2013.06.011PubMedGoogle ScholarCrossref
    32.
    Venkatesh  AK, Wang  C, Ross  JS,  et al.  Hospital use of observation stays: cross-sectional study of the impact on readmission rates.  Med Care. 2016;54(12):1070-1077. doi:10.1097/MLR.0000000000000601PubMedGoogle ScholarCrossref
    33.
    Holloway  RG, Quill  TE.  Mortality as a measure of quality: implications for palliative and end-of-life care.  JAMA. 2007;298(7):802-804. doi:10.1001/jama.298.7.802PubMedGoogle ScholarCrossref
    34.
    Joynt Maddox  KE, Orav  EJ, Zheng  J, Epstein  AM.  Evaluation of Medicare’s bundled payments initiative for medical conditions.  N Engl J Med. 2018;379(3):260-269. doi:10.1056/NEJMsa1801569PubMedGoogle ScholarCrossref
    35.
    Wright  B, O’Shea  AM, Ayyagari  P, Ugwi  PG, Kaboli  P, Vaughan Sarrazin  M.  Observation rates at veterans’ hospitals more than doubled during 2005–13, similar to Medicare trends.  Health Aff (Millwood). 2015;34(10):1730-1737. doi:10.1377/hlthaff.2014.1474PubMedGoogle ScholarCrossref
    36.
    Lind  KD, Noel-Miller  CM, Sangaralingham  LR,  et al.  Increasing trends in the use of hospital observation services for older Medicare Advantage and privately insured patients.  Med Care Res Rev. 2019;76(2):229-239. doi:10.1177/1077558717718026PubMedGoogle ScholarCrossref
    ×