Section 07 - PowerPoint PPT Presentation

part 07 l.
Skip this Video
Loading SlideShow in 5 Seconds..
Section 07 PowerPoint Presentation
Section 07

play fullscreen
1 / 38
Download Presentation

Section 07

Presentation Transcript

  1. PART 07 Evaluating Hospital Performance BIO656--Multilevel Models

  2. PERFORMANCE MEASURES Patient outcomes • Mortality, morbidity, satisfaction with care • 30-day mortality among heart attack patients (Normand et al JAMA 1996, JASA 1997) Process • Medication & test administration, costs • Laboratory costs for diabetic patients • Number of physician visits • Hofer et al JAMA, 1999 • Palmer et al. (1996) BIO656--Multilevel Models

  3. DATA STRUCTURE Multi-level • Patients nested in physicians, hospitals, HMOs, ... • Providers clustered by health care systems, market areas, geographic areas • Covariates at different levels of aggregation: • patient, physician, hospital, ... Variation in variability • Statistical stability varies over physicians, hospitals, .. BIO656--Multilevel Models

  4. MLMs are Effective Correlation at many levels • Hospital practices may induce a strong correlation among patient outcomes within hospitals even after accounting for patient characteristics Structuring estimation • Stabilizing noisy estimates • Balancing SEs • Estimating ranks and other non-standard summaries BIO656--Multilevel Models

  5. The Cooperative Cardiovascular Project (CCP) • Abstracted medical records for patients discharged from hospitals located in Alabama, Connecticut, Iowa, and Wisconsin (June 1992May 1993) • 3,269 patients hospitalized in 122 hospitals in four US States for Acute Myocardial Infarction BIO656--Multilevel Models

  6. GOALS • Identify “aberrant” hospitals with respect to several performance measures • Report the statistical uncertainty associated with ranking of the “worst hospitals” • Investigate if hospital characteristics explain variation in hospital-specific mortality rates BIO656--Multilevel Models

  7. DATA Outcome • Mortality within 30-days of hospital admission Patient characteristics • Admission severity index constructed on the basis of 34 patient attributes Hospital characteristics • Urban/Rural • (Non academic)/(versus academic) • Number of beds BIO656--Multilevel Models

  8. Why adjust for case mix?(patient characteristics) • Irrespective of quality of care, older/sicker patients with multiple diseases have increased need of health care services and poorer health outcomes • Without adjustment, physicians/hospitals who treat relatively more of these patients will appear to provide more expensive and lower quality care than those who see relatively younger/healthier patients • If there is inadequate case mix adjustment, evaluations will be unfair • But, need to avoid over adjusting BIO656--Multilevel Models

  9. Case-mix Adjustment Compute hospital-specific, expected mortality by: • estimating a patient-level mortality model using all hospitals 2. averaging the model-produced probabilities for all patients within a hospital • Hospitals with “higher-than-expected” mortality rates can be flagged as institutions with potential quality problems, but need to account for uncertainty • Need to be careful, if also adjusting for hospital characteristics • May adjust away the important signal BIO656--Multilevel Models

  10. Wrong SEs • Test-based (as we know, very poor approach) BIO656--Multilevel Models

  11. Hospital Profiling of Mortality Rates Acute Myocardial Infarction Patients(Normand et al. JAMA 1996, JASA 1997) BIO656--Multilevel Models

  12. Hierarchical logistic regression I: Patient within-provider • Patient-level logistic regression model with random intercept & slope II: Between-provider • Hospital-specific random effects are regressed on hospital-specific characteristics • Explicit regression BIO656--Multilevel Models

  13. Admission severity index(Normand et al. 1997 JASA) BIO656--Multilevel Models

  14. sevbar 0 + 1(sevij – sevbar) 0 1 BIO656--Multilevel Models

  15. we use b0i + b1i(...) BIO656--Multilevel Models

  16. b0i = *00 + N(..), etc. Interpretation of parameters is different for the two levels BIO656--Multilevel Models

  17. RESULTS • Estimates of regression coefficients under three models: • Random intercept only • Random intercept and random slope • Random intercept, random slope, and hospital covariates • Hospital performance measures BIO656--Multilevel Models

  18. Normand et al. JASA 1997 BIO656--Multilevel Models

  19. 30-DAY MORTALITY2.5th and 97.5th percentiles for a patient of average admission severity Exchangeable model • Random intercept and slope, no hospital covariates log(odds): (-1.87,-1.56) probability,scale: (0.13, 0.17) Covariate (non-exchangeable) model • Random intercept and slope, with hospital covariates • Patient treated in a large, urban academic hospital log(odds): (-2.15,-1.45) probability scale: (0.10,0.19) BIO656--Multilevel Models

  20. Effect of hospital characteristics on baseline log-odds of 30-day mortality • For an average patient, rural hospitals have a higher odds ratio than urban hospitals • Indicates between-hospital differences in the baseline mortality rates • Case-mix adjustment may be able to remove some of this difference BIO656--Multilevel Models

  21. Estimates of Stage-II regression coefficientsIntercepts BIO656--Multilevel Models

  22. Effect of hospital characteristics on association between severity and mortality(slopes) • The association between severity and mortality is modified by hospital size • Medium-sized hospitals have smaller severity/mortality associations than large hospitals • Indicates that the effect of clinical burden (patient severity) on mortality differs across hospitals BIO656--Multilevel Models

  23. Estimates of Stage IIregression coefficientsSlopes BIO656--Multilevel Models

  24. Homework is on front table BIO656--Multilevel Models

  25. Observed and risk-adjusted hospital mortality rates Urban Hospitals Histogram displays (observed – adjusted) BIO656--Multilevel Models

  26. Observed and risk-adjusted hospital mortality rates Rural Hospitals Histogram displays (observed – adjusted) Substantial adjustment for severity BIO656--Multilevel Models

  27. FINDINGS • There is substantial adjustment for admission severity • Generally, urban hospitals are adjusted less than rural • There is less variability in observed or adjusted estimated rates for urban hospitals than for rural hospitals Can you explain why? BIO656--Multilevel Models

  28. Normand et al. JASA 1997 BIO656--Multilevel Models

  29. Average the probabilities Don’t average the covariates BIO656--Multilevel Models

  30. k denotes a draw from the posterior BIO656--Multilevel Models

  31. Plug in the average covariate Keep the hospital variation BIO656--Multilevel Models

  32. BIO656--Multilevel Models

  33. Comparing measures of hospital performance Three measures of hospital performance • Probability of a large difference between adjusted and standardized mortality rates • Probability of excess mortality for the average patient • Z-score BIO656--Multilevel Models

  34. Hospital Rankings: Normand et al 1997 JASA BIO656--Multilevel Models

  35. Hospital Ranks • There was moderate disagreement among the criteria for classifying hospitals as “aberrant” • Nevertheless, hospital 1 is ranked worst • It is rural, medium sized non-academic with an observed mortality rate of 35%, and adjusted rate of 28% BIO656--Multilevel Models

  36. Adjusting for hospital-level charateristics Changes the comparison group in “as compared to what?” • All hospitals (unadjusted at hospital level) • Hospitals of a similar size, urbanicity, ... • Percent of physicians who are board certified • Hospitals with a similar death rate  Variance reduction and goodness of fit should not be the primary considerations • “As compared to what?” must dominate BIO656--Multilevel Models

  37. Discussion • Profiling medical providers is multi-faced and data intensive process with substantial implications for health care practice, management, and policy • Major issues include data quality and availability, choice of performance measures, formulation of statistical models (including adjustments), reporting results • The ranking approaches and summaries used by Normand and colleagues are very good, but some improvement is possible BIO656--Multilevel Models

  38. Multi-level models address key technical & conceptual profiling issues, including • Adjusting for patient severity • Accounting for within-provider correlations • Accounting for differential sample sizes at all levels • Stabilize estimates • Structure ranking and other, derived comparisons BIO656--Multilevel Models