Teaching to the test in medicine

Teaching to the test is controversial. It helps students pass tests and makes educational programs look successful, but everyone knows it does not guarantee improved education.  It can be beneficial if it helps focus teaching on the core curriculum, and if the results are used to evaluate and improve the teaching process. It can also be abused. In its most extreme form, teaching to the test consists of skipping the educational process entirely and simply providing the correct answers to test questions. There is universal agreement that this is fraudulent. In many cases it is illegal.

Why are we taking this approach to quality improvement in medicine?

 

Imagine the following (fictional) scenario:

Studies show that auto mechanics are not doing a good job of installing and repairing automobile brakes, and the result is accidents, injuries and occasional deaths from poor work. The government decides the solution is to incentivize competence and quality and sets up a program of brake repair certification.  A study commission identifies the core information and skills needed to be a high quality brake mechanic and the safety board plans to test mechanics on their mastery of this information, certifying those who score over 85%, and paying certified mechanics more than uncertified mechanics. Educational manuals and training programs are developed to certify mechanics and a test is developed to evaluate competence.

Now imagine the safety board publishes the questions and answers for the certification test. It is no longer necessary to study the manuals or take a training program. Just study the test questions and answers. It is now easy to become certified without actually improving one’s knowledge or competence - and many will.  Perhaps most, if the burden of taking courses is great enough. All of a sudden, nearly every brake mechanic is certified, the safety board boasts of its success, the quality of work remains essentially unchanged, and there is no decrease in accidents, injuries or deaths.

 

In medicine, you do not have to imagine a made-up scenario. This is actually how a great deal of quality improvement initiatives are designed and implemented. For example:

The Institute of Medicine and other reputable groups have identified accidental falls as a serious problem not adequately addressed in primary care practices. Studies show that a cheap screening test (asking the patient if they have fallen and gotten hurt in the last year) can identify a high risk group, and that if this high risk group is evaluated and treatments instituted based on the findings of the evaluation, the frequency, severity and harms from falls can be significantly reduced.

The appropriate clinical approach to this problem would be to:

  • Identify the population to be screened.
  • Make sure there is a diagnostic evaluation process available to those with a positive screen, and appropriate interventions available based on the diagnostic evaluation. (Screening is of no medical benefit unless it leads to effective treatment, and treatments will vary based on the cause of fall risk: Parkinsons, alcoholism, vision impairment, poor lighting in the home, seizures, weakness, arrhythmias, medication side effects, neuropathy...)
  • Set up a screening process. In this case, the nursing staff asks the screening question when putting the patient in a room and alerts the physician to positive responses.
  • The physician (or institution where the physician works)  makes sure that patients with a positive response are evaluated and appropriate interventions are offered.
  • Clinical outcomes improve.
  • Data is collected about the numbers screened, the numbers with positive screens, the numbers with positive evaluations, the treatments used, and perhaps (though this may be beyond the ability of individual physicians or practices) outcomes.

Instead, it is common these days to see poorly thought out incentives that encourage institutions to teach to the test:

  • CMS and insurers identify a population to screen and either offer an incentive to screen or a penalty if screening is not done.
  • Practices and institutions set up a system to screen, designed specifically to meet the requirements for earning the financial incentive or avoiding the penalty.
  • There is no requirement to evaluate positive screens, or to offer treatment based on the evaluations. As a result, there is often no actual diagnosis, treatment or patient benefit. (This is not malevolently intended. It is because it is easier to track whether or not a screening has occurred than the complex behaviors that should follow a screening. This, of course, is the classic Streetlight Effect - looking for your lost car keys under the street light because the light is good there, rather than in the stairwell where you think you probably dropped them.)
  • The practice or institution tracks how compliant they are with screening and gets paid for screening - regardless of whether or not patients are assessed and treatments offered. 
  • Clinical outcomes do not change.

This same failed approach is seen with many other well-meaning but poorly designed and implemented quality programs: screening for depression, reducing readmissions to the hospital, antibiotics within 4 hours of admission for community acquired pneumonia, beta-blockers to reduce heart disease risk, medication reconciliation, BP targets for treatment, lipid targets for treatment, A1c targets for treatment.

Institutions like to claim they have no choice, that they are forced to comply with a myriad of external directives. The existence of external requirements is a fact of life, but this does not have to be an either-or decision.  Striving for quality is not incompatible with striving for reimbursement. It is a matter of framing and priorities. One can create a clinically sound process to identify and address fall-risk in a way that meets regulatory and reimbursement requirements. Or one can design a process to meet regulatory and reimbursement requirements, and hope that clinicians will find a way to insert quality into the process. My general experience has been that where governance is primarily non-clinician, administrative priorities dominate; clinical priorities dominate only in institutions that are primarily clinician run.

It is harder to address fall-risk as a medical issue. It is also better. Every time an institution chooses to frame a complex medical problem narrowly as a compliance or financial issue, they wander from their core mission, miss the chance to get paid for improvement rather than for compliance, and become part of the problem rather than part of the solution. They pass the test but miss the point.

 


 

 




Links to more on this topic::