This Week's Lesson in Comparative Effectiveness

Before I came to my current hospital and health system, I served in a large tertiary referral hospital. My areas of responsibility included several intensive care units. In those units physicians were working with the APACHE System. APACHE stood for “Acute Physiology, Age, Chronic Health Evaluation.” APACHE was a means of predicting outcomes for patients admitted to intensive care units. It went through several generations (some institutions are still using APACHE III), but all were based on the same process. Information about thousands of patients was recorded over time, including their conditions at admission to the ICU and the outcomes of their stay. The point was to use that information to project the outcomes of future patients based on similar conditions at admission.

This was explained to me by a medical resident. He could put in the characteristics of a given patient, and have the computer to compare them with past patients. Based on the results with those past patients, the computer would come out with a set of ratios. For example, they would look at Patient A and put in his circumstances. The computer would then give some percentages – say, of those historical patients with these circumstances, 70% died and 30% lived to leave the ICU.

As I reflected on APACHE and discussed it with physicians, I realized two things. First, it could offer some guidelines that could be helpful. If the statistics were, say, 90% and 10%, the prediction could be pretty clear. However, it also still required a physician, for all the automation. It still required a physician to look at this specific patient – say, Patient A – and make the professional assessment as to whether Patient A more likely fell into the 90% or the 10%.

That came to mind again with the announcement that the U.S. Preventive Services Task Force (USPSTF) had published new guidelines for mammography and breast self examination. The new guidelines were significant change. Indeed, they were so significant that the American Cancer Society (ACS) and the American College of Obstetricians and Gynecologists (ACOG) have publically disagreed, saying they will continue to follow earlier recommendations.

Most folks are aware of the changes from extensive news coverage (I would suggest looking here or here.) What I think makes this interesting is that it coincides with discussions about health care reform, and highlights one of the important issues in that discussion. You see, this is a straightforward example of the promise and the difficulties of comparing effectiveness of procedures in evidence-based medicine.

Comparing effectiveness is how we got to these recommendations. USPSTF is an independent panel of functioning under the auspices of the Department of Health and Human Services (HHS, and specifically under the Agency for Healthcare Research and Quality [AHRQ]). The panel is independent in that none of its members are employed by the federal Government, nor do they represent agencies within the Government. The purpose of USPSTF is “to evaluate the effectiveness of clinical preventive services that were not previously examined; to re-evaluate those that were examined and for which there is new scientific evidence, new technologies that merit consideration, or other reasons to revisit the published recommendations;…” They want to bring the best science to considering and reconsidering those steps we take to prevent illness and reduce its severity.

Of course, changes in science should bring new consideration and so new recommendations. On the other hand, we’re seeing clearly how those new recommendations themselves can bring their own questions. ACS and ACOG and many women are asking about these new recommendations, “What are the risks that disease (and for the women involved, “my disease”) will be missed with fewer screenings and a downplaying of breast self-exams?” And there is certainly some risk that some patients’ lumps will be missed. Actually, there’s near certainty that there will be women whose lumps will be missed and who will suffer as a consequence.

At the same time, the new recommendations are based on some other certain risks. False positives do result in unnecessary procedures, from additional radiological studies to additional biopsies to unnecessary surgeries; and each of those additional procedures has its own inherent risks. An additional x-ray is a radiation exposure. Unnecessary surgeries include the risks of infection and other complications, both from the surgery itself and from the required anesthesia. And that’s without the impact on the lives of women of additional anxiety and disruption of their lives and relationships, all based on false information. Unfortunately, with the current recommendations these unnecessary risks are happening.

Both sets of risks are measurable, at least across populations. That is, looking at medical practice as a whole, these experts can make good estimates of what those risks are.

According to the newly published research analysis:

• 1,904 women between the ages of 39 and 49 would need to be invited for screening to have one breast cancer death prevented.
• 1,339 women between the ages of 50 and 59 would need to be invited for screening to prevent one death.
• 377 women between the ages of 60 and 69 would need to be invited for screening to prevent one death.
  

On the other hand, “about 60% more false-positive results could be expected for every 1,000 mammograms performed when screening is started at age 40 instead of 50.”

So, why all the attention and the anxiety? The numbers make it look straightforward to change practice – unless you’re a woman already anxious, or at least alert and attentive, about your individual risk of breast cancer. Why should 1900 women suffer unnecessarily to prevent the death of the 1901st? It’s a good question, but doesn’t address the difficulty that we can’t know who which woman in those 1901 will actually be the one whose death is prevented.

Now, a part of the answer is the same in this case as in APACHE: it’s up to physicians to speak with their patients and say, “In your case, with your personal and family history and your risk factors, this is what I recommend.” However, there are other potential complications. Most insurers, from Medicare to the smallest commercial insurer, reimburse doctors for procedures and not for conversations. Will the doctor, however well intentioned, feel she or he has the time? Most insurers, from Medicare to the smallest commercial insurer, want to avoid paying for “unnecessary” procedures. Will they be willing to make exceptions? And how much effort will it take from patient and doctor for exceptions to be accepted?

This will be an ongoing issue if our standards for guiding practice are based on evidence of comparative effectiveness. We will continue to struggle to balance good general practice with good practice in specific cases. We will continue to wrestle with how to make the important exceptions to the structures we put in place to spare risks and costs for the majority of patients; for those important exceptions are people first and foremost.

I’m still a great believer in “comparative effectiveness.” I think it will be an important step in “bending the cost curve” in health care – which is just a fancy way of saying that we want to slow the pace at which costs for health care go up. At the same time, we need to be aware that to change based on “comparative effectiveness” will not be easy. It will take more work, and not less. It will involve especially more hard thinking by doctors and professionals, and more hard conversations between professionals and patients. “Comparative effectiveness” will come with its own difficulties; and the current discussion about recommendations for breast cancer screening are giving us a good example.