A purportedly serious publication in a serious forum that was published this week has given rise to a bunch of breathless CAM-related headlines in my news feed. CAM being Complementary and Alternative Therapies.
I presume that this is what the authors and their employer wanted, as the supposed good news story is in fact one of the most blatant examples of quackacademic confabulation I have seen in ages. By this I mean that all is not what it appears with this review.
Firstly, the article in question was not prepared as an original research article. This is important, as some of the headlines seem to claim this is “new evidence”.
Look at the table of contents in the journal. It is part of a “CME Credit” symposium. This is an exercise where the editor invites summary articles on behalf of the journal from experts in a field to create what is supposed to be an uncontroversial and impartial review of the state of consensus opinion based on the literature in any given field.
Busy doctors read the review and complete some questions, then claim the learning time for their professional development. I am responsible for a lot of this sort of thing in my professional life, so I know that they are not always peer reviewed and can sometimes go off the rails if the expert panel is not carefully chosen. They are most definitely not considered a means of introducing new findings.
The article is apparently the first in a “Pain Medicine Symposium” to be offered over several editions of the journal. I would have expected to see a few names I recognized from the American Academy of Pain Medicine or the International Association for the Study of Pain, or maybe even the American Pain Society among the authors. But did I?
Nope.
They all came from the National Centre for Complementary and Integrative Health (NCCIH). The lead author is an epidemiologist, and the co-authors include a naturopath with publications about echinacea and colds, a chiropractor (who at least has published about yoga and pain), a cardiologist who works in the regulatory affairs part of NCCIH and a statistician. Not exactly representative of mainstream opinion within the specialty of pain medicine I would submit.
Not to worry, I thought. There might be something to learn so let’s dive in. They start out reasonably enough, sketching out some basic facts about the huge societal impact of persistent pain in the United States, which is similar to most countries. They nominate a few selected treatments; acupuncture, spinal manipulation, massage, meditation, “natural product supplements” and yoga, Tai Chi and Qigong, which all get lumped together, though they are in fact wildly different in practice and contradictory in concept.
After then setting out a few plausible-sounding excuses for what follows, which constitute the barest academic fig leaf to cover their sectarian belief systems, they also go on to specifically exclude cancer pain from their review. Why? I can guess. Even sincerely deluded cranks have some vestiges of conscience and wouldn’t stoop to taking advantage of cancer patients. (Or would they?)
Having cherry-picked the topics and modalities that have thrown up the most randomly positive studies over the years, they then set out a steady stream of mostly negative studies which they somehow conclude support their argument that there is some useful clinical evidence for some of the therapies.
The Science Based Medicine blog has set out an excellent critique of the methods and results of the review. But the authors would not have gotten this paper published even in an invited slot without the mandatory disclaimers about the shortcomings of the studies. These are worth looking at in detail.
First:
The trial samples tend to be white, female, and older, with very few, if any, minority group participants; as such, the generalizability of the findings to the breadth of patients seen by primary care physicians in the United States is still unresolved.
Ummm, I can resolve it. They aren’t at all generalizable, since the world does not consist entirely of elderly white female people. Next caveat:
Often, the trials reviewed were small, with fewer than 100 total participants. Small trials are prone to more variability and to false-negative results.
No kidding. This is a well-known problem. If you combine the results of lots of small, bad trials you don’t reliably get closer to the truth. So why, after hundreds of trials and tens of millions of research dollars are we still getting acupuncture studies with a couple of dozen participants?
A false-negative result is one in which a real treatment is inappropriately rejected by the results of the study. However, an under-powered study is also far less likely to be reliable if positive. Doing under-powered studies using variable study designs is the exact opposite of what savvy research funders require. You don’t want to waste buckets of cash doing studies which can’t be compared or assimilated down the track to get you closer to the truth.
Given that they admit their analysis is entirely of suspect positive results from poorly-designed under-powered studies, it hardly seems worth discussing any further. But we will look at the next caveat the authors put forward:
In many of the trials in which the statistical superiority of a given complementary health approach was reported, it was not clear if the differences vs the control group were clinically relevant.
In plain English, this means you would not notice the benefit of the treatment in real life. Maybe the treatment helps but it’s not worth bothering with even if it does help. In pain medicine this is all-important, as we are currently on a crusade to reduce the prescribing of drugs such as sustained-release opioids and benzodiazepenes in situations where they have poor efficacy and significant risk. We have a couple of decades of research showing that small, short-term reductions in pain intensity are not associated with improved quality of life or health status.
For most complementary approaches, there are no standard treatment protocols or algorithms, and in the case of dietary supplements, no rigorously established dosages and products; as such, trials of a given complementary approach rarely compare the exact same intervention.
This is the bald truth. There is no standardization of therapies, no quality control of products, nothing at all but spin and fervent belief. The situation with acupuncture for example is nicely summed up by Paul Ingraham of the Pain Science blog:
the most favorable evidence available is also the oldest, weakest and the most biased, and even that evidence is underwhelming, benefits that barely register as clinically significant — much ado about not much, even if it’s actually real, which no better study has ever confirmed.
So to recap, NCCIH released a statement for the press as if this represented new research, when their employees actually published a CME activity rehashing mostly old data, that by their own admission drew from highly unreliable studies and even if accurate was not generalizable to the population as a whole.
Never even mind that some of the therapies are conceptually exclusive of each other and biologically implausible. The press release of this through-the-looking-glass piece of “research” was then widely distributed to inform the public of the exact opposite of what the data presented actually showed. This apparently represents the pinnacle of quality in CAM research. It’s certainly the most expensively-funded.
I will be closely watching with interest the next part of the Pain Medicine Symposium from Mayo Clinic Proceedings. I hope the editors were mortified to see the opportunism with which the solid reputation of their publication was exploited by the media beat-up that followed. They have let their readership and their academic integrity down badly with this review. I can hardly believe they will embarrass themselves so badly again.
By Michael Vagg, Clinical Senior Lecturer at Deakin University School of Medicine & Pain Specialist, Deakin University. This article was originally published on The Conversation. Read the original article.
