A Statistical Magical Mystery Tour

By Anthony Rosner, PhD, LLD [Hon.], LLC

With all due respect to the Fab Four, there is little doubt about the number of the Beatles who made their record-breaking albums in the 1960s (unless you believed all those ominous rumors, such as "Paul is dead," and spent too many hours staring at the album cover of Abbey Road, trying to confirm them). Sometimes, however, numbers can do quirky things, either by chance or by design.

All you have to do is extract some dizzying examples (another Fab Four) from the academic literature. Exhibit A is the full-scale, randomized clinical trial published a few years ago that studied the effects of spinal manipulation on dysmenorrhea. In that particular report, there was a startling disconnect with its preceding pilot study, despite the fact that both trials were conducted by largely the same research team. Whereas the pilot study suggested that pain scores decreased significantly in a manipulated, but not a sham group of patients,¹ the full-scale trial displayed no such effect.² "How could this be?" you ask. It turns out that there were a number of variances of procedures between the two trials, but the big picture shows that the pain entry scores at baseline in the full-scale trial were nearly two points lower on the standard 11-point VAS scale than in the pilot study - so there was virtually no room for improvement, regardless of whatever pleasure was extended to the patients, including winning the state lottery. (The reason for this anomaly turns out to be that, to be able to obtain the necessary number of recruited individuals in the full-scale trial, the research team chose to lower the eligibility criteria, such that patients for the first time were allowed to report to the clinic up to 48 hours after their pain had subsided, which essentially meant that many patients in this trial were coming in the door with no pain at all!)

Exhibit B comes from the Beam Trial, conducted in the U.K., in which eight separate treatment arms were compared with respect to their improvements in pain, impairment and disability. In this case, it was exceedingly difficult to find major differences, one being found for the SF-36 scale, in which manipulation yielded small, but significantly better results. The problem was again one of entry scores: Roland-Morris Questionnaire levels at baseline were only 9.0 out of maximum of 24, providing only limited room for improvement.³

Exhibit C includes an outstanding example of how effects may be obscured by small sample sizes - commonly known in statistics as a "type II" error. In comparing patient groups administered either high-velocity cervical spinal manipulation or low-level laser treatments (as a control), Nilsson observed a tendency of the manipulated group to fare better in terms of pain experienced, headache-hours per day, and use of analgesics to alleviate discomfort. The first trial involving 39 patients showed a trend toward improvement in all categories, but failed to reach the usual level of statistical significance.⁴ When increasing the total patient number to 54 with resumed recruitment, however, the investigators arrived at statistically significant differences in all three parameters.⁵ Had previously published asthma6 or low-back-pain trials,⁷ which reported negative or equivocal results, been repeated with larger patient numbers, trends that appeared in much of the data might have become statistically significant differences, overcoming a type II error. Clearly, the potential exists to misinterpret the results of an RCT if they are not reviewed from a multiplicity of viewpoints, rather than accepting statistical numbers at face value.

It gets better - far better (or worse, depending on your point of view). In Exhibit D, The chancellor of the University of Wisconsin at Whitewater (no relation to the Clinton Whitewater controversy) penned a provocative report earlier this summer, "America's Most Literate Cities,"⁸ that ranked 64 domestic cities using five criteria: newspaper circulation; bookstores; library resources; publishing; and educational attainment. Now, I have to admit, being a Bostonian, I might have harbored some smug preconceptions here (living in the "Athens of America" and so on). But Boston didn't even crack the top 10 - it ranked 13th, tied with (if you can believe this) Las Vegas!

But let's take a closer look. In the "bookstore" category, our doppelganger, Las Vegas, ranked eighth. And Boston? How about 43rd? This had to do with the number of booksellers per capita. You sit there scratching your head, until you hear an interesting admission from the study's author, John Miller: "This may have included adult bookstores." This is in spite of the fact that all bookstores included in the study were supposedly members of the American Booksellers Association. So much for membership criteria. (Could the country club be far behind?)

Then, we turn to newspaper circulation: Newark placed first; Boston, seventh. It turns out that Newark's only paper, the StarLedger, circulates its 407,000 dailies far beyond its nominal city base of 275,000 (which might explain why everyone in TV's The Sopranos appears to be reading a copy).

So, the point here is to be aware of how numbers can be massaged into the most abstruse of reportings. This is something to bear in mind when studies appear in the scientific literature that seem to be egregiously counterintuitive. They must be scrutinized and questioned without mercy, so that there is a clear understanding of what they are reporting - and what they are not. The FCER has consistently taken the challenge and stood ready to interpret the literature as it appears, so that it does not misrepresent the capacities and limitations of clinical practice offered by chiropractic. To neglect this responsibility would open the door to all kinds of intrigue, the most interesting provided by a tidbit from one of my favorite movies from more than 50 years ago, which offers, in retrospect, a prophetic insight into Florida politics and the 2000 election. As the gangster Edward G. Robinson proudly proclaimed to Humphrey Bogart in Key Largo:

Let me tell you about Florida politicians. I make them out of whole cloth, just like a tailor makes a suit. I get their name in the newspaper. I get them some publicity and get them on the ballot.

Then after the election, we count the votes. And if they don't turn out right, we recount them. And recount them again. Until they do.⁹

What is the take-home message here? It's simple: The responsibility FCER insists upon is becoming an indispensable tool for the clinician, as the recognition of their clinical procedures becomes increasingly evidence-based. In other words, they must recognize both the value of research and its critical appraisal, so it is neither misinterpreted nor abused.

References

1. Kokjohn K, Schmid DM, Triano JJ, Brennan PC. The effect of spinal manipulation on pain and prostaglandin levels in women with primary dysmenorrhea. Journal of Manipulative and Physiological Therapeutics 1992; 15(5):279-285.
2. Hondras MA, Long CR, Brennan PC. Spinal manipulative therapy vs. a low-force mimic maneuver for women with primary dysmenorrhea: a randomized, observer-blinded, clinical trial. Pain 1999, 81(1-2): 105-114.
3. Breen A. The chiropractic interventions: their place in effective healthcare. Presentation at the European Chiropractors' Union Convention, Heidelberg, Germany, May 29, 2003.
4. Nilsson N. A randomized controlled trial of the effect of spinal manipulation in the treatment of cervicgogenic headache. Journal of Manipulative and Physiological Therapeutics 1995; 18(7):435-440.
5. Nilsson N, Christensen HW, Hartvigsen J. The effect of spinal manipulation in the treatment of cervicogenic headaches. Journal of Manipulative and Physiological Therapeutics 1997;20(5):326-330.
6. Cherkin DC, Deyo RA, Battie M, Street J, Barlow W. A comparison of physical therapy, chiropractic manipulation, and provision of an educational booklet for the treatment of patients with low back pain. New England Journal of Medicine 1998;339(15):1021-1029.
7. Balon J, Aker PD, Crowther ER, Danielson C, Cox PG, O'Shaugnessy D, Walker C, Goldsmith CH, Duku E, Sears MR. A comparison of active and simulated chiropractic manipulation as adjunctive treatment for childhood asthma. New England Journal of Medicine 1998;339(15):1013-1020.
8. Miller J. America's most literate cities. www.uww.edu/cities, Posted and described by Beam A. The Boston Globe, July 15, 2003.
9. Key Largo. Directed by John Huston. Warner Brothers. 1948.

Anthony Rosner, PhD
Brookline, Massachusetts

---

Click here for more information about Anthony Rosner, PhD, LLD [Hon.], LLC.