Health & Wellness / Lifestyle

Obstacles and Barriers to CAM Research

Anthony Rosner, PhD, LLD [Hon.], LLC

Editor's note: In the last issue, we published "Drs. Meeker and Rosner Testify before White House Commission on CAM Policy." We were able to include much of what Dr. Meeker spoke on, but not on Dr. Rosner's presentation, as he was out of the country. Here now is Dr. Rosner's testimony before the White House Commission on Complementary and Alternative Medicine Policy, October 5-6, 2000.

Until 25 years ago, chiropractic research was vastly underdeveloped and appeared to some as an oxymoron. In 1975, a conference at the NINDS/NIH concluded: "There are little scientific data or significance to evaluate this [chiropractic's] clinical approach to health and to the treatment of disease."¹ From that time onward, both clinical and basic research have advanced to the point at which [i] over 40 randomized clinical trials comparing spinal manipulation with other treatments in the management of back pain have been published in the scientific literature,^2,3 [ii] meta-analysis and systematic reviews attesting to the support of spinal manipulation in the management of back pain^4,5 have also appeared, and [iii] multidisciplinary panels representing the governments of the United States,⁶ Canada,⁷ Great Britain,⁸ Sweden,⁹ Denmark,¹⁰ Australia,¹¹ and New Zealand¹² have expressed similar recognition of the robust evidence base in support of spinal manipulation for managing low back conditions.

Barriers

The efforts to launch and develop a National Center for Complementary and Alternative Medicine within the framework of the NIH are indeed admirable, taking the Center from a humble $2M annual budget in 1991 to one that approaches $70M today. This has taken place despite the comments of highly visible and influential individuals within the medical community to discredit alternative medicine in virtually any shape or form, a topic that I shall return to momentarily. Following are what I believe to be the most significant barriers to research efforts in alternative medicine, the barriers having either remained in place or only recently having been removed.

1. Collaborative Arrangements

Dating from the first RFA in March 1993, the Office of Alternative Medicine required that researchers in alternative medicine collaborate with people from an orthodox medical background, described as "individuals familiar with conventional research methodologies."¹³ The implication was that researchers in alternative medicine, having fewer resources and shorter bibliographies than their allopathic counterparts to begin with, were somehow less qualified to pursue research questions of any kind. With the lack of exposure to either the theory or practice of alternative medicine modalities, potential allopathic medical collaborators had to overcome gaps in knowledge and professional prejudices to become allied with alternative medical researchers, clearly delaying their efforts to launch research programs fundable from a federal point of view.

Furthermore, directing grant funding and their associated indirect costs toward allopathic medical centers, rather than specific institutions in alternative medicine, served to delay the building of the research infrastructure specifically within alternative medicine. NCCAM's establishment of specific research centers (including the chiropractic center at Palmer University) and its recent provision of RO1 programs (in which individual researchers in CAM may step forward as the PI on a fundable grant application) are major steps in overcoming this obstacle.

2. Domestic Institutions

A number of major milestones of research that have significantly lowered barriers to the practice and research of chiropratic have been accomplished abroad. The low-back studies of Meade (Great Britain),^14,15 the cervicogenic and tension headache studies of Nilsson (Denmark),^16,17 the first randomized clinical trial addressing colic - possibl a nonmusculoskeletal condition in infants - (Denmark),¹⁸ and numerous asthma case reports¹⁹ and a pilot for a randomized clinical trial from Australia²⁰ to lay the foundation for future clinical trials directed at this condition are but a few outstanding examples. Thus, the requirement of many past federal programs restricting grants to domestic institutions only represents a major impediment to the accomplishments and potential of research in alternative medicine, which recognizes no national boundaries and which has clearly benefited from the additional resources available beyond American borders.

3. Composition and Proceedings of Institutional Review Boards

Undoubtedly institutional review boards are an indispensable component of ensuring patient safety and knowledgability in a clinical trial.²¹ From this writer's firsthand experience, however, there have been instances in which a proposed randomized clinical trial within a major medical center has been rejected out-of-hand from what was probably the harboring of anti-chiropractic sentiments by the head of the IRB, who among other transgressions referred to this treatment alternative as "chiropracty." While implementing panels to monitor the behavior of IRBs may appear excessive, the issue does bear further scrutiny in the event that viable and safe alternatives in patients' interest fall victim to prejudice within an IRB.

4. Composition of Study Sections

For many of the same reasons as in the previous section, there must be an equitable number of individuals within each study section of a grant proposal who are familiar with and sensitive to the conduct of the therapeutic regimens to be tested. Common sense dictates that an eloquent and sympathetic presentation of the therapies to be studied be made to the study section as a whole. This writer recalls with dismay an egregious violation of this principle in the first round of alternative medicine applications reviewed by the OAM in 1993.Here, only a single member among the eight 17-member study sections drafted by the NIH was a chiropractor, who was unfortunately undistinguished as a researcher, and lacked the necessary background to provide constructive critiques of grant proposals in this field. Adding insult to injury was the fact that the OAM had been provided with the names of over 15 highly qualified and accomplished chiropractic researchers. Completing this sorry state of affairs was the fact that these sections were charged with reviewing over 400 applications, nearly half of which pertained to chiropractic. Providing properly balanced and enlightened study sections reviewing grant applications is clearly an absolute requisite for reducing significant barriers to the conduct of CAM research.

5. Publication Biases and Quotas

Examples can be introduced of editorial biases and quotas, which have prevented the most robust of chiropractic research from reaching necessary audiences. For instance, headache study by Boline,²² rated the highest in quality by two independently systematic literature reviews,^23,24 was denied publication at the New England Journal of Medicine, Headache, and Cephalalgia before finally appearing in the Journal of Manipulative and Physiological Therapeutics two years later. Examples exist in which editorial comments to the principal authors of studies have clearly indicated that obtaining negative results for spinal manipulation was the criterion for acceptability for publication.^25,26 Thus it is with dismay that this writer finds two inferior and widely-publicized studies in chiropractic which did get published in the New England Journal of Medicine,^27,28 extensive rebuttals of which have been published elsehwere.^29-32 Statements by two previous editors of the New England Journal of Medicine offer little encouragement, as they have been patently biased with little qualification in their negative assessments of alternative medicine.^33,34

Publication quotas likewise impede the dissemination, and therefore the incentive, to perform CAM research. The American Journal of Public Health, for example, allows the publication of but one chiropractic study per year based upon current membership. While subsidization of publication costs through membership is entirely appropriate, restricting access of information from a modality which has been experienced by ³⁷ percent of the American population at some point in their lives³⁵ appears arbitrary and draconian.

6. Distortion of Research Results in the Press and in the Journals

The crippling effects of bias and editorial policy of certain medical journals just discussed has ramifications in what is actually stated in papers and subsequently in the lay press. One study published in the New England Journal of Medicine, for instance, stated a conclusion that was far beyond anything supported by the data. Specifically, the study discouraged the routine referral of patients to chiropractic: "Given the limited benefits and high costs, it seems unwise to refer patients with low back pain for chiropractic or McKenzie therapy."²⁷ As egregiously out-of-bounds as a statement such as this is for a scientific journal, the lay press (to which the NEJM reportedly controls half of the health news we receive) only made matters worse. Such "scare" headlines as "Study Targets Worth of Chiropractic,"³⁶ and "Chiropractic Care Blasted in Two Studies"³⁷ only poisoned the atmosphere, inhibiting further research efforts and inducing third party payors to deny reimbursements for chiropractic services in which the outcomes have yet to be definitively disproved. News releases such as these need to be actively discouraged, and the public needs to be further enlightened as to the research and potential of multiple modes of alternative therapy - not just chiropractic.

7. Mainstream vs. Alternative Status of Chiropractic

Primarily due to the aforementioned research accomplishments regarding spinal manipulation and low-back pain,^2-7,14,15 chiropractic intervention has often been regarded as "mainstream" rather than alternative in the management of low-back pain,^8-12 and possibly at least for some types of headache as well.^16,17,22-24 However, in the treatment of asthma;^19,20,28,32 scoliosis;^38,39 otitis media;^40-42 infantile colic;^18,43 enuresis;⁴⁴ repetitive stress disorders;^45-48 dysmenorrhea^49-51 and premenstrual syndrome;^52-54 chronic pelvic pain;⁵⁵ GI dysfunctions;^56-58 and attention deficit disorder/hyperactivity,⁵⁹ chiropractic must be regarded as an alternative therapeutic approach. As a hybrid, chiropractic should be regarded as having important attributes of alternative medicine. Accordingly, it should not be dismissed as only a mainstream specialty ineligible for funding from sources that are supporting research in alternative medicine.

8. The Origins of Mainstream Medicine

As suggested in the preamble to the five-year strategic plan of NCCAM, "As CAM practices once considered unorthodox...are proven safe and effective by rigorous scientific investigation, they become part of mainstream health care." This is certainly the way one hopes to transform good research into practice and clearly represents the mission of both NCCAM and our Foundation. However, since only 15 percent of medical procedures have been reported to be supported by any documentation⁶⁰ and only one percent is considered to be scientifically sound,⁶¹ it is presumptuous to assume that what is currently accepted in standard medical procedures is intrinsically robust from a scientific point of view. Have large-scale clinical trials supported the use, of every variation of catheter used in angioplasty, for instance? One need only consult the Merck index of 100 years ago to realize that the following treatments -now woefully inadequate, outdated, and even dangerous -were unflinchingly accepted into the mainstream as de rigeur⁶² within at least some of our lifetimes:

formaldehyde for the common cold;
arsenic or ammonia for baldness;
opium and morphine for typhoid fever;
blood-letting and chloroform for streptococcal infections; and
strychnine, ice and lemon juice for diphtheria.

Thus, it is my belief that the idealism expressed regarding the origins of mainstream practices (in medicine or elsewhere) has to be tempered with realism. It is simply unreasonable to expect that every procedure and variation in health care delivery will be supported by a randomized clinical trial.

9. Intervention Paradigms

Perhaps most flagrantly illustrated by both studies published in the New England Journal of Medicine,^27,28 chiropractic must never be confused with merely high-velocity thrusting of the spine. Such is to reduce it to a one-dimensional specialty of cracking joints. Indeed, low-force contact procedures that have been incorrectly classified as placebos (shams)²⁸ have actually been shown to produce major improvements in both lung functional tests and patient symptoms with regard to asthma.⁶³

According to the preamble of the charter for the Council of Chiropractic Education, chiropractors are fully trained as a portal of entry primary care health service capable of complete diagnosis. The Council of Chiropractic Education has accrediting status with the U.S. Department of Education (since 1974) and the Council on Postsecondary Accreditation (since 1976). Among the therapeutic regimens for which chiropractors are licensed to administer are the use of hot and cold packs, electrical stimulation, soft tissue procedures (including trigger point therapy), and nutritional counseling.⁶⁴

Building upon preliminary studies appearing just this year,^65,66 more attention needs to be paid to long-term outcomes and supportive care. These attributes may differ somewhat from allopathic medicine's historical approach to disease management. The point is to emphasize the effects of interventions over the long term, which have the potential of forestalling or preventing far more invasive and expensive procedures which are substantially riskier for the patient. In terms of overall cost control and offering the possibility of reducing both the costs and morbidity of medication error-related deaths,^67,68 the implications of earlier intervention by alternative medical procedures are enormous.

10. The role of the RCT

There is no doubt that the RCT remains an important piece of the mosaic of evidence that needs to be assembled to substantiate a clinical procedure. However, it is certainly not the only piece, and in many instances in my experience, it has been overrated:

In a highly publicized randomized clinical trial regarding the use of chiropractic in managing asthma,²⁸ both the use of a highly invasive sham procedure (an inappropriate placebo) and the possibility of small sample sizes obscuring possible effects by a Type II error have led to misleading conclusions, let alone interpretation by the lay press.^36,37
Another highly visible clinical trial comparing three interventions in the management of acute low back pain²⁷ suffered from poor design³⁰ and inappropriate statistical procedures³¹ Worse, it implied that a single intervention represented chiropractic care such that its clinical relevance was highly questionable. Indeed, the Royal College of General Practitioners in a very recent systematic review of the literature designed to update the CSAG Guidelines of the United Kingdom8 has concluded that this trial neither adds nor detracts from the evidence base regarding appropriate interventions for low back pain.⁶⁹
A meta-analysis has shown that contrasting interpretations can be obtained, depending upon which of ²⁵ scales used to distinguish between high-quality and low-quality trials is actually employed.⁷⁰
A review of clinical trials comparing two antifungal agents has indicated that the apparent advantages of one of the instruments could have been obtained by manipulations of the design of most of the trials, in which the competing agent was inappropriately administered.⁷¹
The weight of evidence produced by clinical trials may be overcalculated due to the fact that the clinical trials are overrepresented as duplicate "sausage" publications by the same authors.^72-75
Methdological scores attached to clinical trials create a misleading profile of high-quality and low-quality studies if they place too much emphasis upon sham procedures - which we already know will seriously com-promise controlled studies involving physical methods such as spinal manipulation if they are not true placebos. In other instances, the mere utterance of such terms as "blinded" or "randomized" in the title of the paper cited may be sufficient to glean points in the rating of clinical trials - even though such terms are never defined or qualified. The proper remedy in this instance would be to demote the trial ratings if such terms are inappropriately used.⁷⁰

The point to realize here is that RCTs are subject to misinterpretation and outright abuse. Their generalization from a fastidious, defined laboratory setting is problematical. It is sometimes forgotten that the source of randomized clinical trials remains the sound, well-documented observations in the clinical setting. This has led epidemiologist than David Sackett to conclude that there are essentially two pillars of sound clinical evidence, only one of which is experimentally derived from the RCT:⁷⁶

"External clinical evidence can inform, but can never replace individual clinical expertise, and it is this expertise that decides whether the external evidence applies to the individual patient at all and, if so, how it should be integrated into a clinical decision."

In light of these many arguments, I would maintain that the WHCCAMP should place far greater emphasis upon cohort studies and case series in its research goals rather than assume categorically that they provide inferior guidance to clinical decision-making than RCTs. It should be quite clear from this discussion that a well-crafted cohort or case series is far more informative than a flawed or corrupted RCT.

Solutions

I would recommend that the WHCCAMP pursue the following:

1. Encourage all qualified researchers in alternative medicine to apply for federal and private grant support, taking into account refusals of attempted publication in peer-reviewed medical journals.

2. Encourage researchers from around the world, rather than only from American soil, to apply for such support;

3. Institute an appeals process if a research team believes that an institutional review board has turned down a research proposal unfairly;

4. Ensure that all study sections include a sufficient number of individuals that are familiar with and sensitive to the therapeutic regimens described within the proposal under review;

5. Frame and encourage legislation that encourages better communication of researchers and practioners of alternative medicine with the media, limiting the contacts with those journals found to harbor unjustifiable biases against alternative medical procedures.

6. Ensure that chiropractic is recognized as both a mainstream and an alternative intervention, depending upon the condition for which therapy is indicated. Accordingly, ensure that chiropractic is excluded from neither category in terms of grant eligibility and collaboration.

7. Encourage research directed at long-term outcomes and supportive care, areas that have commonly been neglected in allopathic medical care and which offer the possibility of low cost, preventive health management.

8. Ensure that chiropractic is not limited to referral-only specialty care limited to the back, based upon current accreditation, licensure, and research. In this regard, it is to be appreciated as a direct portal of entry for patient care-appreciating the ability of chiropractors to diagnose and apply treatments that are broader in scope than merely high-velocity thrusts.

9. Appreciate the limitations of randomized clinical trials, admitting well-designed and well-executed cohort and case studies into the evidence base supporting a given intervention.

10. Frame and encourage legislation that does not permit third-party payers to restrict reimbursements beyond the scope of practice currently stipulated by licensure laws within the states.

11. Coordinate activities with NCCAM and avoid duplication of efforts wherever possible.

12. Encourage private sources to invest in all types of alternative and mainstream medical research with adequate oversight. This would have the twofold benefit of offsetting the pharmaceutical industry's virtual monopolizing of the private support of medical research, as well as offering a variety of measures to reduce the chances compromising the quality of the research.

References

Goldstein M [ed]: Monograph No. 15. The Research Status of Spinal Manipulation. U.S. Department of Health, Education, and Welfare, Washington, DC, February 3-4, 1975.
Koes BW, Assendelft WJJ, van der Heijden GJMG, Bouter LM. Spinal manipulation for low-back pain: A updated systematic review of randomized clinical trials. Spine 21(24):2860-2871.
van Tulder M, Koes BW, Bouter LM. Conservative treatment of acute and chronic nonspecific low back pain: A systematic review of randomized controlled trials of the most common interventions. Spine 1997; 22(18):2128-2156.
Anderson R, Meeker WC, Wirick BE, Mootz RD, Kirk DH, Adams A. A meta-analysis of clinical trials of spinal manipulation. Journal of Manipulative and Physiological Therapeutics 1992; 15(3):181-194.
Shekelle PG, Adams AH, Chassin MR, Hurwitz EL, Brook RH. Spinal manipulation for low-back pain. Annals of Internal Medicine 1992;117(9):590-598.
Bigos S, Bowyer O, Braen G, et al. Acute Low Back Pain in Adults. Clinical Practice Guideline No. 14. AHCPR Publication No. 95-0642. Rockville, MD: Agency for Health Care Policy and Research, Public Health Service, U.S. Department of Health and Human Services. December 1994.
Manga P, Angus D, Papadopoulos C, Swan W. The Effectiveness and Cost-Effectiveness of Chiropractic Management of Low-Back Pain. Ottawa, Ontario, CANADA: Pran Manga & Associates, Inc., University of Ottawa, 1993, pp 65-70.
Rosen M. Back Pain. Report of a Clinical Standards Advisory Group Committee on Back Pain. May 1994, London: HMSO.
Commission on Alternative Medicine, Social Departementete. Legitimization for Vissa Kiropraktorer, Stockholm, SOU [English Summary] 1987;12:13-16.
Danish Institute for Health Technology Assessment: Low-back pain, frequency, management, and prevention from an HTA perspective. Danish Health Technology Assessment 1999;1(1).
Thompson CJ. Second Report, Medicare Benefits Review Committee, Canberra, AUSTRALIA: Commonwealth Government Printer, June 1986, Chapter 10 [Chiropractic].
Hasselberg PD. Chiropractic in New Zealand, Report of a Commission of Inquiry. Wellington, New Zealand: Government printer, 1979.
NIH Guide, March 26, 1993; 22(12).
Meade TW, Dyer S, Browne W, Townsend J, Frank AO. Low back pain of mechanical origin: Randomised comparison of chiropractic and hospital outpatient treatment. British Medical Journal 1990; 300:1431-1437.
Meade TW, Dyer S, Browne W, Townsend J, Frank AO. Randomised comparison of chiropractic and hospital outpatient management for low back pain: Results from extended follow-up. British Medical Journal 1995;311:349-351.
Nilsson N, Christensen HW, Hartvigsen J. The effect of spinal manipulation in the treatment of cervicogenic headache. Journal of Manipulative and Physiological Therapeutics 1997;20(5):326-330.
Bove G, Nilsson N. Spinal manipulation in the treatment of episodic tension-type headache. Journal of the American Medical Association, 1998;280(18):1576-1579.
Wiberg JMM, Nordsteen J, Nilsson N. The short-term effect of spinal manipulation in the treatment of in fantile colic: A randomized controlled trial with a blinded observer. Journal of Manipulative and Physiological Therapeutics 1999;22(8):517-522.
Jamison JR, McEwen AP, Thomas SJ. Chiropractic adjustment in the management of visceral conditions: a critical appraisal. Journal of Manipulative and Physiological Therapeutics 1992;15(3):171-180.
Hayek R, Ali S, Brice C. Asthma and chiropractic: An Australian trial [single blind cross over study]. Proceedings of the 1998 International Conference on Spinal Manipulation, Vancouver, British Columbia, July 16-19, 1998, pp. 72-74.
The National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. Institutional Review Boards. Washington, DC: U.S. Government Printing Office, 1978.
Boline P, Kassak K, Bronfort G, Nelson C, Anderson AV. Spinal manipulation vs. amiltriptyline for the treatment of chronic tension-type headaches: A randomized clinical trial. Journal of Manipulative and Physiological Therapeutics 1995;18(3):148-154.
Hurwitz EL, Aker PD, Adams AH, Meeker WC, Shekelle PG. Manipulation and mobilization of the cervical spine: A systematic review of the literature. Spine 21(15):1746-1760.
Kjellman GV, Skagren EI, Oberg BE. A critical analysis of randomised clinical trials on neck pain and treatment efficacy: A review of the literature. Scandinavian Journal of Rehabilitative Medicine 1999;31: 139-152.
Brennan P. Personal communication, 1992.
Nilsson N. Publication bias in the medical journals: An n of 1 study. Presentation at the 2000 International Conference on Spinal Manipulation, Bloomington, MN, September 24, 2000.
Cherkin DC, Deyo RA, Battie M, Street J, Barlow W. Comparison of physical therapy, chiropractic manipulation, and provision of an educational booklet for the treatment of patients with low back pain. New England Journal of Medicine 1998;339(14):1021-1029.
Balon J, Aker PD, Crowther ER, Danielson C, Cox PG, O'Shaugnessy D, Walker C, Goldsmith CH, Duku E, Sears MR. A comparison of active and simulated chiropractic manipulation as adjunctive treatment for childhood asthma. New England Journal of Medicine 1998;339(15):1013-1020.
Rosner AL. Response to the Cherkin article in New England Journal of Medicine. Dynamic Chiropractic November 2, 1998;16(23).
Chapman-Smith D. Back pain, science, politics and money. The Chiropractic Report November 1998; 12(6).
Freeman MD. A critical evaluation of the methodology of a low-back pain clinical trial. Journal of Manipulative and Physiological Therapeutics 2000;23(5):363-364.
Blum CL. Chiropractic and sacro-occiptal technique in asthma treatment. Chiropractic Technique 1999; 11(4):174-180.
Bunk S. Is integrative medicine in the future? Debate between Andrew Weil, M.D. and Arnold Relman, M.D. The Scientist 1999;13(10):1,10-11.
Angell M, Kassirer, JP. Alternative medicine - The risks of untested and unregulated remedies. New England Journal of Medicine 1998;339(12):839-841.
http://www.intersurvey.com, July 2000.
Haney DC. Associated Press, October 8, 1998.
Des Moines Register, October 8, 1998, p. 1.
Plaugher, G, Cremata EE, Phillips, R. A retrospective consecutive case analysis of pretreatment and comparative static radiological parameters following chiropractic adjustments. Journal of Manipulative and Physiological Therapeutics 1990;13(9):498-506.
Tarola GA. Manipulation for the control of back pain and curve progression in patients with skeletally mature idiopathic scoliosis: two case studies. Journal of Manipulative and Physiological Therapeutics1994;17(4):253-257.
Froehle RM. Ear infection: A retrospective study examining improvement from chiropractic care and analyzing for influencing factors. Journal of Manipulative and Physiological Therapeutics 1996;9(3):169-177.
Fallon J. The role of chiropractic adjustment in the care and treatment of 332 children with otitis media. Journal of Clinical Chiropractic Pediatrics 1997;2(2):167-183.
Degenhardt BF, Kuchera ML. Efficacy of osteopathic evaluation and manipulative treatment in reducing the morbidity of otitis media in children. Journal of the American Osteopathic Association 1994;94(8): 673.
Klougart N, Nilsson N, Jacobsen J. Infantile colic treated by chiropractors: a prospective study of 316 cases. Journal of Manipulative and Physiological Therapeutics 1989;12(4):281-288.
Reed WR, Beavers S, Reddy SK, Kern G. Chiropractic management of primary nocturnal enuresis. Journal of Manipulative and Physiological Therapeutics 1994;17(9):596-600.
Sucher B. Palpatory diagnosis and manipulative management of carpal tunnel syndrome. Journal of the American Ostoepathic Association 1994;94(8):647-663.
Strait BW, Kuchera ML. Osteopathic manipulation for patients with confirmed mild, modest and moderate carpal tunnel syndrome. Journal of the American Osteopathic Association 1994;94(8):673.
Davis PT, Hulbert JR, Kassak KM, Meyer JJ. Comparative efficacy of conservative medical and chiropractic treatments for carpal tunnel syndrome: A randomized clinical trial. Journal of Manipulative and Physiological Therapeutics 1998;21(5):317-326.
Bersten G, McCarthy K. Conservative chiropractic approaches to carpal tunnel syndrome. Topics in Clinical Chiropractic 1999;6(4):62-72.
Liebl NA, Butler LM. A chiropractic approach to the treatment of dysmenorrhea. Journal of Manipulative and Physiological Therapeutics 1990;13(3):101-106.
Boesler D, Warner M, Alpers A, Finnerty EP, Kilmore MA. Efficacy of high-velocity low-amplitude manipulative technique in subjects with low back pain during menstrual cramping. Journal of the American Osteopathic Association 1993;93(2):203-214.
Kokjohn K, Schmid DM, Triano JJ, Brennan PC. The effect of spinal manipulation on pain and prostaglandin levels in women with primary dysmenorrhea. Journal of Manipulative and Physiological Therapeutics 1992;15(5):279-285.
Stude DE. The management of symptoms associated with premenstrual syndrome. Journal of Manipulative and Physiogical Therapeutics 1991;14(3):209-216.
Walsh MJ, Chandraraj S, Polus BI. The efficacy of chiropractic therapy on premenstrual syndrome: a case series study. Chiropractic Journal of Australia 1994;24(4):122-126.
Walsh MJ, Polus BI. A randomized, placebo-controlled clinical trial on the efficacy of chiropractic therapy on premenstrual syndrome. Journal of Manipulative and Physiological Therapeutics 1999;22(9): 582-585.
Hawk C, Long C, Azad A. Chiropractic care for women with chronic pelvic pain: A prospective single group intervention study. Journal of Manipulative and Physiological Therapeutics 1997;20(2):73-79.
Falk JW. Bowel and bladder dysfunction secondary to lumbar dysfunctional syndrome. Chiropractic Technique 1990;2(2):45-48.
Wagner T, Owen J, Malone E, Mann K. Irritable bowel syndrome and spinal manipulation: a case report. Chiropractic Technique 7(4):139-140, 1995.
Pikalov A, Kharin VV. Use of spinal manipulative therapy in the treatmentof duodenal ulcer: A pilot study. Journal of Manipulative and Physiological Therapeutics 1994;17(5):310-313.
Giesen JM, Center DB, Leach RA. An evaluation of chiropractic manipulation as a treatment of hyperactivity in children. Journal of Manipulative and Physiological Therapeutics 1989;12(5):353-363.
Smith R. Where is the wisdom: The poverty of medical evidence. British Medical Journal 1991;303: 798-799.
Rachlis N, Kuschner C. Second opinion: What's wrong with Canada's health care system and how to fix it. Toronto: Collins, 1989.
Merck's 1899 Manual, or the Materia Medica. New York, NY: Merck & Co., 1899.

63. Field T, Henteleff T, Hernandez-Reif M, Martinez E, Mavunda K, Kuhn C, Schanberg S. Children with asthma have improved pulmonary functions after massage therapy. Journal of Pediatrics 1998;132(5): 854-858.
Haldeman S, Chapman-Smith D, Petersen, DM Jr. Guidelines for Chiropractic Quality Assurance and practice parameters. Proceedings of a Consensus Conference Commissioned by the Congress of ChiropracticState Associations. Mercy Conference Center, Burlingame, CA, January 25-30, 1992. Gaithersburg, MD: Aspen, 1993.
Rupert RL. A survey of practice patterns and the health promotion and prevention attitudes of U.S. chiropractors: Maintenance care: Part I. Journal of Manipulative and Physiological Therapeutics 2000; 23(1):1-9
Rupert RL, Manello D, Sandefur R. Maintenance care: Health promotion services administered to U.S. chiropractic patients age 65 and older: Part II. Journal of Manipulative and Physiological Therapeutics 2000;23(1):10-19.
Lazarou J, Pomeranz B, Corep P. Incidence of adverse drug reactions in hospitalized patients. Journal of the American Medical Association 1998;279(15):1200-1205.
Johnson JA, Bootman JL. Drug-related morbidity and mortality: A cost-of-illness model. Archives of Internal Medicine 1995;155:1949-1956.
Royal College of General Practitioners, unpublished update of CSAG Guidelines [reference 8], 1999.
uni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. Journal of the American Medical Association 1999;282(11):1054-1060.
Johansen HK, Gotzsche PC, Problems in the design and reporting of trials of antifungal agents encountered during meta-analysis. Journal of the American Medical Association 1999;282(18):1752-1759.
Rennie D. Fair conduct and fair reporting of clinical trials. Journal of the American Medical Association 1999;282(18):1766-1768.
Gotzsche PC. Multiple publication of reports of drug trials. European Journal of Clinical Pharmacology 1989;36:429-432.
Huston P, Moher D. Redudancy, disaggregation, and the integrity of medical research. Lancet 1996; 347:1024-1026.
Tramer MR, Reynolds DJM, Moore RA, McQuay HJ. Impact of covert duplicate publication on meta-analysis: A case study. British Medical Journal 1997;315:635-640.
Sackett DL. Editorial: Evidence-based medicine. Spine 1998;23(10):1085-1086.

Anthony Rosner,PhD
Director of Research and Education
Foundation for Chiropractic Education and Research
Brookline, Massachusetts

November 2000