When medical treatments are the subject of dozens or hundreds of studies - some with conflicting results, others using questionable methodology - how can non-specialists make rational decisions on what works and what doesn't?
The Cochrane Collaboration is a worldwide initiative to make sense of the huge mass of clinical studies available. As a highly distributed collaboration to systematically review all relevant treatment experience, it has intriguing similarities and differences to better-known collaborations like Wikipedia.
From the Newcomer's Guide:
"Cochrane Reviews are systematic assessments of evidence of the effects of healthcare interventions, intended to help people to make informed decisions about health care, their own or someone else's. Cochrane Reviews are needed to help ensure that healthcare decisions throughout the world can be informed by high quality, timely research evidence."
Here's more on the rationale behind Cochrane:
"For all but the last century, decisions on how to treat patients were almost always based on personal experience, anecdotal case histories and comparisons of a group of patients who received one treatment with an entirely separate group who did not. These processes, although subject to many biases, are still in use today but the randomised trial has provided a research design that will provide much more reliable estimates of the relative effects of interventions. In a randomised trial, the only difference between the patients in the groups being compared is that of most interest: the interventions under investigation. Thus, any differences in the outcomes of the patients in the groups being compared will be due to either the interventions they were allocated to receive, or the chance variations that will always exist between groups of people.
In part because of these chance variations, the results of a single trial will rarely be sufficient in many circumstances. Most trials are too small and their results are not sufficiently robust against the effects of chance. In addition, small trials might be too focused on a particular type of patient to provide a result that can be either easily or reliably generalised to future patients. Added to this, the amount of information about health care, including that coming from individual randomised trials, is now overwhelming. Vast amounts of information are readily available in books, magazines, the media and, of course, on the Internet. However, people making decisions about health care - including patients, health care professionals, policy makers and managers - need high quality information and, unfortunately, much of what is available is of poor quality. As a consequence, vast resources are wasted each year on health care that is not effective and may even be harmful, effective forms of care are often under utilised, and people sometimes suffer and die unnecessarily.
To help overcome these barriers to better health care, and to provide a key piece of the evidence needed for this, results from similar randomised trials need to be brought together. Trials need to be assessed and those that are good enough can be combined to produce both a more statistically reliable result and one that can be more easily applied in other settings. This combination of trials needs to be done in as reliable a way as possible. It needs to be systematic...
The Cochrane Collaboration is supported by hundreds of organisations from around the world, including health service providers, research funding agencies, departments of health, international organisations and universities. There are currently more than 10,000 people contributing to the work of The Cochrane Collaboration from over 80 countries, and this involvement continues to grow. The number of people involved has increased by about 20% year on year in the last five years."
Of course, there is always room for improvement.
For complex problems, it can be a challenge to decide which solutions work - even when enough evidence exists, analyzing that evidence overwhelms individual capabilities. So just thinking rationally about subtle aspects of our world may depend on developing new massively-collaborative social structures.
This approach could be useful in other fields. There is now a Campbell Collaboration: "...a non-profit organization that aims to help people make well-informed decisions about the effects of interventions in the social, behavioral and educational arenas." If these collaborations successfully adapt as they approach more politicized areas, they could be models for evidence-based policy making in the future.
