Read the full transcript of Dr Karen Dawe’s talk titled “3 Questions To Ask Before Buying Into Health Trends” at TEDxBristol 2024 conference.
Listen to the audio version here:
TRANSCRIPT:
Why Most Public Research Findings Are False
DR. KAREN DAWE: That’s the title of a paper that came out when I was roughly halfway through my PhD. I spent about 18 months failing to get any of my experiments to work, and I was ready to give up. But then I realized, to an extent, that’s just how science works.
Science investigates things nobody has investigated before. It’s trial and error. So studies start small, maybe too small, and what looked like a really interesting finding was actually just a fluke. “Most” is maybe an exaggeration, but certainly some published scientific findings are wrong.
The Problem with Accessible Health Science Information
What does that mean for you? You currently have access to more health science information than ever before, and more ways to consume it, like TikTok, Facebook, YouTube, and I love that. So what’s the problem? Well, the problem is, it might just be wrong.
We all know that feeling of seeing a headline that really resonates, but maybe sounds a bit too good to be true. How can you tell? I’m a scientist and health technologist. What I do is help companies compile scientific evidence to support their product, and then figure out how reliable that evidence is.
I work with medical devices for drug delivery. I work with software to help you get better sleep. I work with medicines for the most rare and severe types of seizures. I am not an expert in any of those fields, and you don’t need to be either.
The Science of Science
What I do is the science of science. The science of science is a thing, and its principles can help you judge health science information in the media. So some scientific findings are wrong. What do we do about it?
Well, science self-corrects, and one of the main ways it does that is through something called a systematic review. Now, the origins of the systematic review are maybe not what you’d expect. In the early 1930s, a young medical student called Archie Cochrane had a problem. In interviews, he refers to it as a “sexual dysfunction.” It’s the 1930s, you’ve got a sexual dysfunction.
Archie did what any one of us would do. He went to Vienna to find Sigmund Freud. What Archie didn’t know then was that he had a genetic condition, not a psychological one. Psychoanalysis didn’t help, and he became very frustrated. He was seeing one of Freud’s students, and he says he “believes strongly in intuition.” “Well, I couldn’t help asking, why? What’s the evidence for what you’re saying?”
This experience in Vienna really got Archie thinking about evidence, and by the time he returned to medical school, he was kind of a troll. He says, “We started a sort of a club of individuals who would ask consultants, ‘What’s the evidence that your treatment has any effect?’ We found this great fun. There was so little evidence available.“
By the time he left medical school, he was obsessed with evidence, and it came to define his medical career. Among his many great contributions was the idea that single studies can’t be relied upon to make healthcare decisions. He called for research summaries to be produced that compile all the available evidence on a specific question. This is what we now know as a systematic review, and they are a required part of the treatment approval process.
The most robust type of systematic review is called the Cochrane review. And Cochrane reviewers judge how good the evidence is using a tool which is essentially a huge technical questionnaire, and no big deal, but it was developed right here in Bristol, used globally, not just for Cochrane reviews, and it was co-created by the team that I used to work for.
So I kind of felt that I was okay to mess with it a little bit. What I wanted to do was take it out of the hands of the professionals and turn it into something anyone can use to judge health science information in the media.
Three Questions to Ask
So what I did was take the core principles of Cochrane and boil them down to just three questions. Three questions that I want you to remember and think about the next time. Don’t worry, I remember them. If they go, it’s fine.
And you’re going to remember them too. I want you to remember them because I want you to think about them the next time you see a too-good-to-be-true headline or a weirdly compelling TikTok. Okay, so:
Has there been a study done in humans?
Are we sure it’s not a red herring?
Has there been a randomized controlled trial?
If the answer to any of these questions is no, then it’s not bad science, not by any means, but it does mean there’s a high risk that in time it’ll turn out that the findings weren’t reliable.
Before we get into it, we need to be aware that it will take conscious effort because we want to believe. I want to believe. I have an autoimmune disease. I’m sick a lot. I’m on immunosuppressant medication. There’s no cure. And the plan is for me to be on immunosuppressant medication for the rest of my life.
So when I see a TikTok that says it’s definitely going to cure me, I want to believe. I want it to be true. I don’t want to question it. So it takes conscious effort to stop, think, and ask, “Has there been a study done in humans? Are we sure it’s not a red herring? Has there been a randomized controlled trial?” This first one is super easy and a real killer question.
Studies Done in Humans
And once you start asking this question and listening out for the answer, it’ll filter out a lot of health claims.
All medical breakthroughs have to start somewhere. And a lot of them start by growing cells in Petri dishes in a lab. But that doesn’t mean that all cells in Petri dishes are destined to become great medical breakthroughs.
A really good way to remember this is that cancer cells grown in a dish in a lab can be very effectively killed with a hammer. All kinds of things will kill cancer cells in Petri dishes, but that doesn’t make them a viable cure for cancer. And we know this because 90% of candidate drugs will fail when they get to the clinical trial stage. Mostly this is because once we put them in real life humans, they just don’t work in the way the evidence suggested they would.
So if we ask this question, “Has there been a study done in humans?” And the answer is no, we can say, “Okay, that’s super interesting, but maybe it’ll turn out to not be a reliable finding.” If we ask this question, “Has there been a study done in humans?” And the answer is yes, we’re going to take it up a notch.
Red Herrings and Confounding
We’re going to say, “Are we sure it’s not a red herring?” In the context of medical research, what we’re talking about here is confounding. Confounding happens when something is related to both the health factors that we’re interested in and the health outcome causally related to both those things. It’s quite a tricky concept.
I’m going to give you an example that’s going to change the way you listen to health science information in the media. Nice graph here, very clear relationship between whatever’s on X and whatever’s on Y. Looking at this graph, the more X happens, the more Y happens. So if Y is something bad, this graph suggests we want to stop doing X.
This is a real relationship. It is not a chance correlation. But what’s key is the nature of this relationship. So let’s see what our variables are.
On our Y axis, we’ve got total cost of fire damage. And on our X axis, we’ve got number of firefighters at the scene. This is a real relationship, but it is not a direct causal relationship. It’s a red herring. Fire is the culprit. Fire causes both the total cost of fire damage and the number of firefighters at the scene.
Now, we know what firefighters do. We understand that system. We often don’t know what health factors do. And we don’t necessarily understand the underlying system. This is why it’s so important to ask, “How was the study done? What was the study design?”
Randomized Controlled Trials
So what study design would avoid red herrings confounding? That would be randomized controlled trials. Last question, “Has there been a randomized controlled trial?” Randomization is the key to getting rid of red herrings, because then the red herrings occur as much in the treatment group as they do in the control group.
At the end of the first year of World War II, Archie Cochrane found himself as a prisoner of war. As a medical officer, he was in charge of the health of around 8,000 British prisoners. Their diet was awful. Archie noticed everyone, including himself, was beginning to swell.
They were getting very, very sick indeed. Archie suspected it was thiamine deficiency. But Archie is a prisoner of war. He is not a doctor in a hospital. He can’t just order up a bunch of yeast, which is what they need. Archie needs to convince the guards beyond all doubt that yeast is the treatment they need. Archie manages to get a small amount of yeast on the black market. He numbers the soldiers and he sends the odd numbers off to the treatment group, the even numbers off to the control group.
The treatment group get yeast, the control group get a placebo, but it was vitamin C. He might have been tempted to give the sicker patients the yeast, but then they’re not randomized and then yeast isn’t the only difference between the groups. What if the yeast doesn’t help because the sicker patients are too sick? And what if the sicker patients are sick because they also smoke?
What if the sicker patients are more sick because of some fundamental genetic difference? When you’re a prisoner of war trying to convince guards to buy you yeast, there can be no “what ifs.” And when we are spending money and resource on thousands of very sick patients in clinical trials, there can be no “what ifs.”
Within three days, there was a noticeable difference between the groups. Archie wrote up his findings and told the guards. The next day, he woke up to heaps of yeast. The whole camp got better because of what was essentially one of the first ever randomized controlled trials. It was even blinded.
The patients didn’t know if they were getting the treatment or the placebo. And this is another really important thing about randomized controlled trials. The expectations and hopes of both the patients and the researchers impact results. In trials that aren’t double blinded, the treatment effect can be exaggerated by as much 17%.
Ask Tough Questions
Archie Cochrane changed the face of medicine by not accepting what he was being told, by asking tough questions, even of people of authority and experience. Ask tough questions. Even if people are wearing a white coat in their video, even if they have the word “doctor” in their username, even if they are stood on a red dot giving a TEDx talk.
Ask tough questions. The best advice I can give you is that if you see a too good to be true headline or a compelling TikTok, go to the Cochrane online database. I’m not affiliated with Cochrane, by the way, I should just say. But at this time, they are considered the gold standard in healthcare evidence and they are used globally to make policy and healthcare decisions.
They may have already asked the tough questions for you. But if there isn’t a review already on what you’re interested in, we can all be more like Archie by asking the tough questions ourselves. Has there been a study done in humans? Are we sure it’s not a red herring?
Has there been a randomized controlled trial? Change how you listen to this information. Because if you’re listening to something really interesting and the answer to all these questions is yes, then you might want to pay attention. Thank you very much.
