How do you decide the best approach to a problem? There’s an ongoing debate in the fitness and therapy fields about what defines true evidence-based practice. I would define the two camps at the center of the debate as the scientists versus the experientialists.
There are those who view the literature as the ultimate standard, while others contend years of gym experience and time in the trenches outweighs the investigations of white-coated academics. If nothing else, the debate raises an important point: not all evidence is of equal merit.
In this article we’re going to examine several different types of evidence so that you can understand the value of each and how to best come to your own conclusions.
There are many ways to gain information on the path to forming your own conclusions.
You Got a Source for That?
“Scientific evidence” is a broad term. In the most general sense, it means evidence discovered through application of the scientific method. In common usage, most people see scientific evidence as research studies performed in an academic laboratory or clinical setting.
There are several types of studies with varying degrees of validity:
Observational Studies: These are studies of a pre-existing group tied together by a variable, like patients undergoing a specific treatment or a group of people using the same training program. Data is collected and then compared to a group where the variable isn’t present. There are several approaches to this (case series, case control, cohort, etc.) but they all fall into the realm of observational studies. They give us ideas for future research, but generally lack the ability to provide specific and conclusive statements. They’re far more common in the medical field where subjects can be observed in an inpatient setting and medical histories can be used to gather data.
Controlled Trials: In these, the inclusion criteria and data collection are very specific in order to limit the number of variables affecting the outcomes. Fewer variables translates into more reliable data interpretation. Randomized controlled trials are even better because they prevent bias in both the researchers and the subjects. While controlled trials are usually more conclusive than an observational study, their practical implications are limited due to the necessary specificity of their design. The same limitation of variables that strengthens the study’s conclusions ultimately limits its real-world value.
Literature Review: This is a broad, inclusive look at a certain topic that pulls together all of the available research to answer a specific question. A meta-analysis is a statistical method applied to the data of multiple studies that may provide insights not present when the individual papers are viewed in a vacuum. Literature reviews and meta-analyses constitute the gold standard of practical interpretation because they use individual trials as building blocks to a more complex understanding.
Beyond the surface stuff like study structure, there are a few other things to consider. Not all journals carry equal merit and it’s become a somewhat common marketing tactic for supplement companies to finance the creation of journals to give their self-funded and directed research an elevated appearance of validity. This is usually pretty obvious. As they say, follow the money.
“Not all journals carry equal merit and it’s become a somewhat common marketing tactic for supplement companies to finance the creation of journals.”
Despite the place it occupies in the cultural hive-mind, it’s important to remember that even peer-reviewed research is far from infallible. A recent expose of the junk-science industry involved a pair of journalists who conspired with a group of doctors and researchers to publish a poorly designed study with questionable conclusions.
The result? Large media outlets worldwide ate up this peer-reviewed, journal-published study and used it as the basis for hundreds of even more questionable advice columns. But not all peer-reviewed research is meaningless. The take-home point is simply that peer-reviewed status, by itself, doesn’t always mean a study has merit. No more and no less.
Anecdotal Evidence Isn’t Valid, Bro
Despite what Internet comment sections would have you believe, anecdotal and experiential evidence is not the same thing. An anecdote is quite literally a story with a questionable basis in reality. One of the definitions of anecdote is, “an account regarded as unreliable or hearsay.” Anecdotal evidence is when some guy you workout with says, “My buddy accomplished [X] and he always does [Y].” The story contains no real objective understanding of the variables involved. All you’re going on is that someone said it’s true.
“If you know a guy at the gym who did a thing and and got some results, then it must be true!”
Proper experiential evidence has more in common with peer-reviewed research than it does anecdotes. Applying the scientific method doesn’t require that you have a PhD or a strong publication record. All you need is the ability to objectively analyze data presented to you by experience.
This is where most of the trouble with experiential evidence lies. It’s not so much that experiential evidence is innately flawed; it’s that when someone tells you about his or her experience, you don’t have a good method of evaluating that process. It’s possible he or she is an eminently practical and scientific individual. It’s also possible he or she is painfully subjective and incapable of moving past personal biases. Both types of individuals can provide experiential evidence, but one is valuable and the other is not.
What Is Authority?
What this whole debate really comes down to is the question of authority. When deciding on best practices and an ideal approach to a problem, you want to base your methods on something authoritative. But authority isn’t a real thing. It may seem a bit esoteric, but authority is a mental construct, not an externally valid quality. Authority is a status we give others, not something they claim for themselves.
Mental models are properties of our perspective on the world. They organize the data in our surrounding reality into a way we can understand. They are important to us because we only have a certain amount of mental and energetic resources in a given day. Mental models allow us to interpret and use immense amounts of data without having to invest our own resources in the process. They are based on a lot of different things, some on belief and others on science, but ultimately their main concern is one of practicality – they should, at least theoretically, enable us to interact with the world better than we could without them.
“Authority provides us with anchors for our perspective. Science is great for telling us where the dartboard is.”
Viewed through the lens of mental models, it becomes rapidly apparent why we care so much about authoritative sources of information. Authority provides us with anchors for our perspective. Science is great for telling us where the dartboard is. It narrows the options and orients us to the best possible starting point when approaching a new problem.
From there, experiential evidence will often outweigh the significance of scientific research. Intermittent fasting may not be conclusively proven to accelerate weight loss when weekly workload and caloric intake are equalized, but if intermittent fasting is the first dietary approach my client has been able to use effectively and it’s not detrimental to his or her health, then I don’t really give a shit what biochemical pathways are or aren’t involved.
Science Is Not Religion
Science is not a religion. It’s not meant to provide a static and infallible view of the way everything works. It simply gives us a systematic, testable approach to evaluating and developing better mental models.
“You have a source for that?” While a totally valid question, this inquiry has also become the go-to tactic for many of us when we don’t agree with something. The next time you feel yourself calling out for sources, take a moment and ask yourself if you are upset because you think the method in question doesn’t work or are you upset because its effectiveness challenges you to reexamine your perspective?
Don’t be a scientific fundamentalist. Science is meant to empower us to do more. It should never be invoked to limit us from trying.
Check out these related articles:
