There are those who view the literature as the ultimate standard, while others contend years of gym experience and time in the trenches outweighs the investigations of white-coated academics. If nothing else, the debate raises an important point: not all evidence is of equal merit.

In this article we’re going to examine several different types of evidence so that you can understand the value of each and how to best come to your own conclusions.

There are many ways to gain information on the path to forming your own conclusions.

You Got a Source for That?

“Scientific evidence” is a broad term. In the most general sense, it means evidence discovered through application of the scientific method. In common usage, most people see scientific evidence as research studies performed in an academic laboratory or clinical setting.

There are several types of studies with varying degrees of validity:

Observational Studies: These are studies of a pre-existing group tied together by a variable, like patients undergoing a specific treatment or a group of people using the same training program. Data is collected and then compared to a group where the variable isn’t present. There are several approaches to this (case series, case control, cohort, etc.) but they all fall into the realm of observational studies. They give us ideas for future research, but generally lack the ability to provide specific and conclusive statements. They’re far more common in the medical field where subjects can be observed in an inpatient setting and medical histories can be used to gather data.

Controlled Trials: In these, the inclusion criteria and data collection are very specific in order to limit the number of variables affecting the outcomes. Fewer variables translates into more reliable data interpretation. Randomized controlled trials are even better because they prevent bias in both the researchers and the subjects. While controlled trials are usually more conclusive than an observational study, their practical implications are limited due to the necessary specificity of their design. The same limitation of variables that strengthens the study’s conclusions ultimately limits its real-world value.

Literature Review: This is a broad, inclusive look at a certain topic that pulls together all of the available research to answer a specific question. A meta-analysis is a statistical method applied to the data of multiple studies that may provide insights not present when the individual papers are viewed in a vacuum. Literature reviews and meta-analyses constitute the gold standard of practical interpretation because they use individual trials as building blocks to a more complex understanding.

Beyond the surface stuff like study structure, there are a few other things to consider. Not all journals carry equal merit and it’s become a somewhat common marketing tactic for supplement companies to finance the creation of journals to give their self-funded and directed research an elevated appearance of validity. This is usually pretty obvious. As they say, follow the money.

“Not all journals carry equal merit and it’s become a somewhat common marketing tactic for supplement companies to finance the creation of journals.”

Despite the place it occupies in the cultural hive-mind, it’s important to remember that even peer-reviewed research is far from infallible. A recent expose of the junk-science industry involved a pair of journalists who conspired with a group of doctors and researchers to publish a poorly designed study with questionable conclusions.

The result? Large media outlets worldwide ate up this peer-reviewed, journal-published study and used it as the basis for hundreds of even more questionable advice columns. But not all peer-reviewed research is meaningless. The take-home point is simply that peer-reviewed status, by itself, doesn’t always mean a study has merit. No more and no less.

Anecdotal Evidence Isn’t Valid, Bro

Despite what Internet comment sections would have you believe, anecdotal and experiential evidence is not the same thing. An anecdote is quite literally a story with a questionable basis in reality. One of the definitions of anecdote is, “an account regarded as unreliable or hearsay.” Anecdotal evidence is when some guy you workout with says, “My buddy accomplished [X] and he always does [Y].” The story contains no real objective understanding of the variables involved. All you’re going on is that someone said it’s true.

“If you know a guy at the gym who did a thing and and got some results, then it must be true!”

Proper experiential evidence has more in common with peer-reviewed research than it does anecdotes. Applying the scientific method doesn’t require that you have a PhD or a strong publication record. All you need is the ability to objectively analyze data presented to you by experience.

This is where most of the trouble with experiential evidence lies. It’s not so much that experiential evidence is innately flawed; it’s that when someone tells you about his or her experience, you don’t have a good method of evaluating that process. It’s possible he or she is an eminently practical and scientific individual. It’s also possible he or she is painfully subjective and incapable of moving past personal biases. Both types of individuals can provide experiential evidence, but one is valuable and the other is not.

What Is Authority?

What this whole debate really comes down to is the question of authority. When deciding on best practices and an ideal approach to a problem, you want to base your methods on something authoritative. But authority isn’t a real thing. It may seem a bit esoteric, but authority is a mental construct, not an externally valid quality. Authority is a status we give others, not something they claim for themselves.

Mental models are properties of our perspective on the world. They organize the data in our surrounding reality into a way we can understand. They are important to us because we only have a certain amount of mental and energetic resources in a given day. Mental models allow us to interpret and use immense amounts of data without having to invest our own resources in the process. They are based on a lot of different things, some on belief and others on science, but ultimately their main concern is one of practicality – they should, at least theoretically, enable us to interact with the world better than we could without them.

“Authority provides us with anchors for our perspective. Science is great for telling us where the dartboard is.”

Viewed through the lens of mental models, it becomes rapidly apparent why we care so much about authoritative sources of information. Authority provides us with anchors for our perspective. Science is great for telling us where the dartboard is. It narrows the options and orients us to the best possible starting point when approaching a new problem.

From there, experiential evidence will often outweigh the significance of scientific research. Intermittent fasting may not be conclusively proven to accelerate weight loss when weekly workload and caloric intake are equalized, but if intermittent fasting is the first dietary approach my client has been able to use effectively and it’s not detrimental to his or her health, then I don’t really give a shit what biochemical pathways are or aren’t involved.

Science Is Not Religion

Science is not a religion. It’s not meant to provide a static and infallible view of the way everything works. It simply gives us a systematic, testable approach to evaluating and developing better mental models.

“You have a source for that?” While a totally valid question, this inquiry has also become the go-to tactic for many of us when we don’t agree with something. The next time you feel yourself calling out for sources, take a moment and ask yourself if you are upset because you think the method in question doesn’t work or are you upset because its effectiveness challenges you to reexamine your perspective?

Don’t be a scientific fundamentalist. Science is meant to empower us to do more. It should never be invoked to limit us from trying.

Check out these related articles:

• Don’t Just Argue, Debate Like a Scientist
• A Case on How Supplement “Research” Actually Works
• How to Read and Analyze Research Like a Pro

The post PubMed and Bro-Science: How to Use Evidence in Fitness appeared first on Breaking Muscle.

With these contradictory results, it would be easy to give up on science. But I’m not sure science is to blame. Each study gives us a clue to the final answer. Here are a few thoughts to shed light on these contradictions and teach us to take a bird’s-eye view of the issues at play.

We are constantly bombarded with the latest scientific discoveries. One article describes a set of scientific results. Then three months later, another article gives opposite results. What do you do?

With these contradictory results, it would be easy to give up on science. But I’m not sure science is to blame. Each study gives us a clue to the final answer. Here are a few thoughts to shed light on these contradictions and teach us to take a bird’s-eye view of the issues at play.

Air Conditioning Causes Obesity

In a 2006 study, David Allison and colleagues described a correlation between obesity and air conditioning use.

People in warmer climates who used air conditioning were heavier than those who did not. In the southern United States, the percentage of homes with air conditioning increased from 37 percent in 1978 to 70 percent in 1997. The rates of obesity rose at a similar rate.

The media headlines claiming air conditioning was making us fat exploded. Articles pointed to the lack of obesity in old Italian villages without air conditioning. It seemed we found a solution to this large problem (pun intended).

But Wait: Cold Burns More Calories

Before you throw your air conditioner out the window, let me explain the next theory. Ray Cronise, a former NASA scientist, wondered how Michael Phelps could eat 12,000 calories a day without becoming obese.

He hypothesized that it was because Phelps spent three hours a day in cold water, so his body had to burn more calories to keep him warm. If this hypothesis were true, it would suggest we could burn more calories by exposing ourselves to cold.

Cronise tested it out on himself by taking cold showers, walking three miles in the winter cold with just a t-shirt, and sleeping without covers. He lost 27 pounds in his three-month experiment. (For a more eloquent explanation, see Tim Ferriss’ 4 Hour Body.)

Which theory is right? Let me save my answer for the end. In the meantime, stick with me as I bring up another confusing question.

Intermittent Fasting

Bodybuilding-style diets where people eat six meals a day have been popular for years. The theory is that by eating small meals, the body’s insulin never spikes and we don’t store as much fat. Many people have been successful, and this type of diet is still prescribed and practiced today.

On the other hand, intermittent fasting has also become popular. (Ori Hofmekler wrote about this topic quite a while ago.) In this diet, you refrain from eating for a specified amount of time and limit your daily meal window to a few hours.

This theory is related to our animal ancestry where we hunted for hours and then binged when we were successful on the hunt. You will lose fat in this diet because you are in fat-burning mode for most of the day. Many people report exciting results.

Individual Differences

Each person’s metabolism and digestion is different. Some of us have a gene that produces more alpha-amylase, which breaks down starches when they enter our mouth.

These lucky individuals can eat more carbohydrates and not gain weight. Individual differences continue down the digestive tract. We also have different gut biomes, so we each digest our food differently.

The process also changes depending on what we eat. Vegetarians and carnivores have different biomes. If they were to switch their meat eating preferences, their bodies would adapt by changing the gut biome.

I wrote an article explaining why you are not normal and that is a good thing. Think about two people approaching a deadlift. The person with a short torso and long arms will have a much easier time than a person with T. rex arms. But the person with T. rex arms will be great at bench pressing. Each person has some advantage over the other.

The bottom line is that there might not be one right answer for everyone, but there is an answer that is right for you.

Multiple Contributing Factors

We often look for simple solutions to our problems. Studies tend to isolate one causal mechanism. It is more likely that there are multiple contributing causes. In the deadlift, we can think of many factors that might affect our performance, such as muscle weaknesses, imbalances, tightness, previous injuries, and body structure.

So why would we think the cure for obesity is as simple as turning off air conditioning or eating a certain type of food? There are multiple causal mechanisms that work together.

“We have improved our lives with air conditioning and other conveniences. These changes could be the causal factors, either individually, or more likely, in combination.”

A common mantra from research design class is, “Correlation does not equal causation.” I often use the example of religious institutions and crime. As the number of religious institutions in a city increases, the number of crimes also rises.

Does this mean participating in religion causes more people to become criminals? Or do religious institutions come to cities to reduce crime? It is probably neither. Rather, larger cities have more crime and more religious institutions. These two variables are only related because of the city’s size.

Getting a Clue From Science

The same concept holds for the air conditioning theory. We have improved our lives with air conditioning and other conveniences. These changes could be the causal factors, either individually, or more likely, in combination.

My take on the air conditioning question is that our bodies are not exposed to temperature variability. If we live with our thermostat at a constant temperature, our bodies may not know the seasonal changes. It might be good for us to expose ourselves to more extreme temperatures.

So what about the theory that cold burns more calories? At the cellular level, exposing your body to cold leads to the growth of brown adipose tissue (BAT) fat cells. These fat cells regulate temperature and help the body to burn more calories in cold.

Currently, I practice sleeping without covers and walking in the cold without a jacket. In the summer, I try to leave the air conditioning off. I may be on the wrong path, but I base my decisions on clues from our current scientific knowledge base.

Take a Bird’s Eye View

By taking a bird’s-eye view of these studies, we can make more sense of possible answers. But we always have to live with uncertainty. Studies may point us in opposite directions at times, so we need to see what puzzle pieces we have and how they fit together. Some puzzles, like nutrition, have many complicated pieces that don’t always fit. Coaching a deadlift is much simpler.

As I receive more information, I might have to adjust. One study does not lead to a complete change in my behavior. Rather, it forces me to fit this new piece into my existing knowledge. Take a step back and try to see the whole puzzle. Gather as many pieces as possible to determine how it all fits together.

You’ll also enjoy:

• Non-Exercise Activities and Calorie Burning: Good or Bad?
• Ice Water Immersion Improves Performance
• No Time to Lose Weight? Simple Options for Daily Calorie Burning

References:

1. Ferriss, Timothy. The 4 Hour Body: An Uncommon Guide to Rapid Fat Loss, Incredible Sex and Becoming Superhuman. 1st Edition. New York: Harmony, 2010.

2. Hofmekler, Ori, Harvey Diamond, and Udo Erasmus. The Warrior Diet: Switch on Your Biological Powerhouse For High Energy, Explosive Strength, and a Leaner, Harder Body. 2 Rev Exp edition. Berkeley, Calif: Blue Snake Books, 2007.

3. Keith, S. W., D. T. Redden, P. T. Katzmarzyk, M. M. Boggiano, E. C. Hanlon, R. M. Benca, D. Ruden, et al. “Putative Contributors to the Secular Increase in Obesity: Exploring the Roads Less Traveled.” International Journal of Obesity 30, no. 11 (June 27, 2006): 1585–94. doi:10.1038/sj.ijo.0803326.

Photo courtesy of Shutterstock.

The post How to Deal With Confusing Science appeared first on Breaking Muscle.

But these fields lag behind other important sciences where studies report how much or how well an intervention worked. Applying this important distinction can provide us with a great deal more information when we are making health decisions.

Cargo Cult Science

Richard Feynman famously described a cargo cult of islanders in the Pacific during World War II. This group of islanders saw the U.S. military plow down trees, build a runway, and erect an air traffic control center. Following all of this, planes started “falling” from the sky bringing gifts.

The islanders thought they could follow the same process to get the same results. They built control towers out of bamboo, put fires along the runway, and even a created bamboo headsets for the ground crew. But no planes came. Feynman described this story as a metaphor for many “so-called sciences.” They try to do the same things as “real sciences,” but no matter what they do, the real science does not land.

“If the fitness industry wants to tout how its programs are scientifically based, then it would be helpful for the cargo cult science that makes up much of fitness literature to move forward to more modern techniques of science.”

This story came to mind as I was reading through a popular strength and conditioning journal. Much of what is published looks like science and feels like science, but it is missing something. For example, I originally intended to review a research article that showed velocity in a three-minute max run is associated with other outcomes of anaerobic capacity. Everyone reading this article could have simply guessed those outcomes are associated. What we need to know is how much or how important the finding is to us.

Does It Work?

A big problem in sports and conditioning research is that we rely greatly on null hypothesis testing. Null hypothesis testing is a set of statistical techniques that test the probability that there is no effect (i.e., the null hypothesis). The goal is to knock down that null hypothesis and say, “We think we found something.” It is a yes-or-no type of statement that tells us if something is good or bad for us.

A better step would be to tell us how much better an exercise or a nutritional change is for us. Measures called effect size estimates can better get at the question of “how much?” rather than “does it work?”

Other scientific studies report how much or how well an intervention worked.

Examples

Let’s take the example of aspirin to prevent heart attacks later in life. Physicians often recommend aspirin as previous research has shown it to be effective in preventing heart attacks (answering the “does it work” question). In 2010, my colleague and I calculated the results a different way. We found that 176 patients would have to take aspirin on a daily basis for one patient to avoid experiencing a major cardiovascular event, and 208 patients would have to take it for one person to avoid a myocardial infarction. So, aspirin may not be as effective as we believe, and these are similar numbers to expensive medications. In making decisions about making healthy changes in our lives, it is beneficial to have knowledge of how the results can affect you.

“Part of the problem lies in the way researchers are taught statistics. Research methods textbooks are littered with information on the null hypothesis and usually say little on effect size estimates.”

Another example is the supplement CLA. In a 2005 study, researchers found a 4.5% reduction in body fat over the 24-month period for the group of people taking CLA. This number sounds as if we are getting closer to the “how much,” but it is a bit more complex. The 4.5% reduction is actually a 1.8kg reduction over the two years. The placebo group lost 1.7kg over the two-year period. If we calculate an effect size estimate, the effect of taking CLA over the placebo is minimal. So, is it worth spending money on a supplement with a minimal effect over the placebo?

Part of the problem lies in the way researchers are taught statistics. Research methods textbooks are littered with information on the null hypothesis and usually say little on effect size estimates. Today’s textbooks are still similar to those used in the 1960s (with the exception of some great new textbooks, such as Geoff Cumming’s). But now, many journals are demanding that researchers include measures of effect size and avoid using null-hypothesis testing. As consumers of research, we would benefit greatly from these changes.

Maybe we’re not so different from Feynman’s islanders.

Summary

When we think of physics, we think of laws. Physics gives us a statement of “how much.” Newton’s law of cooling states how quickly something will cool, given the initial temperature of the object and the room temperature. Nutrition, as well as sports and conditioning research would benefit greatly from using measures of effect size that will tell us how much or how effective the substance is.

If the fitness industry wants to tout how its programs are scientifically based, then it would be helpful for the cargo cult science that makes up much of fitness literature to move forward to more modern techniques of science.

More Like This:

• Don’t Just Argue, Debate Like a Scientist
• How to Read and Analyze Research Like a Pro
• Is Chocolate a Weight-Loss Food? How to Be a Reasonable Skeptic
• New on Breaking Muscle Today

References:

1. Feynman, Richard P., Ralph Leighton, and Albert R. Hibbs. Surely You’re Joking, Mr. Feynman!. Edited by Edward Hutchings. Reprint edition. New York: W. W. Norton & Company, 1997.

2. Gaullier, Jean-Michel, Johan Halse, Kjetil Høye, Knut Kristiansen, Hans Fagertun, Hogne Vik, and Ola Gudmundsen. “Supplementation with Conjugated Linoleic Acid for 24 Months Is Well Tolerated by and Reduces Body Fat Mass in Healthy, Overweight Humans.” The Journal of Nutrition 135, no. 4 (April 1, 2005): 778–84.

Photo 1 courtesy of Shutterstock.

The post How to Avoid Cargo Cult Science in Fitness Literature appeared first on Breaking Muscle.

In this article, I’ll cover some of the basic concepts about how a study is designed and how to separate the good from the bad.

Internal Validity and External Validity

Judging the quality of a study consists of examining two areas: internal validity and external validity. Internal validity refers to the controls of a study. External validity refers to whether or not the results of that study can be projected back onto a larger population.

Internal validity is primarily built through accurate measurement instruments (both the equipment and type of test) and control of confounding factors. A confounding factor is any unaccounted for item that caused the effect you observed in your study. Did that new supplement increase the subject’s the back squat weight or was it something else, like the training program the subject used or a change in diet?

RELATED: A Lesson in Study Design (And the Bench Press)

External validity is where you need to examine who the subjects were in the study. Were they all men? All women? Elite athletes? Novices? Old? Young? The demographic of the subjects is incredibly important. Look closely at who was included in the study and ask if those subjects represent you or your athletes. The less varied the demographic of the subjects, the smaller the population the results apply to.

What Do Studies Do?

Now we know these concepts, but how in the world do we figure out the level of internal and external validity in the study we are reading? To start, let’s go back to the basics of what a study is and what it does.

A scientific study tries to answer a question by creating a situation that can be observed and measured. In health and fitness, we are mostly looking to observe cause and effect. This caused that. Did taking creatine increase your one-rep-max (1RM) back squat? Did having that last beer make you that much more awesome? I don’t know how many beers you had, so we’ll use a fake study about creatine and increasing the 1RM back squat as our example.

“An important part of understanding studies is to never, ever believe that just because a study found such-and-such then that is now a fact.”

The most basic step in designing a study is developing a dependent variable and an independent variable. In our study, the weight of back squat is the dependent variable and the amount of creatine taken is the independent variable. The researcher determines the amount of creatine the participants will take in the study design and then measures back squat weights to determine if they changed. That means the this is the independent variable and the that is the dependent variable in your question of this caused that.

The Role of Bias

One of the most important factors that can affect the quality of a study is bias. Bias is a situation where either the researcher did not account for something that affected the results or subjects were selected that were more likely to have a specific outcome. Bias affects both internal and external validity. Not accounting for something that affected the results is a confounding factor, which reduces internal validity. Selection bias is tremendously important in health and fitness research and affects external validity.

The Importance of the Randomized Controlled Trial

Another important aspect of study quality is the type of study conducted. While there are several types of studies, including observational and case studies, the only true way to determine cause and effect is using a randomized controlled trial, or RCT. Other types of studies are nice for presenting data, but to say this caused that, you have to use an RCT. Yes, that is a must.

“The goal of a good study design is to change just one factor, the independent variable, and keep everything else between the study groups as similar as absolutely possible.”

An RCT is the necessary study design for showing cause and effect because it reduces selection bias. Meaning, every subject in the study has an equal chance of being selected for either group. In an RCT, you collect your test subjects and randomly assign them into two or more groups. At a minimum, you must have a test group and a control group. In our fictional study, the test group would take creatine and the control group wouldn’t take any supplements. The goal of a good study design is to change just one factor, the independent variable, and keep everything else between the study groups as similar as absolutely possible.

The one exception to this is for review studies, which include literature reviews and meta-analyses. In these studies, the author compares as many studies as can be found that relate to a certain topic and analyzes the study designs and results. This is a great way to get a picture of a much larger body of evidence about that topic.

Studies Don’t Prove Anything

It is important to understand that scientific studies don’t prove anything. They suggest. They provide supporting evidence. An important part of understanding studies is to never, ever believe that just because a study found such-and-such then that is now a fact.

“You need to examine everything closely, ask hard questions, and be a difficult sell on any idea.”

Sometimes researchers are biased and interpret results incorrectly. Sometimes there’s a confounding factor that wasn’t accounted for. Sometimes you just can’t figure out why the results were the way they were. Look for multiple studies that support the hypothesis. For extra credit, look for studies that don’t support that hypothesis and compare them with the ones that do.

Guidelines to Help You Get the Most From Reading Studies

• Type of study. As stated above, the study has to be an RCT. An abstract won’t always come straight out and call a study an RCT, so look for the key word randomized. If you can’t find it, it’s not an RCT and the results aren’t worth your time.
• Were the researchers blinded? Single blinding means the researchers did not know whether the participants they were testing belonged to the test or control group and is an absolute requirement for reducing bias and establishing internal validity. Like using an RCT, this is a necessity and the results of a study that doesn’t use it shouldn’t be trusted.
• Who were the participants? Remember, we’re looking for good external validity here. Does the study population represent the general public, you or your athletes, or a group that you don’t even work with? If the participants don’t sound like you or your athletes, the results won’t necessarily apply to you.
• What was the level of training of the participants? One of the golden rules of exercise research is to never, ever use novice athletes. Novice athletes will respond positively to just about anything. Seriously, never buy into the results of a study that used novice athletes.
• What is the difference between the test and control groups? This is going to tell you what the study actually examined, and it’s not uncommon for a poorly designed study to end with the results reflecting something completely different from what was intended. Was the topic of the study the back squat, but the results were determined by testing leg press or leg extension weight? Seriously, this stuff is out there.

This all may sound a bit cynical, but in all honesty, that’s the scientific way. You need to examine everything closely, ask hard questions, and be a difficult sell on any idea. To do anything else is to be a sucker for every fad that comes along. Practice looking at studies, asking these questions, and thinking critically about the quality of the study and meaning of the results. Keep at it and you’ll be crunching through studies like a pro in no time.

READ: Correlation and Causation: What Pubs Have to Do With Your Clean and Jerk

There is much more to truly understanding how to read scientific studies, so this article is only an overview to get you started. Have a question? Ask it in the comments!

Photos courtesy of Shutterstock.

The post How to Read and Analyze Research Like a Pro appeared first on Breaking Muscle.