Computer Writing and Research Lab | University of Texas at Austin

Evaluating Research Studies

On Monday, my students read a chapter called "How Good is the Evidence?" in Asking the Right Questions that provided a checklist for evaluating research studies. Some of the questions:

What is the quality of the source of the report?
Has the study been replicated?
Is any reason for someone to have distorted the research?
How far can we generalize, given the research sample?
Are conditions in the research artificial?

Anyway, I found a great article in Popular Science called "Science Confirms the Obvious 2008." It featured several short summaries of different research studies. For some of the studies, they even explained the methodology of the scientists. For instance, in "Sleep and Caffeine Combat Sleepiness," they mentioned that only 24 people participated in the study.

I broke the students into groups of three and assigned them one of the studies to evaluate. They simply had to answer several of the questions on the checklist. Most of the studies were freely available on-line, and others were available through the UT website. (I double-checked this before class.) The students didn't have to read the studies--luckily, scientific papers have a consistent format. At the beginning, they usually reference other studies. Then they describe their methods.

The students liked the topics of the studies ("Loneliness Sucks" and "Rock Stars Live Hard, Die Young"). More importantly, it demonstrated to them that it's possible to find out a great deal about the evidence cited by an author in a short period of time. Most of them ended up questioning the accuracy of their study in a couple of different ways.

The article is in the April 2008 issue of Popular Science, which is available in HTML through the library. However, I can think of other magazines that summarize research studies regularly--everything from Psychology Today to Real Simple.

This is great.

What I love about this assignment, Liz, is how generally useful it is for students, particularly for those in the sciences and the social sciences. I'm always looking for ideas outside my cushy Humanities perspective--

Thanks!