Cognition Research Methods

Dealing With Confounds

Imagine an experiment in which research participants are asked to recognize letter strings briefly presented on a computer screen—let’s say for 30 milliseconds—-followed by a mask. In the first 50 trials, the letter strings are random sequences (“okbo,” “pmla,” and so on). In the next 50 trials, the letter strings are all common four-letter words (“book,” “lamp,” “tree,” and so on). Let’s say that the participants are able, on average, to identify 30% of the random sequences and 65% of the words. This is a large difference; what should we conclude from it?

In fact, we can conclude nothing from this (fictional) experiment, because the procedure just described is flawed. The data tell us that participants did much better with the words, but why is this? One possibility is that words are, in fact, easier to recognize than nonwords. A different possibility, however, is that we are instead seeing an effect of practice: Maybe the participants did better with the word trials not because words are special, but simply because the words came later in the experiment, after the participants had gained some experience with the procedure. Conversely, perhaps the participants did worse with the nonwords not because they were hard to recognize, but because they were presented before any practice or warm-up.

To put this in technical terms, the experiment just described is invalid—that is, it does not measure what it is intended to measure—namely, the difference between words and nonwords. The experiment is invalid because a confound is present—an extra variable that could have caused the observed data pattern. The confound in this particular case is the sequence, and the confound makes the data ambiguous: Maybe words were better recognized because they’re words, or maybe the words were better recognized simply because they came second. With no way in these data to choose between these interpretations, we cannot say which is the correct interpretation, and hence we can draw no conclusions from the experiment.

How should this experiment have been designed? One possibility is to counterbalance the sequence of trials: For half of the participants, we would show the words first, then the random letters. For the other half of the participants, we would use the reverse order—random letters, then words. This setup doesn’t eliminate the effect of practice, but it ensures that practice has the same impact on both conditions. Specifically, with this setup, practice would favor one condition half the time and the other condition half the time. Thus, the contribution of practice would be the same for both conditions, and so it could not be the cause of a difference between the conditions.

If this point isn’t perfectly clear, consider an analogy: Imagine a championship football game between the Rockets and the Bulldogs. As it turns out, there’s a strong wind blowing across the field, and the wind is coming from behind the Rockets. The wind helps the Rockets throw and kick the ball farther, giving them an unfair advantage. The referees have no way to eliminate the wind. What they can do, though, is have the teams take turns in which direction they’re moving. For one quarter of the game, the Rockets have their backs to the wind; then, in the next quarter, the direction of play is reversed, so it’s the Bulldogs who have their backs to the wind, and so on. (This is, of course, how football games operate.) That way, the wind doesn’t favor one team over the other, and so, when the Rockets win, we can’t say it was because of the wind; in other words, the wind could not have caused the difference between the teams.

Returning to our word/nonword experiment, we know how it would turn out when properly done: Words are, in fact, easier to recognize. Our point here, though, lies in what it takes for the experiment to be “properly done.” In this and in all experiments, we need to remove confounds so that we can be sure what lies beneath the data pattern. Several techniques are available for dealing with confounds; we’ve mentioned just one of them (counterbalancing) here. The key, however, is that the confounds must be removed; only then can we legitimately draw conclusions from the experiment.

Critical Questions

1.
fiogf49gjkf0d
What is an experimental confound, and how might it render an experiment invalid?
2.
fiogf49gjkf0d
Consider the following experiment. Participants are shown pairs of words such as “horse” and “cow” and are asked to judge as quickly as possible whether the words are related in meaning. When the words are related, the participants press a button under the right hand, and when the words are not related, participants press a button under the left hand. The researcher finds that responses to related words are faster than to unrelated words. What is the confound in this experiment, and how could it be corrected?
3.
fiogf49gjkf0d
Researchers sometimes use the term orthogonal to describe two variables in an experimental design that are not confounded with each other. This word is also used to describe a pair of perpendicular lines that meet at a right angle. How are the two senses of the word orthogonal related to each other?

Submit to Gradebook:

First Name:
 
Last Name:
 
Your Email Address:
 
Your Professor's Email Address:
 
Section: