Back

One of the most common ways of deceiving someone, including ourselves, is by unfair sampling.

In the 1976 American presidential election I recall seeing campaign commercials run by Republicans against Jimmy Carter. They went something like this: "We are here in Jimmy Carter's home state of Georgia. We're asking people what they think of him." They then show five clips of people on the street being asked about Carter. Every one of them has something bad to say. They don't like him, they think he did a bad job as governor, they wouldn't vote for him as president. When I first saw this my reaction was surprise. I didn't realize Carter was so unpopular in Georgia! Then I thought for a minute. Suppose the film crew spoke to 100 people, which could probably be done in less than a day. Even if 70% of the people in the area liked Carter, they still could find about 30 people who didn't. Then the campaign team selects five out of the 30 that sound the most unflattering. So even if Carter were very popular in his home state, they would have no trouble making a technically truthful commercial that made him sound very unpopular. I found the ad to have surprising impact. I guess the Carter campaign did also, since I recall them running a similar spot attacking Ronald Reagan four years later.

How is it that people can be fooled this way? Suppose we had gone to Georgia and talked to five different people we encountered in a variety of places and circumstances. If all five of them disliked Carter, would it be reasonable for us to conclude that Carter was unpopular? I think it would. If Carter were well liked, it would be very surprising if we talked to five randomly chosen people and all five of them happened to be from the minority that disliked Carter. But in the commercial they weren't randomly chosen. They were selected specifically to be used in a commercial to persuade people to vote against Carter. Obviously the people making the commercial were not going to include comments that made Carter look good. The selection was biased. It did not represent a "typical" group of five people. Nevertheless, if we were not paying close attention, our psychology was likely to lead us to the conclusion that Carter is in fact unpopular.

Many of the truths that concern us are generalizations: each is a single statement that we claim is true for a large number of cases. We might feel the statement is always true, or just that it is true so often that we should make decisions with the expectation that it is true. Here are some possible generalizations:

• Apples are red.
• Newspapers are liberal.
• Members of some ethnic group are ignorant.
• Cigarettes cause cancer.
• Four-leaf clovers bring luck.

While we sometimes learn general principles by being told about them, it is certainly an essential part of human learning for us to form generalizations by observing what goes on around us. We don't have to be told peaches have pits if we have eaten a lot of them and found a pit in every one. The basketball coach evaluating a new player can decide how good she is by watching what happens when she takes a lot of shots at the basket. A cancer specialist might observe that almost all his lung cancer patients are also cigarette smokers.

When our minds form a generalization, we are always making a judgment about a large number of cases based on a small sample of those cases. Our observation about peaches having pits is based on a small number of peaches, but we expect it to apply to all peaches. The estimate of the basketball player's shooting ability is based only on shots taken during the tryout, but the coach expects this sample to be representative of the shots that will be taken over the course of many games. Noticing that most lung cancer patients are smokers is based on a small number of patients, but is assumed to apply to lung cancer patients in general.

Almost all our behavior is based on expecting things to respond to our actions in the same way that they have in the past. Making and using generalizations based on limited experience is as fundamental as any aspect of human thinking can be. Without doing this, we could not survive. Unfortunately, as in the case of the Georgia residents who didn't like Jimmy Carter, our ability to form generalizations can be faulty.

We need to understand what is going on when this process fails if we are to be responsible thinkers who can avoid making false generalizations. First, we need to recognize when sampling works correctly. Suppose there are a million people in a city, and 85% of them are less than 6 feet tall. If we chose 100 people from the city, completely at random, we would expect about 85 of them to be shorter than 6 feet. By the same token, if we didn't know what percentage was this tall, and we took a random sample, it would be reasonable to assume the percentage of people in the city under 6 feet tall was about the same as the percentage in the sample.

The problem comes when the sample is not random. If our sample included only basketball players instead of people at random, the percentage of tall people would be likely to be very different from the percentage in the general population. Similarly if the sample was taken at a department store where most of the people were women, we wouldn't expect the same result as for the general population where almost half the people are men. Even if we took the sample by having people call into a radio station on the telephone, there would be a chance that the sample was misleading because tall (or possibly short), people were more concerned about the issue and more likely to call in. It is rare that we can get a perfectly fair sampling of all the cases in a category, but the more arbitrary the sample is, the safer it is to assume that the general case has the same characteristics as the sample.

In addition to unfair selection, errors in sampling can come from the randomness itself. If a sample is small, luck will often result in the sample having very different properties from the overall group. If we pick four people at random, one time in 16 they will be all women and one time in 16 they will be all men. In either case it would not give a realistic picture of people as a whole. If our random sample was of 1000 people, we would expect approximately half of them to be women, and would have a much better approximation to the overall population.

Let's look at some realistic cases where unfair sampling might lead to false conclusions.

The danger of air travel: Many people don't like to fly because they feel like it is dangerous. Part of the reason is probably because they see newspaper articles about plane crashes and feel that this is a relatively common event. What seems to happen in our minds is that we compare the number of crashes we hear about with an intuitive estimate of how many don't crash. The trouble is, we don't hear much about the planes that don't crash, while we hear, often in great detail, about the ones that do, even when the crashes occur on the other side of the world. We may mainly know about safe landings when we or our friends travel, and occasionally when a celebrity is shown arriving at an airport in the news. So the sampling we are aware of is extremely biased. We are aware of virtually every disaster but only an extremely tiny percentage of the non-disasters. Our natural mental estimation is that flying is far more dangerous than it actually is.

Crimes committed by released prisoners: Occasionally we hear a news story about some person recently paroled from prison who then commits a murder or some other heinous crime. As a result, many people clamor for changes in prison policy to prevent people from being released on parole. The problem is that people are likely to greatly overestimate the percentage of parolees who commit serious crimes because the news media rarely report the cases of people who are paroled and don't commit crimes. Obviously it is undesirable to keep people in prison, at public expense, when they would not cause further problems for society. To make intelligent decisions about public policy we need to know the actual percentage of people released who cause problems; we cannot trust our intuitive impression we get from news stories that don't reflect a fair sample of people who are released from prison.

The danger of terrorist attacks: In some years there have been terrorist attacks on tourists in Europe. If we read that, say, eight tourists have been killed in three separate incidents during the past few months, we might feel that it would not be a good idea to travel to Europe until things have settled down. We might also see comments in the media that this is not a safe time for European travel. However we hear about those tourists that were killed, but most of us know nothing about the millions of tourists who were not killed. As a result out mental perception is that a significant percentage of tourists are being killed, while in fact the percentage is very tiny, probably far less than the number that die of heart attacks or other natural causes while on vacation.

Superstitions based on selective memory: There are many superstitions about events that bring good or bad luck, such as the idea that finding a four-leaf clover is good luck. Part of the reason these superstitions sometimes seem to work is the result of the vagueness of the prediction combined with biased sampling. After finding a four-leaf clover, a variety of things happen to us. If we are eager to believe in the luck of the clover, we can ignore all the things that happen that aren't good and select something that happens that is better than expected. Even if nothing particularly good happens, we may forget that the clover failed but remember other cases where finding a four-leaf clover was indeed followed by some unusual good fortune. If we selectively remember the cases that work and ignore the cases that don't, we can get the impression that the superstition is confirmed by experience.

Predictions with no time limit: Psychics and fortune tellers frequently make predictions that have no time limit associated with them, for example that you will have financial success or start a new romantic relationship. If we keep track of successes and failure for this type of prediction, we will find occasional successes and no failures. The reason there are no failures is that there is always the possibility that the prediction will come true at a later time. As a result, our sampling of the cases by which we judge the psychic always includes the successes and never includes the failures! Obviously this is not a fair sampling. Even when we might reasonably assume a time limit, it is easy to forget predictions that fail while remembering those that succeed.

Movie endorsements: I've seen commercials for newly released movies that show people coming out of the theater talking about how great the movie was. While it is possible these are actors, it is likely that they are unpaid patrons giving their honest opinion. That does not mean that the average person likes the movie, though. The crew filming the commercial would certainly interview a large number of people and then only include the reactions that are most favorable. Since the selection of cases to be presented is made by the advertising agency, it is clear that the cases shown will not be an unbiased cross-section.

Successful people: I've seen celebrities and other prominent people interviewed on television and they'll often say something like "You've got to take risks" or "You have to pursue your dream". It may be that most people who are huge successes take this attitude. On the other hand, many people who take this attitude may wind up being failures. The failures are never interviewed. The sample contains only the successes. A closely related problem is all the attention given to famous entertainers and athletes. Only a tiny percentage of people who choose a career in sports or entertainment actually become famous, but in the sample we see, all are famous, so we are likely to greatly overestimate the extent to which these are wise career choices.

Whenever a claim of some general principle is important enough to merit examination, we should immediately ask whether the sample of data that led to that conclusion was a fair sample. We should be equally tough on ourselves if we think we have noticed some generality that would be of interest to others. If we care about the truth we have to be on guard against the problems of unfair sampling.