Bias in surveys: Year 11 Standard sampling and relationships in data

Warm-up

Three quick ones from earlier work. Try each in your head or on paper before you open the answer.

From Lesson 3: a stratified sample of 30 is taken from 150 people. A group has 60 of the people. How many are in the sample?
Answer
12. 60 ÷ 150 × 30 = 12.
From Year 8: the square root of 81 is ?
Answer
9, because 9 × 9 = 81.
From Year 7: 120 out of 400 as a percentage is ?
Answer
30%. 120 ÷ 400 = 0.3.

Words used in this lesson

bias: a lean that pushes results one way every time. For example: a clock that always runs five minutes fast.
self-selection: taking part by your own choice, so the sample is made of volunteers. For example: people who choose to post a review of a restaurant.
non-response: members chosen for a sample who do not reply. For example: questionnaires never returned.
response rate: the share of people asked who actually reply. For example: replies divided by forms sent.
leading question: a question worded to push towards one answer. For example: a question that hints at the answer wanted.
random wobble: the chance difference between a sample result and the truth, which shrinks as n grows. For example: the gap between two fair samples.

The phone-in

A radio station asks listeners to call in about a plan to build a market on the old railway land. By the end of the programme most callers are against it. The presenter says the town has spoken. A listener says that the callers are not the town.

Predict first: if ten times as many people called in, would the result be closer to what the whole town thinks? What would have to be different about the way the opinions were collected? Use the first figure for who replies and the second for sample size.

Group B who reply (%) 100

200replies received

45estimate of yes in percent

100% of group B reply: 200 replies, 90 say yes, estimate 45% against the true 45%.

The figure needs JavaScript. In words: a survey of 200 people has group A of 120 (25% say yes) and group B of 80 (75% say yes). If everyone replies, 90 say yes and the estimate is 45%, the true value. If only half of group B reply there are 160 replies and 60 yes, so the estimate is 37.5%. If only 10% of group B reply the estimate falls to 28.1%.

Show me the steps

Everyone replies: 200 replies and 90 yes. The estimate is 90 ÷ 200 = 45%, equal to the true value.
Half of group B reply: 160 replies and 60 yes. The estimate is 37.5%, which is 7.5 points too low.
One tenth of group B reply: 128 replies and 36 yes. The estimate is 28.1%, close to group A's own 25%.
Group A are always sure to reply. When the group that disagrees goes quiet, the estimate drifts towards the group that is heard.

Bias is a lean; wobble is a spread.

Sampling bias: some members of the population cannot be in the sample, or are much more likely to be. Self-selection is the case where people choose themselves.
Non-response bias: the people who do not reply differ from those who do. The response rate is replies ÷ people asked.
Wording bias: a leading or loaded question pushes the answer one way.
A bigger sample shrinks the random wobble: about 19 fair samples in 20 land within roughly 100 ÷ √n percentage points of the truth. It does not shrink a bias.

Sample size (step 1 to 4) 25

25sample size

20rough margin of error, plus or minus, in points

Sample of 25: wobble about 100 ÷ 5 = 20 points. The method that reaches group A only sits at 25 ± 20, the fair method at 45 ± 20, and the true value is 45.

The figure needs JavaScript. In words: the rough margin of error of a percentage from a sample of n is about 100 divided by the square root of n, in points. For samples of 25, 100, 400 and 1600 that is 20, 10, 5 and 2.5. A method that reaches group A only is centred on 25 whatever the sample size, so at 1600 it reads 25 plus or minus 2.5 while the true value is 45. A fair method is centred on 45 and tightens around it.

Show me the steps

Sample of 25: the wobble is 100 ÷ 5 = 20 points. Both bands are wide. The biased band, 5 to 45, just reaches the truth, only because it is so wide.
Sample of 100: the wobble is 10 points. The biased band, 15 to 35, has stopped reaching the true 45.
Sample of 400: the wobble is 5 points. The biased band is 20 to 30. It is tight, and wrong.
Sample of 1600: the wobble is 2.5 points. The fair band, 42.5 to 47.5, is close to the truth, and the biased band, 22.5 to 27.5, is as far off as ever.

“The sample was too small” is not an answer to every doubt. If the method leans, a larger sample of the same method leans just as far, and looks more convincing.

Worked examples

A gym asks every member leaving at 6 pm on one Tuesday whether it should open earlier. What could be biased?

Only members who come at 6 pm can be asked. Early-morning members, the very people the change is for, are missed.
It is also one day, and the members present may be regulars.
So the sample leans towards evening users: sampling bias, from who could be asked.

A questionnaire is sent to 500 households and 150 are returned. Of the returned forms, 90 say yes. What can be said about the 500?

The response rate is 150 ÷ 500 = 30%. Of the replies, 90 ÷ 150 = 60% say yes.
The 350 who did not reply are unknown. If none of them would say yes, the figure for all 500 is 90 ÷ 500 = 18%.
If all of them would, it is (90 + 350) ÷ 500 = 88%. So the true figure could be anywhere from 18% to 88%: a 30% response rate says very little.

Your go. The first step is given. A sample of 625 people is used to estimate a percentage. Use the rule of thumb that the rough margin of error is 100 ÷ √n points. How big is the wobble?

The square root of 625 is 25, because 25 × 25 = 625.
Divide 100 by the root.

Check the answer

100 ÷ 25 = 4 points. A common slip is to divide by 625 itself, giving 0.16 points, which would make a survey of 625 sound almost exact. The root is the part that matters: to halve the wobble, the sample must be four times as big.

With a calculator

The wobble rule needs a square root. Use the square root key: for a sample of 900, the root is 30, and the wobble is 100 divided by 30. To go the other way, divide 100 by the wobble you want, then square the answer (the squared key or multiply the number by itself).

Write the root on paper before dividing. A mistyped number under the root gives a believable but wrong answer, and a sample size should come out as a whole number.

Check before you go on

1. A town's local news website puts a poll on its front page asking whether a new cycle lane should be built in the town, and 4000 readers click an answer. Most are against it. What is the main weakness of this poll?

The readers chose themselves, and people with strong views are the most likely to click. 4000 readers is too few to say anything about a town. It is a stratified sample, and a stratified sample cannot be trusted to represent a town. Website readers are not part of the population of the town, so their views do not count.

Your turn

Start at Level 1 and go as far as you can. Type negative numbers with a minus sign, for example -7. A wrong answer gets a reason, not just a cross: read it, then try again.

Level 1 · Getting started

2. A school wants students' views on its uniform. Which question is the fairest wording?

Don't you agree that our scruffy uniform spoils the look of the school? Should students finally be free of the outdated uniform that everybody dislikes? Should the school keep the current uniform, change it, or have no uniform? Wouldn't you prefer a smart new uniform designed by students?

3. A survey form is sent to 450 households and 180 forms are returned. What is the response rate, as a percentage?

Level 2 · Building

4. A survey asks 400 people. 250 reply, and 150 of those say yes. What is the highest percentage of all 400 people who could say yes, if every one of the people who did not reply would say yes too?

5. In a town, 70% of homes have a landline and 30% do not. A phone survey of landline homes finds 60% support a plan. Among homes without a landline the support is 30%. What percentage of all homes support the plan?

Level 3 · Stretch

6. A café chain surveys customers about its menu. The first plan is a card handed out with the bill, and 500 are returned. Which change would reduce the bias the most?

Hand out cards with the bill for a whole month, so that 5000 are returned instead of 500. Choose 300 customers at random from the till records and follow up those who do not reply. Ask the staff to fill in extra cards, as they hear what customers say about the menu every day. Print the cards in a brighter colour, so that more of the customers notice them and reply.

7. Use the rule of thumb that the rough margin of error is 100 ÷ √n percentage points for a sample of n. What sample size gives a wobble of 2 points?

Something to think about

A survey sends a questionnaire to 1 000 randomly chosen households and 200 reply. A second survey sends it to 300 randomly chosen households, visits everyone who does not reply and gets 270 answers. Which is more trustworthy, and why?

One way to think about it

The second, even though it is smaller. Its response rate is high, so what the non-responders could have said can change the result very little. The first has 800 silent households, and the people who do not return a form may differ from those who do.

Exit ticket

8. A teacher wants to know how many hours students spend on homework each week. She gives a questionnaire to the students who come to the after-school homework club. Which is the best description of the problem?

A leading question: the questionnaire suggests how much homework students ought to do. Non-response: the club students did not reply, so the results come from too few forms. The sample is a stratified sample of the school, so it is random and can be trusted. Sampling bias: club students may spend more or less time on homework than other students.