Does that ‘latest poll’ really tell us what it claims to tell us?

This is the 'text only' portion of the activity. All text and pictures can be found at

https://docs.google.com/document/d/1cFmdT_TATu4ZaHhFRTyMoibacoc3MfG3/edit?usp=sharing&ouid=116219559430514027988&rtpof=true&sd=true

What kids need to know about polls, surveys, and their use in political campaigns

Civic education programs encourage students to learn about and to be involved in local, community, regional, and national issues and policy decisions. Much available information about these issues claims to use statistics in one way or another to support a point of view. Students need to master a few key statistics-based ideas to judge the validity of much of that information. The following lessons show students how to validate information without getting into the world of standard deviations, correlations of coefficients, regression, analyses of variance, or more advanced statistical concepts.

Often almost a dozen sets of poll results are published daily during ‘election season’, the six months or so before a major election. During more normal times surveys regularly report everything from which car is best to which hamburger tastes best. To make sense of these results students need to learn how to analyze polls and surveys to better understand how seriously polling results should be taken as truth. In the political and civic world, some surveys and polls are designed to support a particular candidate or policy idea.

The lessons below have students use any one, several, or all five criteria to assess survey/poll results. The criteria are:

‘Sample size;’ were there enough responses to the survey to make a valid conclusion? Major national polls usually have around 1000 reported responses. As part of this issue, how many people were actually surveyed and how many of these did not respond to the survey?
‘Representative;’ did the sample accurately represent the target population? (in a presidential election poll, for example, the target population is all the registered and likely voters in the US). How were people selected to be sure that the sample is a valid representation of the target population?
‘Misleading presentation of survey results;’ Were the poll or survey results shown on charts or graphs to maximize or minimize certain aspects of the results? Was one of the scales on a results chart made larger or smaller to create a misleading presentation?
What polling methods were used in the poll? Were methods used that tend to reduce accuracy of poll results? Using land line phone calls, for example, reaches mostly older people, as many if not most younger people use cell phones exclusively.
Apply media literacy analysis to polls. Who paid for the poll? Who wrote the questions? Are the poll results similar to other polls dealing with the same issues? How are survey questions written? Are questions neutral or do they reinforce a particular view?

Explanations and examples of each criterion are included in the student materials section below.

Often almost a dozen sets of poll results are published daily during ‘election season’, the six months or so before a major election. During more normal times surveys regularly report everything from which car is best to which hamburger tastes best. To make sense of these results students need to learn how to analyze polls and surveys to better understand how seriously poling results should be taken as truth. In the political and civic world, some surveys and polls are designed to support a particular candidate or policy ideas.

The lessons and ‘option’ below have students critically examine two real polls by major polling companies that reported very different conflicting results about the same issue. Students will decide which, if either poll’s results are accurate.

Objectives:

1) Students will describe how poll and survey results need to be reviewed carefully. Students will explain how the issues of sample size, sample representation, question wording, polling techniques all determine the accuracy of any poll. Students will explain how traditional ‘media literacy’ vetting should be used when examining polling results.

2) Students will explain how poll and survey results are used in political campaigns and the creation of government policy.

Standards: (California)

12.2.4: Understand the obligations of civic-mindedness, including voting, being informed on civic issues, volunteering and performing public service, and serving in the military or alternative service.

12.6.3: Evaluate the roles of polls, campaign advertising, and the controversies over campaign funding.

As part of each lesson, materials called ‘handouts’, one for each of the five methods of poll validation above, will be given to small groups of students along with actual polling results from two real polls.

Both lessons below have students examine each of the five different ways to evaluate polls. Both lessons take class time, and the simulation requires a number of days. Students will need to do some online research as part of their analysis. Instead of doing one of the lessons, a class could be given any one (or several) criteria to evaluate poll results, using the appropriate Expert Group Handout(s) along with the two poll handouts to critique the poll results. The entire class would learn that method of evaluating the accuracy…or lack thereof…of opinion polls and surveys.

The Simulation Lesson Activity: overview

The simulation lesson will have students roll play as directors of a gubernatorial campaign. In this roll they will be asked by their candidate to evaluate the results of two conflicting opinion polls (which actually happened in a real statewide gubernatorial election campaign). The issue is whether the state should financially support the expansion of ‘green’ energy in the state (wind, solar, possibly nuclear power) or should better support expansion of fossil fuels, as the state has both oil and coal reserves as yet untapped. The candidate would prefer to support green energy expansion but is being attacked by an opposition leader who supports fossil fuels using a poll that says the public overwhelmingly supports fossil fuel expansion Another poll reports strong support for ‘green’ energy growth. The candidate is asking their campaign committee to research the polls and give her a recommendation as to which energy policy she should support in her campaign. Students will take on the rolls of being on the candidate’s election committee and having to decide, based on their evaluation of the conflicting opinion polls, what policy the candidate should support.

Students will participate in three ‘Civil Conversations’ in the activity, two in very small groups of 5, and one in a whole class conversation. Students in the whole class should discuss how poll results might be used in political campaigns.

The ‘academic’ lesson Activity: Overview

In the non simulation or ‘academic’ lesson, students in small groups of five students each will analyze the two polls, each small group tasked to examine the polls using one specific question about poll results accuracy. Each group will report its findings to the whole class, which will then decide which poll results are more accurate and believable.This activity may be modified to use only one or two or so of the criteria above to evaluate the polls.

The Process: academic lesson directions for the teacher

Divide the class into groups of five students each. Letter these as Group A, Group B, and on to Group E. If necessary, letter the sixth group as Group A-2. Followed by Group B-2, and so on.

Each group will examine a different set of handouts that explain and demonstrate a specific set of questions to ask of each poll. All Groups will be given the two polls in question (these are real polls)and a handout explaining their issue. To save paper, the poll URLs are shown on the top of each poll. If there is time, students can be encouraged to use Internet resources to read more about the polls.

Materials for students will include explanations and examples of different accuracy questions asked of poll results.

Expert Group A will try to find out how representative of the whole population was the survey. Group B will ask how many people were surveyed by each poll. Group C will examine how charts and graphs sometimes report survey results so as to be misleading. Group D will examine how the polls were conducted; by telephone, cell phone, door to door, online survey questions, etc. Group E will use ‘media literacy’ methods (who paid for this? Who made up this pool? Who wrote the questions? Are the poll results similar to results from similar polls?) to examine the poll results.

After each group has finished their analysis, have each group report to the whole class their findings and conclusions about how each poll’s accuracy should be judged.

The Process: Simulation activity directions for the activity for the teacher:

Students will be first in an Advisory Group and then move to a second group, called the Expert Group. Divide the class into groups of 5, calling these groups the Advisory Groups, numbered Advisory 1, Advisory 2, Advisory 3, and so on. Expert Groups will be lettered as Expert A, Expert B, through Expert E. If necessary, additional Expert Groups need also to be lettered as Group A-2, B-2, and so on.

In advance of the activity make a three by five card for each student, showing which Advisory Group the student will be in and also which Expert Group each student will be in. Each Advisory Group should have five students with their cards indicating Expert Group A through E. Each Advisory Group will have students who attend each of the Expert Groups. Thus Advisory Group A student cards will have on them Advisory A and Expert 1, Advisory A, Expert 2, and so on. Advisory Group B cards will have on them Advisory B, Expert 1, Advisory B, Expert 2, and so on. Make enough cards for all students in the class or in each class. Assign a leader for each Expert Group.

Each Expert group will examine a different set of handouts that explain and demonstrate a specific set of questions to ask of each poll. All Expert Groups will be given the two polls in question (these are real polls). To save paper, the poll URLs are shown on the top of each poll. Students should be directed to use Internet resources to read more about the polls and do research about the aspect they are studying. In each case, if information asked about is not available from the poll, that should be considered as part of the analysis of the poll’s accuracy.

Expert Group A will try to find out how representative of the whole population was the survey.

Group B will ask how many people were surveyed by each poll.

Group C will examine how charts and graphs sometimes use to report survey results can be created so as to be misleading.

Group D will examine how the polls were conducted; by telephone, cell phone, door to door, online survey questions, and how each method impacts results.

Group E will use ‘media literacy’ methods (who paid for this? Who made up this pool? Who wrote the questions? Are the poll results similar to results from similar polls?) to examine the poll results

After the Expert Groups have finished with their analyses, the students will return to their Advocacy groups. Each Advocacy Group will have each of the students report their Expert Group findings. The Advocacy groups will then decide how accurate each poll’s results are. They will then decide how those findings should be used to advise the candidate: How should the candidate use the findings to decide which policy should be supported in the upcoming campaign? Each Advocacy group will decide which policy statement the candidate should use.

The whole class will then discuss the Advocacy Group findings and make a final recommendation to the candidate as to what her campaign should propose regarding energy.

The class will read the overview of the simulation activity. The class will then read the ‘Energy policy options’ listed by the gubernatorial candidate. The activity will use the ‘two group’ model, where students move from the ‘home’ group to an ‘expert’ group to examine one aspect of verifying poll accuracy. Students return to their ‘home’ group, each shares what was learned, and the group then decides which policy option they will recommend based on their analysis of the two polls.

Student Materials

You will role play being the chair or members of a political campaign committee for the governor of your state. Your job in this role among other things is to recommend to your candidate how she should speak to specific political issues in her campaign. She has been as a former representative in your state legislature a supporter of ‘green’ energy in your state, meaning solar and wind power. She is concerned about climate change caused by global warning and has sometimes invited university climate scientists to speak at her past campaigns.

Her opponent has, however, just released the results of a recent poll taken in your state showing that between 80 and 90% of the public strongly supports expansion of fossil fuels in your state. There are big as yet untapped reserves of coal and oil in your state, and your opponent says expansion of extraction and processing of these reserves, which probably also includes natural gas, will create many jobs in the state that will vastly increase your state’s economy.

There is also a fairly recent poll taken in your state and others that says most people, at least 70% in some cases, want to see great expansion of ‘green’ energy and a reduction in the use of polluting coal, especially in your state.

Your candidate has asked you, as her campaign chair, to research these polls to determine which is more accurate about how people in your state think about energy issues. She wants you to do that and then recommend back to her, based on your research, that she follow one of the three options in her upcoming campaign

Energy policy options

The candidate in her campaign speeches should propose significantly increasing state spending on solar and wind energy expansion. If necessary to do this reduce spending on fossil fuel (like extraction of coal, oil, and natural gas; the state has untapped reserves of these). The candidate has supported this position in the past and personally would like to see more ‘green’ energy in the state, partly because of environmental and climate change issues,
The candidate in her campaign should propose continuing existing support for fossil fuels and their extraction in the state. She should also talk about supporting existing programs for developing solar and wind energy. In each case she should point to job creation, especially in the case of solar and wind programs. Her campaign should speak to energy issues but not as the most important issue facing the state.
The candidate should agree with her opponent in promoting substantial support for expansion of fossil fuel extraction and processing. She should speak to increasing jobs in these areas. She should only speak to ‘green’ energy promotion as a minor issue and generally only when asked by media or when speaking to an interested group.

Evaluations:

Student Self Evaluation

Summarize what was new learning for you in your Expert group

Summarize what your Advisory group decided to recommend

What do you consider the most important thing you learned about validating the results of a poll or survey?

Evaluation by Teacher:

How thorough was the Student’s Expert Group report?

How well did the student’s Advisory Group use the Expert Groups information to reach its conclusion?

Were the student self evaluation summaries thorough and accurate?Objectives and Standards:

Objectives:

2) Students will explain how poll and survey results are used in political campaigns and the creation of government policy.

Standards: (California)

12.6.3: Evaluate the roles of polls, campaign advertising, and the controversies over campaign funding.

Activity Materials

The Criteria Handouts for Expert Groups

Handout for Expert Groups A and B: issues of adequate Sample Size and Sample Representative of Target Population.

Every poll or survey needs to have a substantial number of people responding to the poll who are ‘representative’ of the ‘target’ population. Some national polls survey public opinion in general, some polls survey specific groups within the general population. In political polls, some target only registered voters. Some target only members of one political party. In any case, a valid poll needs to have an adequate number of people actually surveyed. This is called adequate sample size.

Major national polls usually have at least 1000 people who respond to the survey. When a poll is administered, however, usually many more people are included In the group to be surveyed than actually respond. This can raise a question about representation. How were all the people selected for the poll? How did the poll decide who to select? What about those who did not respond to the poll? If only a small percentage of those asked actually responded, this could be ‘voluntary response bias’, meaning those who did respond may have stronger feelings one way or the other than those who did not respond. We don’t know how all those non responders would have responded to the poll.

Expert Group A is to consider the issue of sample size used in the two actual polls. Group B is to consider how representative was the sample? Look at the polls and also look them up online.

This shows what happens when the people polled are not representative or the number of people who responded to the poll is too small or too large. In these examples the target population claimed was all registered voters in the United States, those who could vote in a presidential election.

From the Math Center at Oxford College in Emory University:

https://mathcenter.oxford.emory.edu/site/math117/historicalBlunders/ Accessed March 2, 2026

Famous Statistical Blunders in History

Literary Digest, 1936

In 1936, Literary Digest, a national magazine of the time, sent out 10 million "straw" ballots asking people to tell them who they planned on voting for in the 1936 presidential election. They received back 2.4 million ballots (not bad for a pre-internet time when polling was still in its infancy, but still only about a 25% return rate, which leaves the door open for voluntary response bias). They predicted Alf Landon would beat Franklin Delanor Roosevelt 57% to 43%. As it turned out, Roosevelt won 62% to 37%.

There were two large problems:

First, the sample chosen was not representative of voters. Literary Digest used lists of phone numbers, drivers' registrations, and country club memberships to select its sample. But 1936 was the height of the depression. At that time, phones, cars, and country clubs were more often available only to rich. The economy was the central campaign issue of the election with Roosevelt advocating the New Deal that was attractive to lower income people, so the opinions of the rich and poor were bound to differ on this issue. As such, their sample turned out to be incredibly biased, and ultimately, useless.

The second problem with the Literary Digest poll was that out of the 10 million people whose names were on the original mailing list, only about 2.4 million responded to the survey. Thus, the size of the sample was about one-fourth of what was originally intended. People who respond to surveys are different from people who don't, not only in the obvious way (their attitude toward surveys) but also in more subtle and significant ways. When the response rate is low (as it was in this case, 0.24), a survey is said to suffer from nonresponse bias. This is a special type of selection bias where reluctant and nonresponsive people are excluded from the sample.

The Literary Digest "lost face" and later went out of business.

Chicago Tribune, 1948

In the 1948 presidential election, Thomas Dewey, then governor of New York, ran against the incumbent, Harry S. Truman. Three major polls (Gallup included) predicted Dewey would win. He did not. The Gallup poll came closest with 49.5% Dewey, 44.5% Truman, 5.5% other, and 0.5% undecided.

There were several problems:

First, they stopped polling too soon, and Truman was notably successful at energizing people in the last days before the election. The prediction came two weeks in advance of the election with 15% announcing undecided. It was assumed that the 15% undecided would split in the same proportions as those who had decided, leaving only 0.5% truly undecided. (The 0.5% undecided was Gallup's error factor.)

Second, the telephone polls tended to favor Dewey because in 1948 telephones were generally limited to more well-to-do households, and Truman was less popular among elite voters.

Third, one of the methods used sent interviewers into an area and told them to interview a certain number of people who met some given demographic criteria. For example, a pollster might be told to choose and interview 10 men, 11 women, 8 african americans, 2 asians, and 11 caucasians, etc... where the numbers chosen for each demographic is representative of the overall population.

Unfortunately for the pollsters, just because a sample is representative of the population demographically, doesn't mean that it will be representative with regard to the issue at hand.

Even worse, as long as the pollsters matched the quotas for their demographics, they were free to choose whom they like -- which can lead to substantial bias in the sample when pollsters avoid certain people that for some reason are hard to approach.

Draft Lottery, 1970

The days of the year, from 1 to 366, were written on slips of paper and the slips were placed in plastic capsules. The capsules for each month, starting with January and ending with December, were mixed in a shoebox and then they were dumped into a deep glass container. Capsules were then drawn from the jar one at a time.

The first number drawn was 258 (which corresponds to September 14). Men of draft age (those born between 1944 and 1950) whose birthday was Sept. 14 would then constitute the first group to be drafted. The second group of men to be drafted corresponded to birthdays agreeing with the second capsule/number drawn, and so on.

It was quickly noticed that men with birthdays in December seemed to get drafter earlier, on average, than birthdays in other months. With hindsight being 20/20, it's easy to see how this happened. The capsules were put into the jar month by month. So January capsules probably stayed on the bottom of the jar, while February capsules were on top of them, and so on, until December capsules ended up on top. Whatever subsequent mixing efforts were taken was not sufficient to completely overcome this ordering, producing a sample biased toward later months in the year.

Handout for Expert Group C: Are poll results presented in misleading charts or graphs?

Data visualization is one of the most important tools we have to analyze data. But it’s just as easy to mislead as it is to educate using charts and graphs. In this article we’ll take a look at three of the most common ways in which visualizations can be misleading. Charts or graphs can also ignore the ‘margin of error’ that should be reported by the poll. This is a statistic calculated using the number of responses to the poll and determining how much the poll results might be ‘off’. For example, a poll result reported as ‘75% of responders did not like sushi’; if about 1000 people responded to the poll, a margin of error would probably be around 3% one way or the other( 3% is a pretty good margin) meaning the accurate finding would be somewhere between 72 and 78%. A larger number of responders generally produces a smaller margin of error, and a large margin of error probably indicates a smaller sample size reporting on the poll.

One of the easiest ways to misrepresent your data is by messing with the y-axis of a bar graph, line graph, or scatter plot. In most cases, the y-axis ranges from 0 to a maximum value that encompasses the range of the data. However, sometimes we change the range to better highlight the differences. Taken to an extreme, this technique can make differences in data seem much larger than they are.

From Stevenson University. By

https://web.stevenson.edu/mbranson/m4tp/version1/fake-news-misleading-graphs.html. Accessed 11 May, 2026.

3.6 Misleading Graphs

Graphs can be a great way to display information in a way that helps people understand the relationships among numbers. For example, Figure 3.6.1 shows temperature changes over the last two thousand years. There is, of course, a lot of variation over time. Then, when we hit the Industrial Revolution, when

carbon emissions increased substantially, and the line starts trending upwards very quickly.

🔗

Figure 3.6.1. Graph of global average temperature change from year 1 CE to 2019. Image by RCraig09 from an original by Ed Hawkins, CC-BY-SA-4.0 1 . [3.10.1.23]Image Description 1.

Along the bottom, we see the years marked off in even increments. On graphs, the bottom line starting on the left and going right is the x-axis. On the left side, the temperature range is indicated. It shows the change in average temperature for each year relative to the temperature in the year 1 CE. That left side line with increments of some important variable is the y-axis. In some cases, the x-axis represents time and the y-axis indicates the degree of some phenomena. In other graphs you might see the x-axis as a causal variable and the y-axis as indicating some outcome. So, in the context of climate change, you might see a graph with the amount of carbon emissions on the x-axis and the average world temperature on the y-axis so the viewer can see how the two are related.

But because graphs appear to show objective truth, some people try to abuse that trust people have in graphed data to make things appear differently than they are. As a simple image, such graphs are easily shared online, sometimes with a caption indicating what the viewer is expected to believe as a result. There is a variety of ways to manipulate graphs, we’ll look at two. First, we’ll talk about truncating the y-axis. How wide of a range should you show on the y-axis? What should be the lowest value and what should be the highest? The question is: the y-axis range should go from what to what? In the case of counting things, you might expect the y-axis to range from zero to some plausible number for the most extreme case. But if you wanted to make something look like it was changing more over time you might include a smaller range in the y-axis, which we call truncating the y-axis.

Look at Figure 3.6.2. It shows data from the Uniform Crime Reports from the American F.B.I. [3.10.1.24]. There are plenty of criticism of the accuracy of these data, but setting aside those for a moment, we can use their data to see how to manipulate graphs. I have created a bar graph for the data about the number of murders in the United States in a way that makes it appear that murders went up a lot.

Figure 3.6.2. Graph of murders in the United States in 2018 and 2019.

But take a look at the y-axis. The lowest number is not zero, it is 16,340 murders in that year. Take a look at the graph in Figure 3.6.3 of the same data.

Figure 3.6.3. A better graph of murders in the United States in 2018 and 2019.

In this graph, the y-axis runs from 0 to 18,000 murders. On this graph, it might even be hard to tell which year had more murders unless you zoomed in or looked at it really closeWindow()ly. To be sure, it is horrible that over 16,000 people were murdered in each of those years. But it seems absurd to argue that there is a big increase in murders when the graph with zero as the lowest point is used.

🔗

Let’s look at the same problem with a line graph showing change over several years. In this case, we are looking at the number of robberies in the United States over several years. Figure 3.6.4 shows the change over time in the number of Robberies in the United States, again graphed from F.B.I. data [3.10.1.24].

🔗

Figure 3.6.4. Graph of robberies from 2014 to 2016.

🔗

Once again, it looks like robberies were really going up during that time. People seeing this graph might worry that if these trends continue, the streets will not be safe anywhere. They might start thinking about harsher penalties for crimes as possible solutions. But you have probably already noticed the y-axis here. The lowest point is 316,000 robberies, not zero. Figure 3.6.5 shows the same data graphed with zero robberies as the lowest point.

🔗

Figure 3.6.5. A better graph of robberies from 2014 to 2016.

🔗

Now it hardly looks like robberies are changing at all. They might have gone up slightly, but it is hardly a massive crime wave happening in 2016 that was not happening in 2014. With the graph in Figure 3.6.5, would you want to devote a lot of public resources to stemming the rising tide of robberies? No one wants to be robbed, but it is not exactly a terrifying increase.

🔗

But some of you may have also started wondering about the x-axis. Where’s the data for 2017, 2018, and 2019? The F.B.I. had data for those years for murders in the last graph, where are they for robberies? That gets to the other side of manipulating graphs. Just like you can shrink the range of the y-axis, you can pick whatever range you want for the x-axis. Sometimes people call that “cherry-picking” the data. You only show the data in a graph that supports an argument you want to make and you exclude the data you want to hide. Once again, we’re looking at shady methods of trying to manipulate people. Take a look at Figure 3.6.6, which shows the same graph as Figure 3.6.5, but includes the later years from the report [3.10.1.24].

🔗

Figure 3.6.6. A graph of robberies in the US from 2014 to 2019.

🔗

Someone who wanted to argue that the United States was becoming an increasingly dangerous place would be more persuasive if they presented Figure 3.6.5, than Figure 3.6.6. Figure 3.6.6 shows that there is a noticeable drop in robberies in more recent years after the slight uptick for a couple of years.

🔗

What counts as an honest graph that is not “fake news”? It is tricky to decide and a lot of it depends on the context. Consider the climate change graph back in Figure 3.6.1. It had a y-axis that ranged from -0.5 C to + 1.0 C. For those of you more accustomed to Fahrenheit rather than Celsius, that’s roughly -1 F to +2 F. You may be thinking, only two degrees hotter? I would barely notice that outside, is this a misleading graph? The thing to keep in mind with this graph is the context. Scientists think that every 0.5 C increase has extremely important impacts on our ability to survive on this planet [3.10.1.25]. Although it might seem like the graph is making a lot out of a little, it is probably a fair graph rather than a misleading one. It is important to consider the context when trying to judge where the y-axis and the x-axis should begin and end. Interpreting numbers always requires critical thinking about what is “a lot” and what is “a little.”

On the left, we’ve constrained the y-axis to range from 3.140% to 3.154%. Doing so makes it look like interest rates are skyrocketing! At a glance, the bar sizes imply that rates in 2012 are several times higher than those in 2008. But displaying the data with a zero-baseline y-axis tells a more accurate picture, where interest rates are staying static.

Expert Group C is to review how the results of the polls were published and see if either poll used misleading charts and graphs. Was the reported margin of error ignored in the chart or graph?

Handout for Expert Group D

The polling methods used in the poll can influence the results of the poll. Was the poll taken by phone call? In that case, were cell phones called as well as landline phones? If not, younger generation people who often do not have landline phones will be excluded from the poll. If door to door polling was used, what time of day was polling done? If during the day, only retired, unemployed, or stay-at-home family members would be polled. People who work outside the home during the day hours would not be polled.

In each of these cases, the poll results would not reflect the thinking of all the people selected as representative of the ‘target’ population.

From the Pew Research Center.

https://www.pewresearch.org/methods/2023/04/19/how-public-polling-has-changed-in-the-21st-century/#:~:text=Online%20opt%2Din%20polls.,take%20a%20one%2Doff%20survey. Accessed 4 March 2026

PEW RESEARCH CENTER APRIL 19, 2023

How Public Polling Has Changed in the 21st Century

61% of national pollsters in the U.S. used methods in 2022 that differed from those of 2016

BYCOURTNEY KENNEDY,DANA POPKYANDSCOTT KEETE

The 2016 and 2020 presidential elections left many Americans wondering whether polling was broken and what, if anything, pollsters might do about it. A new Pew Research Center study finds that most national pollsters have changed their approach since 2016, and in some cases dramatically. Most (61%) of the pollsters who conducted and publicly released national surveys in both 2016 and 2022 used methods in 2022 that differed from what they used in 2016. The study also finds the use of multiple methods increasing. Last year 17% of national pollsters used at least three different methods to sample or interview people (sometimes in the same survey), up from 2% in 2016.

Pollsters made more design changes after 2020 than 2016. In the wake of the 2016 presidential election, it was unclear if the polling errors were an anomaly or the start of a longer-lasting problem. 2020 provided an answer, as most polls understated GOP support a second time. The study found that after 2020, more than a third of pollsters (37%) changed how they sample people, how they interview them, or both. This compares with about a quarter (26%) who made changes after 2016. As noted above, though, these changes did not necessarily occur because of concerns about election-related errors.

The number of national pollsters relying exclusively on live phone is declining rapidly. Telephone polling with live interviewers dominated the industry in the early 2000s, even as pollsters scrambled to adapt to the rapid growth of cellphone-only households. Since 2012, however, its use has fallen amid declining response rates and increasing costs. Today live phone is not completely dead, but pollsters who use it tend to use other methods as well. Last year 10% of the pollsters examined in the study used live phone as their only method of national public polling, but 32% used live phone alone or in combination with other methods. In some cases, the other methods were used alongside live phone in

a single poll, and in other cases the pollster did one poll using live phone and other polls with a different method.

Several key trends, such as growth of online polling, were well underway prior to 2016. While the 2016 and 2020 elections were consequential events for polling, the study illustrates how some of the methodological churn in recent years reflects longer-term trends. For example, the growth of online methods was well underway before 2016. Similarly, some live phone pollsters had already started to sample from registered voter files (instead of RDD, random-digit dialing) prior to 2016.

Group D is to examine the poll information to determine the polling methods used and consider whether the poll results were slanted or skewed in some way because of those methods.

Handout for Expert Group E

Information needs to be examined for accuracy. This is true for news stories, news broadcasts, social media postings, even school textbooks. And of course published results of opinion polls and surveys are near the top of the list for examination.

Who paid for this (survey, news article, etc.)
Who wrote this?
For surveys and opinion polls: Are the questions available to review? If not, make no assumptions. If they are available:

Who worded the questions
Are the questions directed at the topic of the poll?
Are the questions phrased in a neutral manner

What do other articles, surveys, websites say about this issue?

Examples of slanted wording: the following two surveys were sent to specific subgroups of people within the target populations.

From “2026 California Senior Rights Survey” by the’National Committee to Preserve Social Security and Medicare’

Which of the following statements td you agree with? (check all that apply)

__ Social Security is my right as an American

__ I have earned by social security benefits through my hard

work

— I have paid for my Social Security benefits through

deduction during my working years

__ None of the above

Do you approve of Social Security and Medicare funds beiing cut to pay for other fiscal priorities, like reducing the budget deficits and offsetting massive tax cuts for the wealthy?

__ Yes

__ No

__ Unsure

From National Opinion survey on the Marxist Hijacking of American Education

Commissioned by Hillsdale College

Critical race theory has a divisive effect in our schools, increasing race consciousness and racial division in the name of’anti-racism.’

– Strongly agree

_ Agree

– Disagree

_ Strongly disagree

_ Insure

Group E is to examine the two polls and ask these questions of each poll. If the information is not available, consider whether the accuracy of the poll might be slanted or skewed because it was intended by whoever paid for the poll wanted to see certain results.

The two polls to be examined: one by Gallop, and one by the American Petroleum Institute

Gallup Poll: A long time national polling company, Gallup polls about many subjects. Usually has about 1200 responses to a specific poll and margin of error around 3%. This means that, for example, if 55% favorable was reported as the response to a que

stion, with 95% certainty the actual answer lies between 52 and 58% favorable, Smaller response samples usually have larger margins of error.

https://news.gallup.com/poll/2167/energy.aspx. Accessed 3 March 2026

(Edited to show findings about energy)

A long time national polling company, Gallup polls about many subjects. Usually has about 1200 responses to a specific poll and margin of error around 3%. This means that, for example, if 55% favorable was reported as the response to a question, with 95% certainty the actual answer lies between 52 and 58% favorable, Smaller response samples usually have larger margins of error.

Do you think the United States will have an energy shortage in the next five years?

Do you think that the United States is or is not likely to face a critical energy shortage during tnext five ye

Yes, is

No, is not

Already facing one (vol.)

No opinion

2022 Mar 1-18

2021 Mar 1-15

2016 Mar 2-6

2012 Mar 8-11

2011 Mar 3-6

2010 Mar 4-7

2008 Mar 6-9

2007 Mar 11-14

2006 Mar 13-16

2005 Mar 7-10

2004 Mar 8-11

2003 Mar 3-5

2002 Mar 4-7

2001 Mar 5-7

1978 Nov 10-13

2001 Apr 6-9 ^

2001 Mar 5-7 ^

Do you think that as a country, the United States should put more emphasis, less emphasis or about the same emphasis as it does now on producing domestic energy from each of the following sources -- [RANDOM ORDER]?

More emphasis

Less emphasis

Same emphasis

No opinion

Coal

2021 Mar 1-15

2019 Mar 1-10 ^

2015 Mar 5-8

2013 Mar 7-10

Oil

2021 Mar 1-15

2019 Mar 1-10 ^

2015 Mar 5-8

2013 Mar 7-10

Natural gas

2021 Mar 1-15

2019 Mar 1-10 ^

2015 Mar 5-8

2013 Mar 7-10

Wind

2021 Mar 1-15

2019 Mar 1-10 ^

2015 Mar 5-8

2013 Mar 7-10

Solar power

2021 Mar 1-15

2019 Mar 1-10 ^

2015 Mar 5-8

2013 Mar 7-10

Nuclear power

2021 Mar 1-15

2019 Mar 1-10 ^

2015 Mar 5-8

2013 Mar 7-10

^ Asked of a half sample

We'd like to ask you about proposals to dramatically reduce the use of fossil fuels such as gas, oil and coal in the U.S. within the next 10 or 20 years, in order to reduce greenhouse gas emissions. Would you strongly favor, favor, oppose or strongly oppose establishing policies aimed at achieving this?

Strongly favor

Favor

Oppose

Strongly oppose

No opinion

2019 Mar 1-10 ^

Would you prefer the government to increase, decrease or not change the financial support and incentives it gives for producing energy from ... ?

Increase

Decrease

Not change

No opinion

Alternative sources such as wind and solar

2009 Mar 5-8

Traditional sources such as oil and gas

2009 Mar 5-8

Poll by the American Petroleum Institute

https://www.api.org/news-policy-and-issues/news/2024/08/15/new-poll-swing-state-voters-support-policies-to-encourage-oil-gas-production#:~:text=WASHINGTON%2C%20August%2015%2C%202024%20%E2%80%93%20The%20American,production%20and%20limit%20reliance%20on%20foreign%20sources.

Accessed 3 March 2026

Issues / News & Press Releases / New Poll: Swing State Voters Support Policies to Encourage Oil and Natural Gas Production

Poll shows overwhelming support for permitting reform; strong opposition to energy and vehicle mandates

WASHINGTON, August 15, 2024 – The American Petroleum Institute (API) today released new battleground state polling conducted by Morning Consult demonstrating widespread support for policies that encourage domestic oil and natural gas production and limit reliance on foreign sources. The poll shows inflation remains a top concern for voters in Arizona, Georgia, Michigan, Nevada, North Carolina, Pennsylvania and Wisconsin, and an overwhelming majority oppose vehicle mandates. As Congress continues to debate bipartisan permitting reform legislation, voters in battleground states expressed strong support for reforms to streamline the approval process for energy infrastructure projects.

With less than 100 days until Election Day, API is urging policymakers on both sides of the aisle to support commonsense energy policies, including those outlined in API’s Five Point Policy Roadmap to unleash America’s energy security and help reduce inflation.

“The U.S. continues to be a global leader in energy production, but the American people recognize that our leaders in Washington must advance an agenda to grow our nation’s energy advantage for decades to come,” API President and CEO Mike Sommers said. “With geopolitical tensions rising and inflation remaining a top concern, we need policies that reinforce the role of American energy on the world stage and support access to the affordable, reliable energy consumers need.”

The poll conducted by Morning Consult found:

More than 9 in 10 voters are concerned about inflation. (AZ: 94% GA: 93% MI 94% NV: 94% NC: 92% PA: 91% WI: 91%)
8 in 10 voters agree that producing more oil and natural gas here in the U.S. could help lower energy and utility costs for American consumers. (AZ: 83% GA: 88% MI: 84% NV: 86% NC: 85% PA: 83% WI: 80%)
A majority of voters oppose government mandates that restrict consumer choice, including banning new gasoline, diesel and hybrid vehicles. (AZ: 69% GA: 76% MI: 80% NV: 75% NC: 75% PA: 77% WI: 78%)
Approximately 8 in 10 voters support fixing our broken permitting system to streamline the process of approving energy infrastructure projects. (AZ: 79% GA: 81% MI: 82% NV: 81% NC: 80% PA: 84% WI: 80%)
More than 8 in 10 voters agree producing oil and natural gas here in America helps make our country more secure against foreign adversaries. (AZ: 89% GA: 91% MI: 87% NV: 85% NC: 88% PA: 86% WI: 87%)
More than 8 in 10 voters support leveraging America’s domestic resources rather than relying on other regions of the world. (AZ: 85% GA: 89% MI: 86% NV: 82% NC: 82% PA: 86% WI: 84%)
Nearly 8 in 10 voters support advancing a sensible federal tax policy that encourages American energy production and strengthens the country’s economic and energy security. (AZ: 82% GA: 85% MI: 80% NV: 76% NC: 80% PA: 78% WI: 76%)

View the full poll results by state here.

Methodology: These polls were conducted between August 2-7, 2024, among a sample of Registered Voters (AZ=600; GA=602; MI=602; NV=400; NC=601; PA=600; WI=502). The interviews were conducted online, and the data were weighted to approximate a target sample of Registered Voters by State of Residency based on race, educational attainment, marital status, home ownership, 2020 presidential vote, and gender by age. Results from the full surveys in the individual states have a margin of error of plus or minus 4 percentage points, except for Nevada which has a margin of error of plus or minus 5 percentage points.

API represents all segments of America’s natural gas and oil industry, which supports nearly 11 million U.S. jobs and is backed by a growing grassroots movement of millions of Americans. Our approximately 600 members produce, process and distribute the majority of the nation’s energy, and participate in API Energy Excellence®, which is accelerating environmental and safety progress by fostering new technologies and transparent reporting. API was formed in 1919 as a standards-setting organization and has developed more than 800 standards to enhance operational and environmental safety, efficiency and sustainability.

State by state results and poll questions below

TO: The American Petroleum Institute (API)

FR: Morning Consult

DT: August 14, 2024

RE: Battleground State Voters’ Views on Energy Policy Issues

Voters in electoral battleground states are concerned about inflation and are experiencing

financial pain, and they see a role for domestic natural gas and oil production in helping

consumers and small businesses.

• Over nine-in-ten voters in every state tested are very or somewhat concerned about inflation.

How concerned are you about inflation?

• Voters experience financial strain or notice other Americans experiencing financial strain due to the

current price of day-to-day necessities (81%).

Do you feel the price of day-to-day necessities is causing you and other Americans financial pain?

Methodology: This poll was conducted between August 2-7, 2024, among a sample of 3,907 Registered Voters in AZ, GA, MI, NV, NC, PA, and WI.

The interviews were conducted online, and the data were weighted to approximate a target sample of Registered Voters by State of Residency based

on race, educational attainment, marital status, home ownership, 2020 presidential vote, and gender by age. Results from the full survey have a margin

of error of plus or minus 4 percentage points. Some of the percentages do not total 100% due to rounding.• Voters believe producing more natural gas and oil here in the United States could help lower energy

and utility costs for American consumers and small businesses.

Do you agree or disagree with the following statement? Producing more oil and natural gas here in the U.S. could

help lower energy and utility costs for American consumers and small businesses.

Voters support policies that encourage American energy production, streamline energy

infrastructure building, and strengthen the nation’s energy security.

• The majority of voters oppose government mandates that would ban gas stoves, gas furnaces, or

new gasoline, diesel and hybrid vehicles.

Do you support or oppose government mandates that would ban gas stoves, gas furnaces or new gasoline, diesel

and hybrid vehicles?

Methodology: This poll was conducted between August 2-7, 2024, among a sample of 3,907 Registered Voters in AZ, GA, MI, NV, NC, PA, and WI.

The interviews were conducted online, and the data were weighted to approximate a target sample of Registered Voters by State of Residency based

on race, educational attainment, marital status, home ownership, 2020 presidential vote, and gender by age. Results from the full survey have a margin

of error of plus or minus 4 percentage points. Some of the percentages do not total 100% due to rounding.

• Four-in-five voters support permitting reform to streamline the federal government’s approval process for energy infrastructure. Support for permitting reform cuts across party affiliation.

Do you support or oppose permitting reform which would streamline and speed up the federal government’s years

long process for approving energy infrastructure projects here in America?

• About four-in-five voters support a federal tax policy that encourages American energy production

and strengthens the nation’s energy security.

Do you support or oppose a federal tax policy that encourages American energy production and strengthens

the country’s economic and energy security?

.The majority of voters agree producing natural gas and oil in the United States promotes

energy independence and improves national security

• Voters support increasing the domestic production of natural gas and oil to help limit U.S. reliance on other countries for energy sources.

Do you support or oppose the U.S. increasing its own domestic sources of oil and natural gas rather than relying on

other regions of the world?

• More than eight-in-ten voters agree producing oil and natural gas here in America makes the country

more secure against actions by countries such as China and Russia.

Do you agree or disagree with the following statement? Producing oil and natural gas here in America helps make our

country more secure against actions by countries such as China and Russia.

• More than eight-in-ten voters agree producing oil and natural gas here in America makes the country

more secure against actions by countries such as China and Russia.

Inland Empire Council for the Social Studies

Famous Statistical Blunders in History

Literary Digest, 1936

Chicago Tribune, 1948

Draft Lottery, 1970

3.6 Misleading Graphs

How Public Polling Has Changed in the 21st Century

61% of national pollsters in the U.S. used methods in 2022 that differed from those of 2016

Poll shows overwhelming support for permitting reform; strong opposition to energy and vehicle mandates