The Effects of Satisficing on Survey Response Lonna Atkeson Alex Adams University of New Mexico
Data Quality • Important for public opinion scholars • Motivation and engagement of respondents contributes to data quality – Some respondents don’t pay much attention increasing the noise in the data and reducing the signal to noise ratio – reducing our power to answer hypotheses with precision and accuracy – What are the effects of satisficing on the point estimate?
Cognitive Process of Survey Response • Comprehension: respondents interpret the words into a meaningful question • Retrieval: involves the scan of memory to place the question in context for the respondent and pull necessary information to form an accurate opinion • Judgment: aggregation of the information into an opinion • Response: matches the judgment to the response options
Cognitive Process • Optimizing survey response • Satisficing survey response – Questions are demanding • Economic Exchange vs social exchange • Definitions • Differentiate between targets – Surveys are long • ANES FTF averages 90 minutes, Internet 83 & 94 – Respondents motivation and engagement may be low or may wane
Problems with Satisficing • Question: Do satisficers have significantly different opinions than optimizers on question where there is not an easy satisficing response (no middle category, no neither agree nor disagree, ½ full ½ empty?) • If so, when do satisficers have significantly different opinions than optimizers?
Theory: Easy Versus Hard questions • Cognitively easy versus hard questions matter • Easy: an issue that has ideological content, is symbolic and relates to policy ends (e.g. abortion); an issue that is about oneself or a well known target; a gut issue –there should be no difference between optimizers and satisficers • Hard: a hard issue has low ideological content and pertains to policy means; an issue that is about someone else, an issue that requires thought –there should be a significant difference between optimizers and satisficers
How do satisficers respond? • Primacy • Recency • Acquiescence
Data • ANES 2012 Time Series – FTF – On-line Gfk panel • Model Controls: Mode, education, black, Hispanic, other race, married, employed, age, strength of partisanship and strength of ideology
Defining Satisficers • Scale of nondifferentiation • Five pre-election questions on ideology –7 point scale: Where would you place [TARGET] on this scale, or haven't you thought much about this? (Extremely liberal, liberal, slightly liberal, moderate/middle of the road, slightly conservative, conservative, extremely conservative) – Self, Democratic Presidential Candidate, GOP Presidential Candidate, Democratic Party, GOP Party – Counted the number of times the R chose the same response option across all items
LIberal Moderate Conservative DK/REF 0 46.4 48.1 40.7 93.1 1 32.0 30.4 33.9 1.9 2 17.9 12.1 20.1 1.7 3 3.5 6.1 5.2 .6 4 .15 1.6 .10 2.3 5 .02 1.8 .05 .5 N 5914 5914 5914
Dependent Variables Group 1 • Easy Issues • Self – Abortion: There has been some discussion of abortion in recent years. Which one of the opinions on this page best agrees with your view? By law abortion should never be permitted, the law should permit abortion only in the case of rape, incest, or when the mother’s life was in danger, the law should permit abortion for reasons other than rape, incest or when the mother’s life is in danger, by law a woman should always be able to obtain an abortion as a matter of personal choice. • Hard Issue – Abortion Presidential Candidates : Where would you place [TARGET] on abortion? Same response. – Abortion Political Parties: Where would you place [TARGET] on abortion? Same response.
Results Groups 1: Abortion Self Obama Romney Dem Pty GOP Pty St. Satisficers -.15 -.309*** .266*** -.389*** .387*** Weak Satisficers -.05 -.012 -.07 -.087* .028 Mode .02 -.020 .09** .076* -.068* R-square .17 .087 .036 .111 .074 N 5515 5044 4995 4988 4972
% Of Responses of Satisficer and Optimizers Abortion Self Abortion Obama Abortion Romney Opt Sat Opt Sat Opt Sat Never 11 18 5 11 38 37 For rape, etc 27 28 13 26 44 31 For other reasons 15 15 16 17 11 15 Always 46 38 66 46 7 17
Dependent Variable Group 2: Easy Issue: • Presidential Approval: Do you APPROVE or DISAPPROVE of the way Barack Obama is handling his job as President? Do you [approve/disapprove] STRONGLY or NOT STRONGLY? Hard Issue: • Congressional Approval: Do you APPROVE or DISAPPROVE of the way the U.S. Congresshas been handling its job? Do you [approve/disapprove] STRONGLY or NOT STRONGLY?
Results: OLS Presidential Approval Congressional Approval Strong Satisficers .063 .213*** Weak Satisficers .033 .127** Mode -.121*** .213** R-Square N 5458 5227
Dependent Variables: Group 3 • Hard issues: – Belief in various conspiracy theories • Birthers: Was Barack Obama definitely born in the United States, probably born in the United States, probably not born in the United States or definitely not born in the United States? • 911: Did senior federal government officials definitely know about the terrorist attacks on September 11, 2001 before they happened, probably knew about the terrorist attacks on September 11, 2001 before they happened, probably did not know about the terrorist attacks on September 11, 2001 before they happened, or definitely knew about the terrorist attacks on September 11, 2001 before they happened?? • Death panels: Does the health care law passed in 2010 definitely authorize government panels to make end-of-life decisions for people on Medicare, probably authorize government panels to make end of life decisions for people on Medicare, probably not authorize government panels to make end of life decisions for people on Medicare, or definitely not authorize government panels to make end-of-life decisions for people on Medicare • Hurricane Katrina: Some people say that when Hurricane Katrina hit the Gulf Coast in the summer of 2005, the federal government intentionally breached flood levees in New Orleans so that poor neighborhoods would be flooded and middle-class neighborhoods would be spared. Do you think the federal government definitely did this, probably did this, probably did not do this, or definitely did not do this?
OLS Results Birthers Death Panels 911 Katrina Strong Satisficers -.145* .042 .213*** .280*** Weak Satisficers -.029 .005 -.003 .053 Mode .018 .140*** .052 .087** R-Square .265 .166 .056 .103 N 5015 4759 5079 5047
Discussion • We find evidence that satisficers have different opinions on hard issues, but not on easy issues. • We cannot find a consistent picture of how satisficers may be responding, it looks somewhat random • When satisficers are a large part of the sample it may compromise the quality of data. – One study suggests that between 14% and 46% of respondents are not motivated and are not attentive