How Can You Be So Sure? Seeking the Gold Standard of Survey Accuracy

By Tom Miller

Humans are programmed to find solutions. But, even so, we’re not that great at it. Every day, from kids to public managers, people decide what works and what’s good. A small conclusion can affect your lunch while a big one might change the lives of 5 million immigrants.

Sure, we rely on guts and brains to make our choices. But if we really explore the foundations of our decisions, we might recognize that we often are shooting in the dark – or concluding wrongly that we can see clearly what the right actions should be. In the public sector knowing what the public thinks shines a beacon on what otherwise can be a tangle of possible paths to national well-being. Insight into citizen sentiment helps illuminate the direction that democratic institutions should follow.

Bill Clinton was pronounced the poster child for political decisions that were based on the winds of public opinion . Since then almost every president has suffered the same accusation. But knowing what the public thinks is neither a malicious activity nor an exercise reserved for national politics. Local councils and government staff, too, rely on survey results to set the course of communities. Elected officials use surveys to allocate resources, assess the success of policies, track performance or make long-range plans.

If surveys deliver public opinion used for setting and tracking community direction, how can we be confident they are accurate?

I’ve written elsewhere that the traditional touchstone of surveys is how well they predict or match voting. By this standard, the national and statewide polls generally did well in the 2014 midterms, calling most races correctly and getting within about 4 percentage points of the actual vote outcomes. But in local government where surveys of residents have become a management best practice, there rarely exists a behavioral marker like voting results by which to judge the accuracy of a survey.

The value of an accurate survey is not in the opinions collected from the few hundred or so who respond. The value comes only from the extent to which those hundreds report perspectives or behaviors that reflect the thinking or activities of the rest of the community for which they are only proxies.

So how can we be so sure that when a survey shows that respondents think the water is bad, we have solid evidence that the population thinks the water is bad? Or if survey results show low ratings for clean streets that the target population of adults thinks the streets are not clean? Or if respondents think their economic future is bright are they really representing the same sanguine sentiment as their neighbors? The second part of this blog, appearing next month, will propose answers useful to any leader who relies on survey data to help make decisions.


3 thoughts on “How Can You Be So Sure? Seeking the Gold Standard of Survey Accuracy

  1. Great question, Matt. Too few managers consider the quality of the data they rely on – especially survey data. Years ago, colleagues and I studied the quality of citizen surveys conducted across America and published our results in PAR (“Assessing Excellence Poorly”, 1991). A new meta-analysis is overdue.


  2. Ostensibly many political polls are done properly and are scientifically based. But how often do local governments go to the trouble and expense of a properly designed and carried out survey? In my experience, many local government surveys are either mailed out with water bills or posted online, or they hold a series of town hall meetings and ask for comment. And we all know that only the most motivated/passionate of the public participate in those sorts of activities, creating a high potential for skewed results. Has there ever been a study done of local government studies – how many are scientifically done versus how many are essentially anecdotal based?


Tell Us What You Think

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s