Designing the Best Questionnaires: Using Psychology to Maximize the Accuracy of Your Measurements,
Two-day shortcourse taught by Professor Jon A. Krosnick (Stanford University), 5th and 6th May
This is a two-day course on questionnaire design and measurement, organised by the LSE's Methodology Institute, IPSOS-MORI and the Department of Sociology at the University of Essex.
-
Day 1 Morning Session 10:00 – 13:00 TW1.U8, Tower 1
-
Day 1 Afternoon Session 13:45 – 16:00 NAB 2.04, 2nd Floor, New Academic Building
-
Day 2 All Day STICERD Conference Room LRB 505, 5th floor, Lionel Robbins Building
Dr. Krosnick's M.A. and Ph.D. are in social psychology from the University of Michigan, and for 18 years, he was professor of psychology and political science at Ohio State University. He is now Professor of Communication, Political Science, and Psychology at Stanford University, where he holds the Frederic O. Glover Professorship in Humanities and Social Sciences and is Director of the Methods of Analysis Program in the Social Sciences, Associate Director of the Institute for Research in the Social Sciences, and Principal Investigator of the American National Election Studies.
Surveys and questionnaires are a common way of gathering data in the social sciences. The structuring, wording and ordering of questions has traditionally been viewed as an art, not a science, best guided by intuition. But in recent years, it has become clear that this is an antiquated and even dangerous view that does not reflect the accumulation of knowledge throughout the social sciences about effective question-asking. Intuition often leads us astray in the questionnaire design field, as becomes clear when putting intuitions to the test via scientific evaluation. A large body of relevant scientific studies has now accumulated, and when taken together, the findings point to a series of formal rules for how best to design questions. Yet the vast majority of contemporary questionnaire design fails to follow these rules, because they are not yet widely understood. During the past 25 years, Jon Krosnick has been gathering up the huge body of evidence on optimal questionnaire design for this forthcoming book, The Handbook of Questionnaire Design. In this course, Dr. Krosnick will review the contents of the first half of the book.
How could this body of scientific knowledge have been overlooked for so long? The reason is that it is scattered across the publication outlets of numerous disciplines, and this literature has never been comprehensively gathered and integratively reviewed in a central place. Doing so has been a principal project for Jon Krosnick during the past twenty years. In doing this work, he has discovered the emergence of general principles of optimal questionnaire design that are often at variance with common practices in questionnaire construction. And in many cases, these departures in practice have reduced the accuracy of collected data, even though other question forms could easily have been employed instead that would have avoided bias and improved precision of measurement.
Because of the complexity of this literature, it does not yield a short and simple list of rules to guide every decision to be made by a questionnaire designer, each supported by a few documentary references, and each obviously justified by all relevant studies. The issues addressed are multifaceted, and many are still in the process of being resolved by innovative and creative new research. But there is a great deal of richness in the existing literature that provides useful guidance for scholars interested in maximizing the reliability, validity, and efficiency of the measurement instruments they employ in their research. Reviewing this literature and its implications is the focus of the two-day course.
Goals of the two-day course
The two-day course serves many purposes. First and foremost, it educates participants about the optimal techniques for questionnaire design, for guarding against measurement artifacts, and for analyzing data in order to overcome the biasing impact of such artifacts. And because the huge accumulated literature on questionnaire design does not provide guidance to researchers about how to handle every decision they will face, the second and equally important goal of the course is to teach participants a new way of thinking about questionnaire design, rooted in psychology. The aim is to get researchers into the heads of their respondents in a way that few have done before. By gaining insight into how respondents approach their tasks of interpreting questions, searching their memories for information, integrating that information into judgments, and expressing those judgments in words, workshop participants will begin to develop skills that will help them to mange design issues for which there are not yet formal rules.
In addition to helping participants to design better questionnaires, the two-day course will equip analysts of questionnaire data to look from a new perspective as they evaluate the meaning of their findings, knowing how measurement artifacts can lead them astray.
Audience
The two-day course will be engaging and useful for participants with a wide variety of backgrounds. In the academic world, questionnaire designers and analysts of questionnaire data can be found throughout the social sciences. Such scholars work in political science, psychology, sociology, economics, communication, health, business, law, journalism, anthropology, the health professions, business, and many other disciplines. Individuals from all of these areas of the field would find this course useful for improving their work.
Outside of academia, researchers at marketing and advertising firms, survey research firms, the news media, consulting companies, the research arms of government, and research departments of manufacturing and service corporations all routinely design questionnaires and use questionnaire data.
In short, whether a researcher uses questionnaires in laboratory experiments involving 50 participants or in large-scale representative sample surveys of tens of thousands of respondents or simply reads and interprets questionnaire-based data collected by others, this course is intended to help him or her do better work.
During twenty years of experience teaching this workshop, it has become clear that participants benefit no matter what their level of expertise and experience with questionnaire design and data analysis. For people who are new to questionnaire design, this can be an introduction to design issues they had never thought much about before. For people with a great deal of questionnaire design experience, the workshop challenges many of their long-standing assumptions and energizes their high-level thinking about the questionnaire response process, often leading them to become leaders in advancing our growing understanding of this phenomenon.