The Hook: Why Your Survey Might Be Asking the Wrong Question
You spend weeks designing a survey. You recruit respondents and collect data. Then you run the analysis and the results look… off. Also, the model barely fits. The willingness-to-pay estimates are absurdly high. Because of that, you carefully craft choice scenarios. Something feels wrong, but you can't quite put your finger on it.
Sound familiar?
Turns out, the problem often isn't your data or your model. On the flip side, the field has moved on. It's that you're still designing your stated preference study like it's 1995. The best practitioners now think differently about everything from attribute design to sample selection to how they frame trade-offs That's the whole idea..
Here's the thing — contemporary guidance for stated preference studies isn't just about better statistics. It's about building studies that actually reflect how real people make decisions.
What Is Stated Preference?
At its core, stated preference (SP) methodology asks people to make choices in hypothetical scenarios. Plus, unlike revealed preference — where you observe what people actually did — SP gives you control over the attributes people face. You decide what options appear, what prices are shown, what features are available.
This matters because it lets you isolate specific factors. You can hold everything else constant and just change that one variable. But want to know how much people value faster internet speeds? Want to understand trade-offs between cost and quality? Present clear choices and watch what people pick Worth keeping that in mind..
Counterintuitive, but true.
But here's what most people miss: SP isn't just about presenting options. It's about presenting realistic options that mirror the decision environment people actually face Simple, but easy to overlook..
The Shift From Lab to Life
Early stated preference studies looked like lab experiments. Clean, controlled, artificial. People were presented with abstract scenarios and asked to choose.
Contemporary guidance says: stop pretending people live in a vacuum.
Modern SP work recognizes that context shapes choices. The way you describe alternatives, the order you present them, even the time of day someone takes your survey — these all matter more than we used to admit.
Why It Matters
Get stated preference right, and you access insights that revealed preference simply can't give you. You can estimate demand for products that don't exist yet. You can value non-market goods like clean air or reduced risk. You can test policy interventions before rolling them out The details matter here..
But get it wrong, and you waste resources, mislead stakeholders, and potentially make decisions based on garbage data The details matter here..
I've seen cities spend millions on infrastructure projects based on SP studies that assumed people would accept ridiculous trade-offs. I've seen companies launch products nobody wanted because their surveys asked about features in isolation rather than as part of a coherent package Worth keeping that in mind..
The short version: when stated preference studies go wrong, real consequences follow Not complicated — just consistent..
What Changes When You Do It Right
When you apply contemporary guidance, your results become more predictive. Think about it: your willingness-to-pay estimates become more realistic. Think about it: your models fit better. More importantly, your findings actually inform decisions in ways that stakeholders trust.
How Contemporary Guidance Actually Works
Let's break down what modern stated preference practice looks like in action.
Start With the Decision Context
This is where most studies fall apart. Researchers jump straight to designing choice tasks without really understanding what decision people are making.
Contemporary guidance says: map the actual decision process first. Worth adding: what constraints do they face? Day to day, what information do people have when they make this choice? What alternatives are realistically available?
I worked on a transportation study once where the team spent months designing elaborate choice scenarios — only to realize that most commuters in the area didn't actually have meaningful mode choices. Here's the thing — they were stuck with what was available. The entire study needed to be reframed around a different decision: whether to relocate for work No workaround needed..
Design Attributes That People Actually Process
Here's what most people get wrong: they treat every attribute as equally important. Practically speaking, in reality, people have cognitive limits. They can only process so much information before they start making random choices or dropping out entirely Which is the point..
Modern guidance suggests:
- Limit the number of attributes per choice task (usually 4-6 works well)
- Make sure attribute levels are realistic and familiar
- Avoid dominated alternatives unless you're specifically testing for them
- Think about how people actually evaluate trade-offs in real life
Not the most exciting part, but easily the most useful.
Use Efficient Design — But Don't Overdo It
Efficient experimental design has been around for decades, but contemporary practice has refined how we use it. The goal isn't just statistical efficiency — it's cognitive efficiency too Less friction, more output..
A well-designed efficient experiment presents people with choices that are informative for your model while remaining realistic and engaging. But here's the catch: ultra-efficient designs often look nothing like real decisions. People notice when every choice task seems carefully constructed to extract maximum information.
The sweet spot: use efficient design principles but build in enough realism that respondents don't feel like lab rats.
Sample Selection Matters More Than You Think
Gone are the days when you could just recruit any sample and call it good. Contemporary guidance emphasizes matching your sample to your target population — not just demographically, but behaviorally Surprisingly effective..
If you're studying electric vehicle adoption, don't just survey people who already drive. In practice, include people who don't drive at all. Include people who bike or take transit. Include people who say they'd never consider an EV Simple as that..
Why? Because the people who say "never" often tell you the most about what barriers exist.
Think About Response Strategies
People don't always choose the option they prefer. Sometimes they avoid extremes. Sometimes they choose the option that seems safest. Sometimes they just get tired and pick randomly Easy to understand, harder to ignore..
Contemporary guidance acknowledges these response strategies and designs studies that either account for them or minimize their impact.
This means thinking about survey length, the order of tasks, whether you include "none of the above" options, and how you handle dominant alternatives.
Common Mistakes That Still Happen
Despite all the advances, I still see the same mistakes over and over.
Treating All Attributes as Independent
This is probably the biggest one. Day to day, researchers design choice tasks where attributes vary independently, assuming people evaluate each attribute separately. But real decisions involve compensatory and non-compensatory trade-offs That's the part that actually makes a difference..
People don't just add up utility points. So they think about packages. They have reference points. They consider what's "good enough.
Ignoring the Description of Alternatives
How you describe options matters enormously. Practically speaking, calling something "eco-friendly" versus "energy-efficient" versus "low-emission" can change responses significantly. Contemporary guidance emphasizes careful wording and pre-testing of descriptions Simple, but easy to overlook..
Overlooking Sample Diversity
I've reviewed studies where the entire sample came from a single online panel. That said, sure, you got responses. But you also got systematic bias that made the results useless for the broader population.
Forgetting About Engagement
Long surveys with repetitive tasks lead to satisficing — people just trying to finish quickly rather than answering thoughtfully. Modern SP work builds in engagement checks and varies task formats to keep people attentive And that's really what it comes down to..
Practical Tips That Actually Work
Here's what I've learned works in practice:
Pilot Test With Real People
Not just colleagues. Not just other researchers. And actual people who represent your target population. Watch them complete the survey. Which means listen to how they talk about the choices. You'll catch problems you never would have anticipated Still holds up..
Keep It Short, But Not Too Short
There's a sweet spot for survey length. Too short and you don't get enough data. Too long and quality drops. Most successful studies I've seen fall in the 15-25 minute range.
Use Visual Design Strategically
Choice tasks that are well-formatted and visually clear perform better. In practice, use spacing, bolding, and layout to help people focus on what matters. But don't go overboard — visual complexity can be distracting And that's really what it comes down to..
Include Attention Checks
Not as a way to exclude people, but as a way to understand your data quality. If someone fails attention checks consistently, that tells you something about your survey design, not just about that respondent.
Plan Your Analysis Before You Collect Data
This sounds obvious, but you'd be surprised how often it doesn't happen. Know what you'll do with dominated alternatives. On the flip side, know what model you're going to run. Know how you'll handle missing data.
FAQ
How many choice tasks should respondents complete?
Most contemporary guidance points to 12-16 choice tasks per respondent. Fewer than 8 and you risk noisy data. Think about it: more than 20 and you risk fatigue effects. The exact number depends on complexity and your target population Worth keeping that in mind..
Should I use a none-of-these options?
Yes, almost always. People face opt-out
Should I use a none‑of‑these options?
Yes, almost always. People face opt‑out when none of the presented alternatives truly reflect their situation or preferences. But including a well‑crafted “none of the above” (or “not applicable”) choice preserves realism and prevents forced selections that could introduce bias. When you do add this option, make sure it is mutually exclusive from the other alternatives and that the wording is unambiguous (e.That's why g. , “None of these options describe my situation”). This small addition often improves the validity of the data without inflating the number of tasks Surprisingly effective..
How should I order the choice tasks?
Order effects can subtly shape respondent behavior. Consider this: research suggests that placing the easiest or most distinctive tasks early can boost engagement, while ending with a “catch” task (one that includes an obvious red flag) helps verify attention. Randomizing the position of alternatives within each task is advisable, but keep the overall sequence of tasks stable across respondents to maintain comparability Small thing, real impact..
What’s the best way to handle “dominant” alternatives?
A dominant alternative is one that is clearly superior to the others, either because of superior attributes, lower cost, or higher perceived benefit. If you leave dominant options in the design, respondents may gravitate toward them, reducing variability and potentially skewing parameter estimates. Two practical remedies are:
- Attribute reduction – strip the dominant option of its advantageous attributes, making it more comparable.
- Constraint imposition – add a realistic limitation (e.g., “only available on weekdays”) that nullifies the dominance.
Both approaches preserve the informational value of the choice set while mitigating bias Simple as that..
How do I treat missing data?
Missing responses can arise from item non‑response, skip logic, or incomplete attention checks. g.Also, the first step is to conduct a missing‑data audit: identify patterns (e. , entire blocks missing, random item omission) and assess whether missingness is random or systematic Not complicated — just consistent..
- Imputation – use regression‑based or multiple‑imputation techniques for small amounts of missing data, but be cautious when entire choice tasks are missing, as they can alter the choice set structure.
- Exclusion – remove respondents with extensive missingness, but document the decision and its impact on sample size and power.
- Model‑based handling – some advanced choice‑model frameworks (e.g., mixed logit) can accommodate partial data, but they require careful specification.
How should I report the sample size and power?
Transparency about sample size is essential for reproducibility. Report the final number of completed surveys, the response rate, and any trimming performed (e.In practice, g. But , due to failed attention checks). Conduct a priori power analyses using the expected effect size and number of choice tasks to justify the target sample. Post‑hoc power reports are helpful but should be framed as estimates rather than definitive guarantees.
Conclusion
Designing a discrete‑choice experiment that yields reliable, actionable insights is a balancing act between methodological rigor and practical constraints. By carefully defining the choice set, pre‑testing descriptions, ensuring a diverse and engaged sample, and planning analyses in advance, researchers can avoid the common pitfalls that undermine data quality. Incorporating thoughtful details—such as a well‑placed “none of these” option, strategic task ordering, and strong handling of missing data—further strengthens the validity of the findings. When these best practices are applied consistently, the resulting estimates not only reflect true consumer preferences but also provide a solid foundation for evidence‑based decision making in both academia and industry.
And yeah — that's actually more nuanced than it sounds.
