Chapter 8. Designing surveys with AI — questions that carry less bias

Key points

• A survey's design decides the quality of everything it returns
• Use binary, Likert, multiple-choice and free text according to purpose
• AI is a partner in writing items — but spotting a leading question is a human's job

The survey is the nearest way to collect data

Of all the ways to collect data, a survey is the easiest and the nearest to hand. Online, it is simple to distribute and collect, and it yields data that quantifies readily.

But because it is easy, there are many pitfalls. A weakly designed set of items can make the whole dataset useless. The quality of the items is the quality of the data.

Use the item formats appropriately

Each format has its strengths.

• Binary (yes/no) — simplest and easiest to answer, but captures no nuance or degree.
• Likert scale (five points and so on) — good for capturing satisfaction or evaluation in steps. The most-used format (an ordinal variable).
• Multiple choice — several options can be chosen, capturing several facets.
• Free text — reflects individual accounts and can serve as qualitative data, though aggregating it is laborious (which is where AI helps).

For early validation of a new service, a five-point purchase-intent scale is standard.

1. I would definitely buy it. 2. I would probably buy it. 3. Neither. 4. I would probably not buy it. 5. I would definitely not buy it.

This makes an intuitive reaction quantitatively visible and gives a sense of market viability while the idea is still an idea. Add a free-text "why did you want (or not want) to buy it?" and you capture quantity and quality in one pass.

The chief enemy: biased items

The trap beginners fall into most often is items that lead the answer.

• "Were you satisfied with this wonderful service?" → "wonderful" is leading
• "Do you think a lack of exercise is bad for health?" → the answer is obvious; there is no point asking
• "Which matters more, price or quality?" → not a question that can honestly be reduced to two options

Leading, ambiguous and falsely binary items distort data. A good item is neutral, concrete, and asks about one thing only.

AI as both designer and critic

AI plays two roles here.

One is designer: "write ten survey items for early validation of this service, combining Likert scales and free text" gets you a draft.

The other, easily overlooked but powerful, is critic. Hand your own items to the AI and ask:

"Do any of these items carry bias that leads the answer, or ambiguity? Point them out and propose improvements."

This is an application of the self-critique technique from Foundations. People find it hard to see the slant in questions they wrote themselves. AI's third-party eye polishes them.

The final judgement is still yours. Has the AI's improvement drifted away from what you actually want to know? That is for you to see.

A prompt to try (item bias checker)

You are a specialist in survey design. Below are draft items for a survey I wrote. Check each on three counts: (1) bias that leads the answer, (2) ambiguity or multiple meanings, (3) difficulty in answering. Where there is a problem, rewrite the item so it is neutral and clear. Also suggest which items should be Likert scales and which should be free text.

[Draft items] (paste your own survey items here)

Chapter 8 summary

• In a survey, item quality is data quality; match the format to the purpose
• The chief enemy is leading, ambiguous items; bias distorts the data
• AI is both designer and critic; the axis of what you want to know stays human