Latent Response Modelling

What comes back is a distribution, or a crosstab against specific demographics. Compare this to comparable methods which simply return the most likely answer. Whilst they can look accurate on some measures, they collapse the distribution to a point mass, making them unsuitable for most use cases within survey data. We instead train our system to reproduce the shape of the distribution itself, measured as total variation distance (TVD) on held-out questions. Every estimate is returned with a confidence interval around it, and the interval is per answer option rather than one blanket figure for the question. This allows you to see which parts of an answer carry weight and which may not.

To evaluate the system, estimations from the model are scored against answers it could not have seen. We benchmark against the trivial baselines (uniform distributions) as well as the more difficult ones (always-return-the-modal-answer), and also assess the performance against a noise floor obtained by resampling the truth against itself, which bounds how much of any gap is sampling variation.

We have also made that test harder on ourselves. When evaluating the performance of LRMs we ensure that the survey data could not have entered the LLM's training data. Additionally we hold back groups of questions rather than just single questions. Surveys tend to ask about the same underlying attitude several different ways: whether an atheist should be allowed to teach, to speak publicly, to have a book in the library. Hold just one of those back and train on the rest, and a model can very nearly read the answer off the questions it has already seen. By holding out the whole group at once, we get a truer test of answering something genuinely new.

One of the striking properties of LRMs is that crosstabs are not a separate exercise: once a population has been distilled, asking how women over 55 answered, or renters in the North East, is the same inference applied to a different slice. A conventional crosstab is bounded by how many sampled respondents happen to fall in the cell. Slice a national sample three ways and you are often reading a handful of people. An LRM estimate instead draws on the whole sample through the latent representation, and because it is a model output rather than a raw count, its reliability is measured, not assumed: subgroup estimates are certified against held-out data, anchored where needed, and withheld where the model cannot support them.

Anchoring every prediction to the population's real answers is designed to correct for the model's own priors rather than let them stand in for the survey population. We can then measure how much prior bias survives our training. This is what separates LRM from “silicon sampling” methods built on synthetic personas, and from synthetic-record approaches that add respondent rows without answering novel questions. Benchmark findings and the underlying methodology will follow in a white paper.

LRMs mean you can ask fewer questions of your panel, and they allow companies that spend a lot on research to get more value from existing assets. In addition, we think they will fundamentally change the questions that get asked of respondents. In the past surveys tended to be narrowly focused, which limits the novel questions that can be asked of the data using AI. In the future it will be advantageous to ask a set of questions which gives the optimal coverage for distilling information into the latent representation.