Latent Response Modelling

What comes back is a distribution, or a crosstab against specific demographics that existed within the survey you are querying. Comparable methods only return the most likely answer. Whilst they can look accurate on some measures, they collapse the distribution to a point mass, making them unsuitable for most use cases within survey data. We instead train our system to reproduce the shape of the distribution itself, measured as total variation distance (TVD) on held-out questions. Every estimate is returned with a confidence score for each answer option rather than one blanket figure for the question. This allows researchers to see which parts of an answer carry weight and which may not.

To evaluate the system, estimations from the model are scored against answers it could not have seen. We benchmark against the trivial baselines (uniform distributions) as well as the more difficult ones (always-return-the-modal-answer), and also assess the performance against a noise floor obtained by resampling the truth against itself.

We found that simply masking random questions was not enough. Instead we hold back semantically similar groups of questions rather than just single questions. Surveys tend to ask about the same underlying attitude several different ways: opinions on whether an ex-con should be allowed to teach, to speak publicly, to have a book in the library; these all correlate very strongly. Hold just one back and the model can almost read the answer off the questions it has already seen. By holding out the whole group at once, we get a truer test of answering something genuinely new. Additionally, when evaluating model performance we must also ensure that the survey data could not have entered the LLM's training data.

One of the striking properties of LRMs is that crosstabs are not an additional layer at inference, they are included by default. Once a population has been distilled, asking how women over 55 answered, or renters in the North East, is the same inference applied to a different slice.

A conventional crosstab is bounded by how many respondents happen to fall in the cell. If you slice a national sample two ways you often end up reading about only a handful of people. The survey world's answer to this is MRP (multi-level regression with stratification) which fits a model to the whole sample and uses it to estimate the small cells. An LRM makes a similar statistical move but with a much richer model of how answers move relative to each other across a population.

By anchoring every prediction to the population's real answers we can correct for the model's own priors (the biases introduced in training and RL). We can then measure how much prior bias survives our training. This is what separates LRM from “silicon sampling” methods built on synthetic personas. Our benchmark findings will follow in a white paper.

LRMs mean you can ask fewer questions of your panel, and they allow companies that spend a lot on research to get more value from existing assets. In addition, we think they will fundamentally change the questions that get asked of respondents. In the past surveys tended to be narrowly focused, which limits the novel questions that can be asked of the data using AI. In the future it will be advantageous to ask a set of questions which gives the optimal coverage for distilling information into the latent representation.