AI-moderation: amplifying human-led research
AI-moderation: amplifying human-led research
The rise of AI-moderated platforms has caused a lot of industry debate. To be sure, while they have great benefits they certainly have limitations.
The obvious benefits are the speed and scale. You can run multiple sessions simultaneously across the world in 24hrs and be ready to analyse the next day. That opens up a lot of possibilities. They shine when you have standardised questions and want a quick, consistent perspective: immediate sentiment reads, instantaneous product launch feedback, or simple concept reviews - essentially, anything where pattern recognition and theme collation is useful.
Conversely, they’re not effective for any deeper exploration or nuanced evaluation on their own. They’re particularly ineffective when understanding context is important to unpacking people’s decision-making or behaviours. AI moderation can’t use contextual inquiry or flexibly explore interconnected facets of a person’s life. Nor can it adapt the line of questioning creatively when the need arises. Most importantly, while I’ve come out of countless groups and interviews feeling inspired and full of ideas, I’m yet to feel that viewing an AI interview. They’re primarily fact finding machines so better to avoid when you’re in need of inspiration or a transformational insight.
However, what we’ve found most interesting about AI-interviewers is the ability to create new study workflows that amplify the impact of human-led sessions. AI interviews enable us to build in much more iteration into our studies, which in turn provides us more occasions to come together with client teams to progress our collective understanding and feed this into the in-home sessions or focus groups we run. Invariably, this leads to better outcomes.
To exemplify this, here are two examples that capture the types of studies we are increasingly running:
Example 1: Iterative global concept/idea development in five weeks.
A five week, three stage, concept development study across four international markets with:
Stage one: start with a wide set of concepts and evaluate these with 20-30 people across each market using AI moderation - recruited using a specialised panel. It takes a week to agree and develop the concepts and the weekend to complete 120 x 30 minute interviews.
Stage two: you can run a quick analysis to get a read of which concepts are working better, deliver that in a word report as a pre-read before running a refinement workshop to develop the best two ideas to take into further research.
Stage three: while the other two stages are happening, you can recruit 2-4x targeted online groups in each market to then evaluate in more depth the evolved ideas.
In this scenario, you not only get speed but you get better answers. Stakeholder teams are more actively engaged as they get to see results within a week and then develop ideas further ahead of the live research. We find this elevates engagement with the live groups too - something that has tragically been lost post Covid. This leads to a more collaborative process where teams are more unified and confident in the outcome as they have all been part of it.
Example 2: tracking product usage
A qualitative 6-month tracker of how people’s usage of digital tools evolve over time.
AI moderation lets us watch behaviour across a large group of people each month - the video/voice response engenders more system 1 responses that traditional communities just can’t tap into.
Alongside it, we run a smaller set of traditional interviews, pointed at the themes the larger-scale data throws up which are recruited freefind.
Clients can get monthly reports - rich with video - and then deep-dive on areas of interest through more nuanced sessions, which then lead to interactive client sessions.
Again, this dual approach is much richer and collaborative than what was achievable before. The AI interviews bring a systematic and trackable view overtime. The additional human interviews provide a deeper and nuanced exploration of the areas of real interest - building on the existing understanding. Again, this dual approach leads to more frequent collaborative interactions, which leads to more engaged teams, and ultimately, better decisions.
The outcome: the human work gets more pointed
The common theme across these and many of our studies is that when we can get a quick and thorough read through AI-moderation, live human engagements become better and even more impactful. In the same way as you might increase the context window when prompting your LLM, here we are increasing the team’s collective context window. With AI moderation, by the time we run focus groups or in-home visits, we have covered the breadth of baseline questions and have better hypotheses to work with. We’ve also had an opportunity to share ideas with clients and evolve thinking further without losing time. Thus, the human sessions don’t get bogged down by the long list of stakeholder questions in the brief, because they no longer have to cover the basics and thinking has already moved forward. Human interactions can focus on more pointed areas of exploration which gets you to the human understanding that really moves the needle.
In sum, we are proponents of AI interview and they have become an important part of our toolkit - but mainly because they make our human-led sessions better, our client interactions more impactful and, in turn, lead to more confident decision-making.
Five principles for using AI-interviewers
In the two years of using these tools, we’ve made some mistakes and have some principles around how to use them. Here are five of the most important ones:
Match the method to the question. Use AI moderation for breadth and speed or an initial read on standardised questions. Save the developmental and exploratory work for real, live engagements that give you more depth. Ideally, combine the two. To date, we have not run one study that only leverages AI moderation - it’s complementary and helps amplify other methods.
Good in, good out. The most important time you can take is ensuring what you put in is going to be effective.Think through the wording and the sequence yourself, ensure the stim is right for the medium, and use appropriate exercises that will get the most out. Good questions in, good data out.
Find good people. Ensure you are recruiting from great, specialist panels. Or run a free-find recruit to get exactly the right people - fraudulent participants are rife. And always use video so you can manually check the quality of response. Interestingly, AI interviews are shining a light on this issue: when you can see who is responding through your panel, you might be unpleasantly surprised by what you see.
Test and run pilots. Run sessions with your internal team before launching to ensure they flow, then run 2-3 pilot sessions in each market before launching your whole batch to ensure that the translations and market adaptations are effective and you’re getting what you need back.
Human analysis first. Spend time in the sessions yourself before looking at the platform’s output - the latter can surface good themes but also some poor ones. More importantly, it’s better to listen properly to some sessions to get a feel of what people are saying. What they say is not always what they mean and what they say in context of the wider interview is just as important to get an understanding of how and why people are responding to a particular question.
How MTM can help
Mixed-method study design. We choose the combination of AI-moderated research, AudienceGPT and traditional methods to fit your question in a bespoke manner..
AI-moderated research. We write the guides, run quality control on every session and analyse the results ourselves before using AI analysis to create a holistic perspective.
Workshops and client engagement. We run interim and final workshops, remote or in person, that help turn broad evidence into decisions.
Human-led qualitative research. Focus groups, interviews and home visits, building on wider inputs and understanding.
If any of this is live in your organisation, we’d welcome the conversation. Get in touch today.








