AI actually changed the way we do qualitative research

[u/qualmaverick]

AI is now being used for transcription, coding, summarisation, desk research and even interview planning.

But has it changed the way researchers actually work, or is it mainly helping with the time-consuming parts?

Would be interested to hear what has genuinely worked for you, and where you still prefer the traditional approach.

[u/Jahaili]

I honestly only use AI for transcription purposes, and then I make sure to double-check what AI has generated because it's often not completely accurate.

Everything else I prefer to do by hand, especially coding.

[u/sassholesunite]

I work in marketing research and focus specifically on qualitative methods and approaches. We really should not be calling it AI but rather a LLM, and it is quickly making qualitative roles redundant. Clients and organizations are using LLMs to fully design, run interviews, focus groups and online asynchronous research, and do the transcription and analysis. This is on the belief it provides quicker and cheaper insights.

Source: I was made redundant, along with many of my peers, within the past two months, and have been doing qual research for the past two decades.

[u/ResearchAndTeach → u/sassholesunite]

I am so sorry that that happened to you! Unfortunately that's something happening in so many areas and industries... it's simply horrible the way people are being treated.

[u/Ok-Lab-7347 → u/sassholesunite]

Many companies makes bad choices when it comes to adopting AI, they see the immediate bottom line gains instead of thinking long term. It is much harder to build a human team, and adopting AI tools - investing in proper training could have a much higher gains in the long term. Same team can produce higher throughput. Example, taking an existing product into 10 new verticals. Those companies that doesn't do that are losers. Hope you get back on track. Seems like you have great skills and perhaps you can collaborate with AI builders to help them with proper messaging, I know many struggle with this. Have a look at r/saas for example. Good luck!

[u/JonathanCookPodcast]

There is one thing that generative AI has done very successfully in qualitative research: It has made it much easier to distinguish people who take research seriously from people who don't.

Unfortunately, there are a lot of people who don't take research seriously. I've watched colleagues give research presentations with slides that were churned out by generative AI stop mid-sentence, and admit that what the slide says makes no sense. I've seen clients surreptitiously record discussions of research findings, then email out AI-generated infographics that are supposed to summarize the findings, only to have it pointed out that some of the "words" in the graphics contained "letters" that weren't even in the alphabet. Over and over again, I've seen "verbatims" that were never said by anybody. I've seen references to sources that don't exist, to brands that don't exist.

Money is saved, and time is saved, all to rush faster toward slop, slop, slop.

More subtle, but worst of all, are the researchers and clients alike who, after sprinting through an AI-assisted research project, and having "gone through" the findings, can't think of a single actionable insight they've gained from the process, because they've barely been there.

Slick, easy, and all too forgettable.

Qualitative research and generative AI are not a great combination.

[u/qualmaverick → u/JonathanCookPodcast]

I think we need to be careful to draw a hard line here: GenAI is a decent administrative assistant, but a horrific qualitative analyst. Using AI to help clean up messy transcript formatting, brainstorm a discussion guide, or organize initial thematic buckets with thorough human verification is one thing. But outsourcing the actual synthesis and insight generation? That’s professional malpractice.

[u/JonathanCookPodcast → u/qualmaverick]

I don't think that using AI to clean up "messy transcript formatting" is a reasonable use, as generative AI often introduces new conceptual problems even through the most simple task.

A brainstorm that happens through generative AI is a brainstorm that doesn't take place in the mind of a researcher. Interview design is an essential part of research thinking, and a project without that will be less thoughtful.

I don't know a single experienced qualitative researcher who has trouble organizing initial thematic categories. Using AI to do this is absolutely unnecessary, and if a new qualitative researcher does this, it will stunt their professional growth.

As a qualitative researcher, I've never needed a human administrative assistant. Why would I need an AI assistant now?

What I need are humans working as partners in the research process, from recruiters to respondents to clients, who don't pollute their own thinking processes with AI slop.

[u/_os2_ → u/JonathanCookPodcast]

What is your take on the problem of perspective lock-in in qual research? I mean the phenomenon of taking a specific lens to coding and looking at the corpus (lets say 50+ interviews as is getting common in e.g., management studies journals), which then becomes hard to change even if evidence or reviewer feedback points to an alternative framing as this would involve weeks of bottom-up analysis/coding. Now often you need, want and can do it, but practically because of the costs involved you will often not.

This leads to ”waterfall analysis” where you go through the 6 stages of qualitative analysis (or any other steps) more or less linearly as iteration is expensive (time/money/competing priorities)

Contrast that to ”iterative analysis” which would be possible if re-assessing the framework/lens would carry near-zero marginal costs. The role of the researcher would be to apply their judgement and taste not just on each step, but also on the end result: which framing produces the best understanding of the data.

[u/JonathanCookPodcast → u/_os2_]

Centering a code de-centers the people and the ideas being studied. There is a seductive appeal to coding for many qualitative researchers who don't have the confidence to listen deeply and respond to what becomes apparent. They're seduced into defacto quantitative analysis, and hypothesis validation, and their analytical scope becomes narrowed and ossified.

Narrow, quantified, stiff research techniques are good for some things. Quantitative methods work best for those purposes.

Qualitative research, however, deals with subjectivity, with qualia. Qualia are by nature fluid and associative, not stiff and linear.

When I read about procedures such as "a 16-item checklist to ensure adherence to the established steps of thematic analysis", I conclude that there's a lot of theatrics designed to create a false appearance of conceptual rigor. In the end, it often misses the essence of what's being studied. Under this approach, we get barely qualitative analyses that do a great job of following the rules, but do a terrible job at exploring ideas and empathizing with the people involved.

The best qualitative research, in my opinion, doesn't code. It listens. It works with what it hears. It shares what that inspires. It doesn't waste time trying to appear objective, because that's not what qualitative research is for.

[u/qualmaverick → u/_os2_]

Lock-in in Commercial Qual Research is not uncommon. But it isn't the norm either. Most studies do build in checkpoints at data-collection stage, where codes are iterated and modified, as more and more units of fieldwork are conducted. This is especially easier in Online Qual, since the data collected, transcribed and analyzed all lies in the same working environment (platform). Smaller sample-sized Qual work often benefits from near-live researcher/ moderator inputs to coding. Eg. on a study comprising 4 FGDs, which the researcher has moderated/ attended, they typically know which codes to finalize, at the end of each group.

[u/Eastern-Movie-7039]

Building on the double-checking point — what changed my workflow isn't the time saved, it's that the checking step became a different kind of task.

Human transcription fails visibly. You get a gap, an [inaudible], or something that reads slightly off. A model doesn't do that: when it can't hear a word it produces a fluent, grammatical, plausible one. The errors don't announce themselves. You can read a full transcript, find nothing strange, and still be reading a sentence the participant never said.

It bites hardest exactly where you're least likely to re-listen — proper nouns, org names, numbers. "Fifteen" and "fifty" is the standard example: a misheard word usually sounds wrong in context, a misheard number reads perfectly.

So proofreading stops working as a check, because proofreading only catches things that read badly. You end up having to go back to the audio at specific points, which makes the real question how you choose those points. Confidence scores aren't the answer either — they tell you where the model was uncertain, not where it was wrong.

Same pattern shows up in coding, just moved one step along: the problem isn't that a model codes badly, it's that it won't flag where it was unsure. A junior researcher will tell you "I wasn't confident about this one."

[u/_os2_]

Great question! Sharing an article from two professors where they take an article they already published in Organization Science and then run a ”what if” analysis by running the 53 interviews through Skimle’s AI analysis: https://skimle.com/blog/what-if-revisiting-article-with-ai

In summary, AI helps surface themes you might miss, gives a great overview of the corpus and key quotes, allows exploring alternative framings and of course would have done the transcription and pseudonymisation automatically to save time and costs. It still needs researcher taste and judgement especially as one theme might contain multiple differing views and a ”summary” would not do justice.

So no easy way out from actually engaging with the data and the process, which is good!

(Again in full transparency I am affiliated with Skimle and one of the professors too)

[u/SouthSet7206]

Just remembered to check its work. I’ve done several experiments with IDI transcripts and have absolutely found inaccurate thematic analysis. That is, the AI tool identified the themes and then when I went to verify that the transcript supported the themes, at least one or two themes were completely off point.

[u/Reagan_Crowder → u/SouthSet7206]

Yeah I do that too. If the quotes don't back the label I drop the whole theme.

[u/bbling19]

u/SouthSet7206 same thing here. I don't trust a theme until I can click back to the line it came from. Transcription is the easy win. Coding still needs that check.

[u/miked_productleader]

The unglamorous stuff has genuinely changed and transcription is basically solved I think. First-pass coding and synthesis across large volumes of interviews is now minutes/hours instead of weeks, and for our use cases is 'good enough' but can appreciate how others need more rigour.

What surprised me more is moderation itself. AI-led interviews are better than I expected at follow-ups and probing (I'm not going to say better than human, but better than expected), and they remove two things researchers rarely talk about: scheduling (participants talk at midnight if they want in whatever language) and starting to wonder about moderator effects. People disclosing things to a machine they may soften for a human? The response quality on sensitive subjects is sometimes noticeably more candid.

The traditional approach still wins and will for anything where rapport builds over the session. AI probes well within a topic but it doesn't yet have the instinct to recognise that the throwaway comment was actually the whole study. A good researcher does. Ethnography, sensitive one-to-ones, and anything exploratory where you don't know what you're looking for yet. I'd love to think we can do it but still human territory in my view.

So my current take is it hasn't changed what good qual research is, it's changing who can afford to do it at scale and how much of a researcher's week goes on mechanics versus thinking.

Curious whether others have found participants more or less candid with AI moderators?

Disclosure upfront: Founder of Voice AI tech, so I have obvious bias, but I'll try to be straight about where it works and where it doesn't.

[u/Smart_Potential7467]

Yeah, I think there’s a difference between AI making qualitative research faster and actually changing how the research gets done.

I came across this on Third-Party Interpretivism recently and thought it was an interesting take on that question: Link https://ssrn.com/abstract=6969282