ChatGPT vs QInsights: Testing AI for Analysing Open-Ended Survey Data

Dear Reader,

I was a bit struggling with how I should start this newsletter telling you about experimenting with ChatGPT for coding semi-structured qualitative data. Those who follow me for a while know that my perspective is that we no longer need to code data.

So why on earth should I figure out whether this is possible with a general-purpose chatbot? On top, I am the co-founder of QInsights — should I not look for a thousand ways to lure you away from using general-purpose chatbots?

Well, my academic curiosity got the better of me. Fact is, it is not possible to use a general-purpose chatbot for coding interviews or other long-form data without chunking the data first and using additional tools. However, as semi-structured data like open-ends are already “chopped” into small pieces and thus the coding unit is clear, I thought it might be possible that a chatbot can add one or more fitting codes to each response.

Setting Up the Experiment

For testing, I used an Excel file containing answers to two open-ended questions from 150 respondents.

Before I could begin applying codes, I first needed to develop a coding frame. After some back and forth and teaching the chatbot what you need to pay attention to when building a code system, this I could call a success.

In case you are curious what those aspects of qualitative coding were that the chatbot did not consider by itself: some categories violated the principle of mutual exclusivity, with conceptually distinct ideas grouped under a single heading. For instance, “Work Content” was combined with “Professional Meaning,” although these represent different conceptual dimensions. Another issue was including both positive and negative aspects in the code definition. For example, under “Interdepartmental Communication,” the LLM suggested the definition “Clarity or lack thereof when working with other units.” The purpose of coding is to later retrieve the segments by codes. You do not want a mixed bag of things collected under one code. Thus, you need two or more codes for interdepartmental communication.

To evaluate the validity of the suggested coding frame, I first coded parts of the data myself. Once the final version of the coding frame was ready, I manually applied the codes again to a subset of the data to test whether it worked as intended — with the hope of automating the rest.

Where General AI Fell Short

And here not only GPT-4o and o3 failed, but also Gemini 2.5 could not handle coding 2 open-ends with 150 responses each. The first 20 responses were coded without gaps, and then more and more responses were skipped with larger and larger gaps towards the end of the file. In addition, every model coded the data differently — which can be expected. With 3 humans coding the data, it would be the same. But in the end, it means you, as the human researcher, need to check everything and fill in the gaps!

Getting to this point in the first place (creating the coding frame and then applying the codes) took over a day and along night of trial and error. One could say that if this had resulted in a usable template, the effort would be worthwhile, as you could reuse the prompts for the next analysis. However, given that it struggled with 300 responses already and skipped about 30% of the cells it was supposed to code, plus the need to double-check everything…

I started this exploration in a true researcher mindset with the following null hypothesis:

H₀: Using a general-purpose chatbot (e.g., ChatGPT/Gemini) with a single prompt and no human intervention during the run, the model returns an output that codes all analysis units with at least one qualitative code per row or a pre-defined “uncodable” reason.

H₁: General-purpose chatbots cannot be used for coding semi-structured data.

The result of this experiment was that I had to reject H₀ and accept H₁.

Enter QInsights

Being somewhat frustrated seeing ChatGPT struggling with just 150 responses, I thought — ok, it is time now to see what results QInsights is coming up with.

Maybe this detour was what I needed to truly appreciate the tool we’ve built. While QInsights embodies my vision for AI-augmented qualitative analysis, I’m also its toughest critic — always seeing what could be improved and what more we could implement.

In this case, QInsights generated eight themes that aligned perfectly with the categories from the coding frame, plus two valuable additions that had been overlooked.

QInsights can also extract all data segments for each theme when analysing open-ends, allowing you to export the results as an Excel table — one sheet per theme, with all relevant data segments included.

And that’s just the start. You can take any theme, ask follow-up questions, dig deeper, and compare or contrast responses based on socio-demographics or other variables from the Excel file.

That’s the difference when a tool is purpose-built for qualitative analysis and designed around dialogic, traceable workflows — not just tagging.

Why this matters for you

• No prompt engineering required: Start analysis immediately; the workflow is already set up for qualitative rigor.

• From coding to conversation: Move past surface labels to thematic exploration, role comparisons, and abductive “why” questions — with an audit trail back to the raw text.

• Consistency you can trust: Designed by qualitative researchers for qualitative work.

So… what are you waiting for?

Some of you have already tested QInsights. Others have registered but haven’t taken it for a spin yet. Some are already enjoying the benefits, and others have seen it in action during a webinar.

Don’t waste your time trying to bend ChatGPT or Gemini into doing a job they’re not built for. With QInsights, you can start analysing in minutes. Users tell me it’s intuitive, and if you do need help, there are clear video tutorials, a detailed manual, and — of course — we’re just an email away.

👉 Log in or register now for a free trial and let QInsights work for you and with you.

Warm regards,

Susanne

Dr. Susanne Friese
Rebel Methodologist
Founder of Qeludra and Co-Founder of QInsights