Ray Poynter, 4 August, 2023
When conducting qualitative analysis, we have a wide range of choices. Popular approaches include:
- Discourse Analysis
- Grounded Theory
- Narrative Analysis
- Framework Analysis
- Hermeneutic Analysis
When I tackle a project for non-trivial cases, I tend to choose one of these, balancing the strength of the technique with the time it takes. But, with all the fuss about Large Language Models, I wondered if I could leverage multiple techniques on a single project. Here is a simple experiment using ChatGPT4.
The Kings Cross/St Pancras Travelodge and Tripadvisor
To gather some data to work with, I collected a series of reviews of a budget hotel in central London. I have stayed at this hotel several times, and I have used this hotel several times before for analysis, which means I have some insight into what any new analysis is likely to show.
To conduct the analysis, I asked ChatGPT4 to analyse the same reviews using the five approaches listed above. I then reviewed the five explications to look at the similarities and differences, that is the consonances and dissonances.
Categorising Methods
Discourse Analysis, Grounded Theory and Framework Analysis found a similar set of findings. The location was the main plus, for example Discourse Analysis said “Guests consistently appreciate the hotel’s convenient location, particularly its proximity to King’s Cross station. This aspect is emphasized as a significant advantage for travelers seeking ease of access to transportation.”. Other positives included the friendliness of the staff and the relatively low cost.
The main negatives related to the quality and cleanliness of the rooms. For example, Discourse Analysis said “Opinions on room quality and cleanliness are mixed. While some reviewers express satisfaction with modern rooms and cleanliness, others report concerns regarding unclean corridors and floors. Basement rooms are mentioned as less desirable by some guests.”. Other negatives included the amenities, noise and other disturbances.
Storytelling Methods
Both Narrative Analysis and Hermeneutic Analysis produced outputs in a more storyline format.
For example, Narrative Analysis started with “A recurring plotline in the narratives centered around the hotel’s location and accessibility. Guests highlighted their convenience in accessing King’s Cross station and the surrounding areas, portraying this as a pivotal aspect of their experience.”.
And Hermeneutic Analysis started with “The reviews reflect an appreciation for the hotel’s location, highlighting a sense of place that is intertwined with the urban experience. The proximity to King’s Cross station is not merely seen as a matter of convenience but also connects guests to the vibrancy and allure of London’s bustling city life.”
Putting it all Together
Armed with all five outputs, I can then start my analysis. Given what I already know about this hotel and on the basis of these five explications, I would veer towards the storytelling methods. However, I would also promote the importance of affordability. The ChatGPT4 analyses mentions affordability, but it tends to assign affordability as a secondary or tertiary feature. This understating of the importance of price is typical of the way people in the UK write reviews.
The use of ChatGPT4 helped me look at the text in multiple ways to ensure that I do not fall into the trap of using just perspective. It helps me to do more analysis in less time, focusing on the points of consonance and dissonance.
So What?
In my opinion, in the short-term, one of the main impacts of LLMs on qual analysis will be to enable more and deeper analysis to be conducted in the available time. As in this example, the researcher can request multiple views of the data to help highlight patterns of consonance and dissonance. All qualitative analysis requires immersion, but the options presented by LLMs offer us the opportunity to change the point of immersion. Instead of immersing in the raw text, I can immerse myself in suggested explanations AND the text. The two reasons for doing this second approach are a) in my experience, it is faster (for non-trivial projects), and b) it might reduce the risk of my biases blinding me to something.