Understanding AI Search: A Longitudinal Prompt Method

Understanding AI Search: A Longitudinal Prompt MethodAuditing the Epistemologies of AI Search

I am pleased to share my latest piece on the LSE Impact of Social Sciences blog: “Before AI agents act for us, we need to know how AI searches for us.” 

As AI agents increasingly mediate our access to information, understanding their underlying “search epistemologies” becomes a critical research priority. In this article, co-authored with Giulia Tucci and Aanila Kishwar, we report on a longitudinal test of AI search engines—such as ChatGPT-4o—revealing that their workings are highly conservative.

One of the main findings is that citation diversity was not the default behaviour: it only appeared under sustained and explicit prompting that challenged the model’s credibility and pushed it beyond its usual source loops. The study also found that GPT-4o recycles a substantial share of sources across repeated prompt rounds over time and behaves, by default, as a rather conservative mediator of web knowledge. 

What feels especially relevant beyond the immediate AI discussion is the broader public question this raises: if more robust and diverse sourcing only appears when users know how to push the system over time, what does that mean for AI web search as a tool for everyday information retrieval? The piece argues that this points to the need for stronger observability of how AI search systems scan, rank, and package the web for us. 

Key Contributions & Methodological Innovations:

• The Longitudinal Prompt Method: I demonstrate a technique for “stress-testing” AI over time to expose the inconsistencies in how it ranks and justifies its sources.
• Observability Gap: The research highlights a significant discrepancy between AI’s promise of comprehensive knowledge and the reality of its opaque, restricted source integration. 
• Method-Driven Engineering: This empirical work serves as the foundational “proof of concept” for my upcoming tool, AIsearchTracer, which automates the capture of AI search traces for critically auditing them.