A new Law of AI is emerging - Gell-Mann Amnesia!

 I noticed this throwaway comment recently:

"Whenever I use AI to probe a topic I know something about, it seems to make a number of errors; by contrast, whenever I use it to explore topics I know little about, it knows so much more!"

I must admit I have noticed something similar. When I am asking about something I know about, such as software, AI generated responses tend to be at best flawed, at worst plain wrong. Yet when I ask about something I genuinely know nothing about the results are compelling and I have to rein myself in.

It turns out this is a named effect, observed by Michael Crichton (yes, that Michael Crichton!). "Gell-Mann Amnesia". He noticed that experts reading articles in their field of expertise will tend to find them full of errors, whereas reading other articles, even in the same publication, on topics outside their expertise  are believed to be credible.

That is exactly what is happening with AI. When the results are critiqued by an expert they tend to be average at best. But for a non-expert (even one that has previously debunked AI-generated data in their own field), they have an almost magical ability to convince.

I suspect this is also why LLM-assisted coding is met with lukewarm enthusiasm at best by many XP practitioners. They have lived fast feedback cycles, automation, testing and so on many times before so it is nothing more than business as usual. But to those who have never leveraged testing, never seen the power of automation, never seen how tight feedback loops can improve delivery of value, the LLM becomes this magical tool that somehow solves all their problems. But because of its fallibility, it also introduces multiple traps that will catch the unwary later.

In summary - do not trust the LLM to be reliable. Check and double check what it is suggesting against expert (human!) opinion. Gell-Mann Amnesia is a powerful effect, and can easily catch the unwary.

===

Want to know more about becoming a more knowledgeable software developer who is less likely to fall for the pitfalls of AI-assisted coding? Do you have teams that need help delivering value? Get in touch and let's have a chat. I can help.