The Gell-Mann Amnesia effect is simple. You read something in your field. It is wrong. You notice. Then you read something outside your field. You trust it.
This same pattern now shows up in how developers use AI.
The trap of “it works”
When developers move outside their core expertise, how they evaluate code changes.
In unfamiliar domains, we stop testing for correctness and start testing for plausibility. If the build passes, we assume the logic is sound.
But “it runs” is a low bar. It hides:
- security debt
- fragile architecture
- silent edge-case failures
The single-narrator rule
AI feels more reliable in unfamiliar areas, but that confidence is misplaced.
The model does not get smarter when you switch domains. Your ability to audit it just got weaker. The error rate has not changed. Only your visibility.
Treat the AI as a single, fallible narrator. If it is a junior dev in your core language, it is a junior dev in your infra too.
The real risk
This is not just a developer problem. It affects how organisations adopt AI.
AI has not lowered the need for expertise. It has raised the cost of shallow understanding. The fastest way to build a fragile system is to trust output you are not qualified to verify.
Do not let “it works” become a proxy for “it is right”.
The IDE version of the newspaper
This shows up in everyday development workflows.
You prompt for code in a language you know well. The output has issues. You catch them fast.
Then you switch domains. Infra. CI. Regex. The output looks clean. It runs. You accept it.
Same model. Different scrutiny.
Scepticism is local
How much you question AI depends on what you know.
- High expertise → you test logic
- Low expertise → you trust output
In unfamiliar areas, “looks right” replaces “is right”. AI is good at passing that test.
“It works” is a low bar
A passing build is not proof of correctness.
A green check can still hide:
- security flaws
- poor design
- tech debt
- brittle assumptions
If the model was wrong where you could see it, assume similar risk where you cannot.
Where this bites
These blind spots appear in predictable places.
- Language jump: primary to secondary language
- Stack jump: app logic to infra or security
- Tooling jump: build scripts, pipelines, config
These areas feel harder to review, so they get trusted.
A simple rule
There is a practical way to reduce this risk.
Treat AI as one unreliable narrator.
- Ask it to explain “why”
- Push for edge cases
- Verify outside your head
If you cannot evaluate it, do not trust it.
The shift
This is the real change AI introduces.
You can now ship systems you do not fully understand. That is new.
The mistake is not that AI is wrong. It is that you forget it was.
If this resonates, we can help. We work with teams to apply AI in software development without increasing risk or technical debt. Get in touch to discuss how to use AI with the right level of oversight.