Why We're Getting AI Trust Completely Wrong

When the Sydney University gambling education funding scandal broke — where a A$20m funding request was submitted using AI-generated content riddled with hallucinated citations—LinkedIn feeds exploded with people declaring AI the enemy of truth, and research, and decision making.

This wasn't an AI failure, though. It was a human accountability failure. The problem was that someone hit 'send' on a A$20m funding request to government without doing their job.

The Double Standard We've Stopped Noticing

Here's what strikes me after 20+ years working in the pre-AI world: organisations that would never dream of sending unreviewed work from a junior staff member are treating AI-generated content as ready-to-publish.

You wouldn't let a graduate analyst submit a board proposal without deep reading, checking, updating, peer reviews, and rigorous oversight. You'd never skip the process of interrogating sources, verifying claims, and ensuring every statement could withstand scrutiny. Yet somehow, when AI produces the draft, all that discipline can evaporate.

The Sydney incident isn't an outlier. Research from NeurIPS 2025 analysed over 4,000 documents and found hundreds of AI-hallucinated citations that slipped past three or more reviewers. Nearly a quarter of these documents were approved despite containing fabrications no human would reasonably make.

Someone got lazy. Multiple someones, actually.

The AI Literacy Gap in Leadership

I work with organisations across sectors and I've witnessed something troubling: a growing gap between frontline AI adoption and executive AI literacy.

Many leaders spend enormous energy trying to police AI use. They're developing detection methods, creating new compliance frameworks, looking for 'tells' that signal AI involvement. Meanwhile, they're not spending that time understanding AI themselves—its power, its limitations, how it will transform their industry.

A senior executive recently told me that AI-generated work had obvious tells, and that any output using AI would be inferior. They're wrong on both counts.

The latest releases from Claude and ChatGPT represent step changes in capability. Research shows that people with lower AI literacy view AI as magical and are more likely to use it blindly. But those with higher AI literacy? They're directing AI like strong leaders.

There's your tell. It's just the opposite of what that executive thinks.

What Good AI Leadership Actually Looks Like

The best work I see involves people giving AI guidelines and guardrails, then checking and questioning and directing its work continuously. They're interrogating logic, verifying sources, pushing back on assumptions.

AI allows them to enhance and expand their thinking without getting bogged down in administrative mechanics or being disadvantaged by language or writing skills — obstacles that have held back or excluded many brilliant professionals.

The inferior work? That's people treating AI as an oracle. Accepting outputs as final. Abdicating their professional judgement to a tool.

Sound familiar? It should. It's exactly what happened in the Sydney incident.

The Authority Trap

I've seen this pattern across sectors. People are abdicating their professional judgement to AI outputs, accepting them as final rather than treating them as starting points that need human refinement.

Part of it is efficiency pressure. Organisations are under constant pressure to do more with less. AI promises speed, and speed looks like productivity.

Part of it is a fundamental misunderstanding of what AI actually is. We've positioned AI as either oracle or threat, when it's actually neither. It's a tool that produces draft work requiring the same critical review process you'd apply to any team contribution.

The psychology here matters. When you receive work from a junior team member, you instinctively know your role: review it, refine it, add your expertise, ensure it meets standards before it goes out under your name.

When AI produces the same work, something shifts. The technology carries an implicit authority. It feels finished. Polished. Confident.

Research from Vectara found that even the best AI models hallucinate at least 0.7% of the time, with some exceeding 25%. In legal contexts, average hallucination rates hit 6.4% for top models and 18.7% overall. In medical and healthcare contexts: 4.3% and 15.6%. In scientific research: 3.7% and 16.9%.

These aren't edge cases. These are core functions where accuracy matters most.

The Opposite Problem Is Just As Bad

Here's what worries me about some leadership responses: they're using incidents like Sydney as evidence that AI can't be trusted, that it will never replace 'real professionals. That's equally wrong.

Organisations paralysed by AI suspicion are falling behind. They're missing genuine efficiency gains whilst competitors move forward with proper oversight frameworks.

I see a world where AI will surpass many professionals in speed and depth. The tools are making step leaps with every release. Someone with deep subject knowledge and strong AI literacy will be able to produce better work, faster, with fewer barriers.

The choice isn't between embracing AI blindly or rejecting it entirely. The choice is whether you'll manage it properly.

What Proper Management Actually Means

OpenAI's research argues that language models hallucinate because standard training rewards guessing over acknowledging uncertainty. Models are trained to produce the most statistically likely answer, not to assess their own confidence.

You need to be that confidence check. Proper AI oversight looks like this:

Treat AI outputs as first drafts. Always. Even when they look polished. Especially when they look polished.

Apply the same review process you'd use with a capable junior team member. Deep reading. Source verification. Logic checking. Peer review where stakes are high.

Question confident-sounding statements. AI can sound supremely confident whilst being wrong. Confidence isn't accuracy.

Build verification into your workflow. Don't treat a final and thorough review as an optional extra. Make it mandatory before anything goes out under your name.

Develop your own AI literacy. You can't manage what you don't understand. Spend time working with these tools. Learn their strengths and limitations. Understand how they fail.

This isn't about inventing entirely new governance structures. It's about applying basic management principles you already know.

The Real Cost of Getting This Wrong

When reports serving as the foundation for major business decisions are machine-generated without rigorous verification, it undermines trust in the entire process.

In 2025, courts confronted AI-assisted filings polluted by fabricated citations. Judges responded with formal sanctions and a consistent message: if you submit machine-generated fiction under your name, it's your filing.

The same principle applies everywhere. The person who gives the final approval, or whose name is on the document, owns the content. AI doesn't absolve you of that responsibility.

The Sydney incident cost credibility. It cost trust. It potentially cost A$20m in funding that might have made a real difference.

All because someone forgot the basics.

The Path Forward

The best solutions are usually the simplest ones. You don't need to fear AI. You don't need to worship it either. You need to manage it.

The organisations and leaders that will succeed with AI are the ones that remember this fundamental truth: technology amplifies human capability, it doesn't replace human judgement.

AI can give you superpowers. It can help you work faster, think broader, overcome barriers that used to exclude talented people.

But only if someone stays in the driver's seat. Only if you do your job.

The Sydney University incident wasn't an AI failure. It was a human one. Someone — multiple someones — abandoned the rigorous oversight that high-stakes work demands.

Don't make the same mistake.

Treat AI like the capable but junior team member it is. Review its work. Question its logic. Verify its sources. Add your expertise.

Then, and only then, hit send.