Skip to content
AI Strategy|6 min read

The Layer Of Business Nobody Can Explain Anymore

Claire Byrne · 16 April 2026

Back to Insights
Share

There's a term circulating in engineering circles right now: dark code. Code running in production — at companies you use every day — that nobody can fully explain. Not the engineer who shipped it. Not the team that owns the service. Not the CTO. It works. It passes tests. But the comprehension step never happened. The AI built it, it passed the checks, and it shipped. Nobody sat down and understood it.

"Dark code is code that was never understood by anyone at any point in its lifecycle. It was generated, it passed automated checks, and it shipped — but the comprehension step never happened."

— Nate B Jones — Your codebase is full of code nobody understood, Substack, 12 April 2026

This isn't fringe. It's the fast-evolving reality of AI development that is going to affect all of us.

42%

of all new code written today is AI-generated or AI-assisted — expected to reach 65% by 2027

Sonar, State of Code Developer Survey, January 2026

43%

of AI-generated code changes required manual debugging in production, even after passing QA and staging

Lightrun, State of AI-Powered Engineering Report, April 2026

And this is what it looks like when it breaks:

Incident

In July 2025, an AI coding assistant modified SaaStr's production code despite explicit instructions not to — then deleted the production database during a code freeze. It fabricated 4,000 fake users, generated false reports, and masked failing unit tests. Replit's own CEO responded publicly to the incident on X.

CIO.com, 5 famous analytics and AI disasters, 16 December 2025

Your business already has dark code. It's just not called that yet.

You might think: that's a tech startup problem. My business doesn't write code. But the same dynamic — decisions and processes that run your business that nobody can fully reconstruct, trace, or own — is already present in most organisations. AI is about to accelerate it dramatically.

You probably already have some version of this:

  • A pricing model that grew over ten years and only two people can correctly interpret
  • A quoting process so complex it takes hours and lives entirely in the heads of the people who built it
  • A legacy system customised so heavily that the original logic is effectively undocumented
  • Customer-facing decisions — credit terms, stock allocation, delivery priority — made by experienced people in ways that can't be easily transferred

That's not dark code. That's dark process. It carries exactly the same risk: nobody can explain it, nobody can reconstruct it, and if the people who hold it leave, it breaks.

Now add AI. Companies are beginning to automate pricing decisions, inventory allocation, customer communication, demand forecasting. The logic accelerates. Output increases. And the comprehension gap — who actually understands why this decision was made — widens faster than anyone is tracking.

The International AI Safety Report 2026 documented 233 AI safety incidents in 2024 alone across non-software domains including manufacturing, energy, and logistics.

The real problem: human-in-the-loop patterns aren't scaling

We are running organisations using patterns designed for human speed — and the speed is no longer human. The loop most organisations operate on — someone decides, someone builds, someone reviews, someone approves — was designed for a world where human capacity was the constraint. That constraint is disappearing faster than anyone predicted. And the organisational patterns are starting to crack under the weight of it.

"Engineering organisations need runtime visibility to embrace the possibilities offered by AI-accelerated engineering. Without this grounding, we aren't slowed by writing code anymore, but by our inability to trust it."

— Ilan Peleg, CEO, Lightrun — State of AI-Powered Engineering Report, April 2026

96% of developers don't fully trust AI-generated code — while 84% use it daily. Volume outpaces comprehension. Liability accumulates quietly.

Two groups are forming

The first group is deliberately redesigning where humans sit in the loop — not removing humans, but reconfiguring where human judgement is irreplaceable and where it isn't. They're asking hard questions about which decisions require a person and which can be handed off — and building systems that make that handoff legible, traceable, and recoverable when something goes wrong.

The second group is doing what they've always done, just faster. More output, same patterns, same approval chains. Volume goes up. Comprehension goes down. Key-person risk — which already keeps most operations leaders awake — compounds. The liability accumulates quietly, until it doesn't.

"That loop — mandate the AI, fire the humans, discover you still need the humans, realise they're gone — is not unique to Amazon."

— Nate B Jones — Your codebase is full of code nobody understood, Substack, 12 April 2026

The cost of staying in the second group isn't just operational risk. It's competitive position. Companies in the first group are exploiting margin differences in real time, responding to customers in moments, optimising production in ways that compound over time. Whatever the competitive edge in your industry — AI will unlock it for the prepared first.

The exercise worth doing this week

The useful question isn't "what are we doing with AI right now." It's: what are the problems in your business that have always felt unsolvable? The ones that require too much data, too many variables, too much coordination to crack with the tools you've had. Forecasting that takes too long to be useful. Quoting that's too slow to be competitive. Customer communication that depends on someone remembering to check.

Then ask the harder question: which of those processes could only be reconstructed by one or two people if they left tomorrow?

That's your dark process inventory. That's where the risk sits right now — and where the capability unlock is if you get in front of it intentionally, rather than let AI accumulate it for you quietly.

Where to start

You don't need to solve this in a week. But you do need to start understanding where your organisation's comprehension gaps are — before AI accelerates them past the point where they're manageable.

For the technical people in the room, Nate B Jones has packaged a free toolkit: a Dark Code Audit, a Context Layer Generator, and a Comprehension Gate for reviewing AI-generated changes before they reach production. Prompt kit here — and the full write-up here.

For leaders: start with a whiteboard and your leadership team. Map the decisions in your business that currently live in people's heads. Ask which ones AI could own — and which ones need to stay with a human who can be held accountable. That conversation is what positions you in the first group, before the gap gets any wider.


Resources

AI StrategyRiskOperationsLeadership
CB

Claire Byrne

Founder & Co-Chair of Elements AI. 25 years in product and technology. Now helping mid-market teams build real AI capability.

Related Insights

© 2026 Elements AI

Book Intro Call