Guardrails Made Us Faster

The Reasonable Question

March 2024. A developer on my team asked a reasonable question: "Can I use ChatGPT to help debug this production dashboard script?" The script was straightforward: internal metrics visualisation. The answer seemed obvious: sure, why not? Then I paused. What were our rules on this? Did we have any?

I asked the wider team what AI tools people were using, and for what. The answers were eye-opening. Some developers were pasting internal configuration files into AI assistants. Others were sharing database schemas to get query optimisation help. One person had asked an AI to review authentication logic by sharing code snippets. Everyone was trying to work faster. Nobody thought they were doing anything risky, because we had never told them what the risks were.

It was not a failure of character. It was a failure of governance. We had no policy, no training, no approved tools list, no guidance on what could or could not be shared. We had assumed that smart people would make smart decisions, forgetting that smart requires context we had never provided.

That conversation triggered a six-month effort to build an AI governance framework from scratch. This is what we built, what most policies miss, and why the guardrails made us faster.

The New Liability

The landscape has shifted. 77% of organisations are actively developing AI governance programmes, and 47% rank it among their top five strategic priorities.¹ The EU AI Act entered into force in August 2024; its prohibitions and AI literacy obligations have applied since February 2025, and non-compliance carries fines of up to €35 million or 7% of global annual turnover.

Compliance is not the sharpest pressure, though. Incidents are. Air Canada was held liable for misinformation its customer chatbot gave a passenger. Lawyers have been sanctioned in multiple jurisdictions for citing AI-fabricated case law. McDonald's withdrew its AI drive-thru ordering system after viral videos showed it adding 260 McNuggets to a single order. Documented AI incidents rose 56.4% year on year²: we are deploying AI systems faster than we are learning to use them safely.

Governance is the insurance against joining that list.

Principles Under Pressure

Every AI governance framework needs anchoring principles. Not aspirational statements for an annual report: decision-making heuristics people can actually apply when facing an ambiguous situation at 3 AM.

Value-Driven Use

AI must deliver measurable benefits aligned with organisational goals. This sounds obvious until you count how many AI deployments exist because everyone else is doing it. We established a simple test: can you articulate the specific problem being solved and quantify the expected improvement? If not, the project does not proceed. The test eliminated roughly 40% of proposed AI initiatives, most of which would have created technical debt without corresponding value.

Human Oversight

Humans retain final responsibility for all decisions; AI supports judgement, it does not replace it. The principle is now codified in the NIST AI Risk Management Framework and in the EU AI Act's requirements for high-risk systems. The catch is that oversight must be meaningful, not rubber-stamping. We learned this when a team member 'reviewed' 200 AI-generated test cases in under three minutes. Review requires allocated time, defined criteria, and accountability for the outcome.

Critical Evaluation

All AI outputs are drafts requiring validation. Even the best models still fabricate in roughly 7 of every 1,000 responses,³ and the rates climb steeply with task difficulty: benchmarked AI legal research tools have produced fabricated or mis-cited authority on 17% to 34% of queries.⁴ The working assumption must be that any AI output could be wrong.

The Stop-and-Ask Rule

If it is sensitive, high-risk, or unclear, stop and contact the AI Governance Lead before proceeding. This single rule prevents more incidents than any technical control. It acknowledges that edge cases will always exist and provides a human escalation path rather than forcing people to guess.

The Code Generation Problem

Software engineering presents the hardest governance case because code is simultaneously the highest-value use of AI and the highest-risk attack surface. GitHub's 2024 developer survey found 97% of developers using AI tools, many of them on code that reaches production.

Across an evaluation of over 100 large language models on 80 coding tasks, AI-generated code introduced security vulnerabilities in 45% of all test cases.⁵ Java exceeded a 70% failure rate. Cross-site scripting appeared 86% of the time; log injection, 88%. These are not edge cases: they are the norm.

AI code-generation risk sorts into three categories:⁶ models generating insecure code, models being vulnerable to attack and manipulation, and downstream impacts such as feedback loops in training future systems. Each needs different mitigations.

What AI-Assisted Code Can Do

AI accelerates engineering in specific, bounded ways: scaffolding and boilerplate, functions within well-defined scope, unit tests and documentation, exploratory snippets to evaluate libraries, lint fixes and complexity reductions, upgrade diffs and changelog entries. The common thread: these are starting points, not final products.

Mandatory Controls

Every piece of AI-generated code passes specific controls before it can touch production. Human review by an accountable engineer is non-negotiable. Licensing and provenance checks catch the code with undisclosed origins that models often suggest. Testing commensurate with risk, green CI and coverage thresholds, catches the silent bugs that look correct but behave insecurely. Environment segregation means AI code is generated and trialled in branches or sandboxes, never committed directly to protected branches.

Traceability matters as much as review. We mark every PR with an 'AI-assisted' tag linked to review and security scan artefacts: not to assign blame, but to know our codebase's composition and audit it when something goes wrong. An identified maintainer owns the long-term support of any AI-introduced component, because code without an owner is code without accountability.

Where code touches regulated or safety-critical systems, additional senior sign-off is required. This sounds burdensome until you consider the alternative: discovering during an audit that critical software was AI-generated with no documented review.

The New Threat Landscape

Traditional security models assume clear boundaries between trusted and untrusted inputs. AI systems collapse those boundaries. OWASP's 2025 Top 10 for LLM Applications ranks prompt injection as the number one vulnerability class in production AI systems.

Prompt Injection

Prompt injection manipulates the instructions that guide AI behaviour, turning helpful assistants into unwitting accomplices in data exfiltration. OpenAI has been candid about the challenge, writing in December 2025 that prompt injection, much like scams and social engineering on the web, "is unlikely to ever be fully solved." Most organisations still run no dedicated defence against it.

Indirect Prompt Injection

More dangerous still is indirect prompt injection: malicious instructions embedded in the documents, emails, and web pages an AI system processes. Lakera documented zero-click attacks in AI-powered IDEs where a seemingly harmless Google Docs file triggered an agent to fetch attacker-authored instructions, execute Python payloads, and harvest secrets, without any user interaction. CVE-2025-59944 showed a case-sensitivity bug letting attackers steer Cursor's agentic behaviour through configuration file manipulation.

The governance implications are short enough to memorise: never execute prompts from unverified or external sources, run AI tools with least-privilege access, never share credentials for AI platforms. This is baseline hygiene for an attack surface that did not exist three years ago.

The Default Assumption

The single most important data handling principle is the default assumption: all external AI services may log data unless explicitly confirmed otherwise. Many providers retain inputs for model improvement unless users opt out, and terms of service change faster than policies do.

We maintain a restricted data list that may not be processed in unapproved AI tools under any circumstances: customer lists, source code, financial projections, employee PII, health information, proprietary algorithms, contractual documents. The list is conservative by design. It is far easier to add exceptions for approved use cases than to recover from a breach.

Data discipline matters because everything that leaves your control becomes raw material for the other side. Phishing is now machine-written: personalised at scale, fluent in your company's idiom, and cheap. Every leaked schema, org chart or configuration file makes the next lure more convincing.

The Legal Limbo

AI-generated outputs exist in legal limbo. In March 2025, the D.C. Circuit affirmed that works created solely by AI, without meaningful human input, are not eligible for copyright protection in the United States. The U.S. Copyright Office's May 2025 report concluded that training on copyrighted material may constitute prima facie infringement, and that 'transformative' arguments are not inherently valid defences.

Debevoise & Plimpton is tracking more than 50 lawsuits between intellectual property owners and AI developers in U.S. federal courts. Thomson Reuters v. ROSS Intelligence established that using copyrighted headnotes to train an AI legal research tool did not qualify as fair use. Companies with significant IP assets are litigating rather than waiting for clarity.

For practitioners the rule is simple: assume AI output may not be copyrightable and may contain someone else's copyrighted material. Significant AI-assisted work must be reviewed and modified for originality; legal counsel reviews anything destined for high-value IP, branding, or patentable work. Disney's licensing agreement with OpenAI, allowing the use of 200 characters in Sora, hints that licensing rather than litigation may define the endgame. That future has not arrived yet.

The Tightening Framework

The regulatory environment is crystallising. The EU AI Act sorts systems into risk tiers: unacceptable (banned), high (strict requirements), limited (transparency obligations), and minimal (unregulated). Since February 2025, prohibited systems, including subliminal manipulation, exploitation of vulnerabilities, and real-time biometric identification for law enforcement, are banned outright. High-risk requirements apply from August 2026.

Critically, the Act carries AI literacy requirements for every organisation using AI, not just those building it. Staff must understand the systems they use, their limitations, and their risks. The fines are designed to be felt: up to €35 million or 7% of global annual turnover for banned systems.

The NIST AI Risk Management Framework, extended in 2024–2025 with a Generative AI Profile, remains voluntary but is referenced so widely by regulators and industry bodies that it functions as a de facto requirement. Its four functions (Govern, Map, Measure, Manage) slot cleanly alongside EU AI Act compliance.

The Training Dividend

Mandatory training before first AI tool use is non-negotiable. Our curriculum covers the policy itself, prompt engineering, identifying bias and hallucinations, and data handling procedures, including the Approved AI Tools Matrix. Annual refreshers keep pace as tools and threats evolve. Under the EU AI Act's literacy obligations, this is now compliance, not just prudence.

The quieter payoff is cultural. Shadow AI thrives when official channels seem inaccessible: employees do not stop using AI when you ignore it, they stop telling you. Training materials in accessible formats, an approved-tools list that actually gets updated, and a fast escalation path make the sanctioned route the easy route. Making compliance easy is the most effective compliance strategy we have found.

When Things Go Wrong

Incidents will happen. The question is whether you discover them through your monitoring or through a journalist's phone call. Our incident response triggers on any suspected or confirmed AI-related incident: data exposure, biased outputs, system failures.

The timeline is aggressive by design. Immediate communication to the line manager. Logged and escalated to the AI Governance Lead within 24 hours. Investigation and root cause analysis within 5 business days. Remediation plan executed with weekly progress updates. The cadence stops problems festering while preserving the pressure to resolve them.

Every incident is documented, including near-misses, and the record serves three masters: audit trail, continuous improvement, legal defensibility. ISACA's review of 2025's biggest AI failures found they were rarely technical; they were organisational: weak controls, unclear ownership, misplaced trust. Documentation is how you prove you had controls and ownership.

Good-faith reporting is protected from punitive action. If people fear punishment for reporting problems, they will not report until problems become catastrophes. The goal is early detection, not blame.

Policies With Teeth

AI tool usage is monitored through enterprise account logs, output reviews, and random audits. The audit function is what turns a policy from paper into practice.

The policy itself is reviewed quarterly. That frequency sounds aggressive until you look at the calendar: the EU AI Act phases in new obligations every year until 2030, NIST releases new profiles regularly, and the tools change monthly. A policy written in January may be inadequate by June.

Non-compliance has consequences, including suspension of AI access, and they are enforced consistently: policies without teeth become suggestions. But consequences stay proportionate and corrective. The goal is behaviour change, not termination.

What Most AI Policies Miss

Having reviewed dozens of enterprise AI policies, I see the same gaps appear again and again.

Agentic AI

Most policies cover chatbots and copilots but ignore agents: systems that take autonomous actions over extended periods. The attack surface expands dramatically once an AI has tool access, system permissions, and multi-step execution. Wiz's heuristic is worth adopting verbatim: autonomy multiplied by access equals risk. Policies should specify approval requirements for any system that can act without per-step human confirmation.

Third-Party AI Embedded in Products

AI arrives embedded in enterprise software without disclosure. Your CRM scores leads with it; your HR platform screens CVs with it; your security stack triages alerts with it. Policy scope must include third-party software with embedded AI, and the organisation stays accountable for outputs and data handling even when the AI is a vendor's.

Model Provenance and Supply Chain

NIST's 2025 updates emphasise model provenance, data integrity, and third-party model assessment; most organisations adopt external or open-source models knowing nothing of their training data, biases, or vulnerabilities. CrowdStrike's research on DeepSeek-R1 found that prompts touching topics the CCP considers politically sensitive raised the likelihood of severely vulnerable code by up to 50%. Supply chain risk is real, and due diligence on AI components belongs in policy.

Bias Auditing

Generic commitments to 'avoid bias' are insufficient. An audit of LLM CV screening found models favoured white-associated names in 85% of head-to-head comparisons, with qualifications held identical.⁷ Workday faces a class action over alleged algorithmic bias in its AI-driven hiring tools. Audits must test outputs for bias on a schedule, with mitigations ranging from prompt adjustments and diverse test sets to retraining or removal.

Resource Allocation

Policies that mandate training, compliance, and monitoring without resourcing them are performative. AI ethics spending has risen from 2.9% of AI budgets in 2022 to 4.6% in 2024,⁸ yet many organisations still create governance obligations with no budget behind them. Adopting a new AI tool should require a cost-benefit analysis that includes the cost of governing it, not just the licence fee.

The Framework at a Glance

Guiding Principles: Value-driven use, human oversight, data security, critical evaluation, ethical deployment, transparency, designated governance ownership.

Acceptable Use: Automation of low-value tasks, drafting for human review, prototyping, decision support with oversight. Code: scaffolding, tests, documentation, exploration; always with mandatory controls.

Prohibited Use: Processing restricted data in unapproved tools, production code without review, legal documents without counsel review, final HR decisions based solely on AI.

Code Controls: Human review, licensing verification, testing, dependency hygiene, security checks, environment segregation, traceability, performance verification, designated ownership.

Data Handling: Default assumption of logging, restricted data list, approved tools matrix, governance lead oversight.

Quality Assurance: Review and fact-check all outputs, document human oversight, conduct regular bias audits, implement mitigation strategies.

Security: Approved platforms only, IT/Data Protection review for new tools, no credential sharing, prompt injection awareness, emerging threat monitoring.

Training: Mandatory before use, annual refreshers, accessible formats, coverage of policy, prompting, bias, hallucinations, data handling.

Monitoring: Enterprise logging, output reviews, random audits, quarterly policy review, enforced consequences.

Incident Response: Immediate escalation, 24-hour logging, 5-day investigation, weekly updates, stakeholder communication, comprehensive documentation.

Intellectual Property: Assume outputs may not be copyrightable, review for originality, legal counsel for high-value IP.

Regulatory Alignment: GDPR, EU AI Act (timeline through 2030), NIST AI RMF, sector-specific requirements.

Disclosure: Label AI-assisted outputs in code, communications, marketing, and client deliverables.

The Enabling Constraint

Governance enables innovation. That sounds paradoxical until you have watched a promising AI initiative collapse under regulatory scrutiny or reputational damage. Teams with clear guardrails move faster than teams guessing: they know what is permitted, what data they can touch, and what approvals they need, and they spend their energy on the work instead of the worry.

The data agrees. Databricks' 2026 State of AI Agents report, drawing on telemetry from more than 20,000 organisations, found that those with governance in place deploy AI agents at roughly twelve times the rate of those without. The constraint was never the rules. It was uncertainty.

The developer who asked about the dashboard script is now our AI governance champion. Given clear policies, approved tools, and training on the risks, the same team has delivered AI-assisted work across dozens of projects: faster, safer, and without a single incident in eighteen months. The difference was not their character or their competence. It was context.

Every organisation gets an AI governance framework eventually. The only choice is whether you write it before the incident or after.

Choose before.

Footnotes

1. IAPP, AI Governance Profession Report 2025. ↩

2. Stanford HAI, AI Index Report 2025, drawing on the AI Incident Database. ↩

3. Vectara's factual-consistency (hallucination) leaderboard; rates as measured in 2025 and moving as models update. ↩

4. Stanford RegLab's benchmark of AI legal research tools, which tested Lexis+ AI and Westlaw's AI-Assisted Research. ↩

5. Veracode, 2025 GenAI Code Security Report. ↩

6. The taxonomy is Georgetown CSET's, from their work on the cybersecurity risks of AI-generated code. ↩

7. University of Washington study of race and gender bias in LLM resume screening, presented at AIES 2024. ↩

8. IBM Institute for Business Value. ↩