The Foundation Has to Be Yours

If you ship hardware and software together, you cannot afford surprises at the operating system level. The foundation has to be yours.

Works in London, Fails in New York

In 2024, we travelled from London to the US to deploy our product for a customer. Back home in the UK, everything had been running smoothly. The cameras worked perfectly, the software behaved as expected, and the deployment felt routine.

Then reality did what it always does when engineers get confident.

In the US, the cameras simply refused to work with the new software stack. What had been stable in London suddenly turned into a competitive disaster overseas. The customer was waiting, expectations were high, and we were stuck watching hardware that worked yesterday act like it had never heard of our code.

For a week, we chased ghosts.

We checked drivers, rewrote parts of the pipeline, questioned our assumptions, and wasted hours on theories that sounded smart but weren't true. We examined kernel logs, traced system calls, diff'd configuration files between environments. Eventually, the answer turned out to be painfully simple: the operating system distribution in the US deployment environment was subtly different from what we had been using in the UK. Small differences in system libraries, kernel modules, udev rules, and package versions caused a very real failure in camera initialisation.

And it cost us time. We were late delivering the deployment, not because the product was fundamentally broken, but because we had built it on an environment we didn't fully control.

That week changed how we thought about reliability.

The Decision to Own Our Foundation

Only after the dust settled did we realise the real lesson: the foundation has to be yours. That was the moment we first seriously considered building our own custom Linux distribution: something we could install across all our hardware, everywhere, with no regional inconsistencies, no surprise package updates, no divergent kernel configurations.

Of course, like many good engineering ideas, it sat on the shelf for a while.

It wasn't until October 2024 that I finally had the time to return to that project. And after months of effort, we built ScalpelOS: our own Linux distribution, which we are currently testing.

I am quietly proud of it. Creating your own operating system layer feels like crossing a boundary: from "we run on top of systems" to "we own the system." It's a different kind of engineering confidence.

Starting from Zero

The funny part is: I had zero experience when I started. I had never built a bootable ISO, never configured a custom initramfs, never wrestled with Plymouth boot screens or GDM authentication chains. I leaned heavily on AI tools to help me learn, explore, and move faster.

But there was a catch. I was working with the latest distributions and tooling, and almost none of the online guides were up to date. Autoinstall had replaced preseed. Snap packages behaved differently in live sessions versus installed systems. Systemd service dependencies had subtle ordering requirements that older documentation ignored. The AI could generate answers, but those answers were often based on old assumptions, outdated commands, deprecated approaches, or configurations that simply no longer existed in modern systems.

It was like having a brilliant assistant who occasionally hands you a map from 2017 and insists the road still exists.

So I had to build carefully.

The Virtual Machine Discipline

One thing that helped me more than anything was using a virtual machine to test everything. I've seen people make the mistake of trying to build system-level software directly on their main work machine: modifying boot configurations, installing conflicting packages, running privileged scripts that assume they can overwrite system files. It usually ends in regret: broken environments, corrupted boot loaders, and hours of recovery.

A VM gives you safety. A clean sandbox. A reset button. You can snapshot before risky operations and roll back in seconds. You can test destructive operations without fear. You can validate that your ISO actually boots on a fresh machine, not just your development environment.

That became the first of many guardrails I put in place.

Guardrails for Safe AI-Assisted Development

Over time, I realised that building safely with AI requires discipline. AI can accelerate your work, but it can also accelerate mistakes if you blindly accept suggestions. A wrong command executed with confidence is still a wrong command. Here are the practices that made this project successful:

Always test inside a virtual machine first. Never run system-level experiments on your primary machine. VMs provide isolation, snapshots, and the ability to destroy and recreate environments instantly. This single practice probably saved me dozens of hours of recovery time.

Keep components modular. Split drivers, packages, boot configuration, and system services into separate, testable pieces. When something breaks, you want to know exactly which module caused it, not debug a monolithic build script.

Test modules independently before integration. Verify each component works in isolation. Test your Plymouth theme separately. Test your autoinstall configuration separately. Test your custom packages separately. Integration failures are much easier to diagnose when you know the individual pieces work.

Make the build process repeatable and automated. No pile of manual steps you'll forget next month. Script everything. A build should be one command that produces the same output every time. If you can't rebuild from scratch, you don't really understand what you've built.

Cache resources locally. Downloading the same dependencies repeatedly during iterative development can get you rate-limited or blocked entirely. Yes, that happened. Cache base images, packages, and external resources. Your build process will be faster and more reliable.

Never accept AI-generated code blindly. Read it. Understand it. Trace through the logic. Verify it applies to your specific version and environment. AI suggestions are advice from a knowledgeable but potentially outdated colleague, not gospel. The moment you copy-paste without reading is the moment you inherit someone else's bugs.

Use version control aggressively. Commit early, commit often, write meaningful commit messages. Being able to revert quickly is the difference between progress and chaos. When an AI suggestion breaks something, you want to get back to a working state in seconds, not hours.

Document every non-obvious decision. You will forget why something was done three weeks later. Document the reasoning, not just the implementation. Future you (or your teammates) will thank you when debugging a subtle issue at 2 AM.

Validate assumptions across environments. "It works on my machine" is not a deployment strategy. Test on fresh VMs, different hardware profiles, and clean installations. The whole point of building a custom OS was to eliminate environment inconsistencies; don't introduce new ones in your build process.

Add checkpoints and rollback plans. Especially when building system-level infrastructure, know how to recover from failures. Snapshot your VM before major changes. Keep known-good configurations archived. Have a plan for when things go wrong, because they will.

Verify AI suggestions against official documentation. When an AI suggests a configuration or command, cross-reference it with the official docs for your specific version. Documentation may be tedious, but it's the source of truth. AI hallucinations in system configuration can cause boot failures.

Start with minimal viable configurations. Don't try to implement everything at once. Get a bootable system first. Then add customisation incrementally. Each addition should be tested before the next. Complexity should be earned, not assumed.

Maintain a decision log. Track what you tried, what failed, and why. This becomes invaluable when you revisit the project months later or need to explain your architecture to others. It also helps you avoid repeating the same failed experiments.

Set up automated testing for critical paths. If something must work (like the boot sequence, network initialisation, or camera drivers), write tests that verify it. Don't rely on manual checking every time you build. Automation catches regressions that human attention misses.

The Moment It Worked

Slowly, the distribution started to come together. Piece by piece. Boot screen configured. Installer automated. Packages bundled. Drivers integrated. Services orchestrated.

And eventually, it booted. It installed. It ran our stack. It behaved consistently across different hardware. The camera initialised on the first attempt.

That was the moment the whole detour paid for itself.

The Honest Downside

Maintaining your own distribution is not easy. Once you build it, you own everything: security patches, kernel updates, driver compatibility, package management, long-term maintenance. You are no longer just a consumer of an ecosystem; you are part of the ecosystem. Every upstream vulnerability becomes your responsibility to patch. Every new hardware revision requires your attention.

It's powerful, but it's heavy.

It can also become a trap if you maintain it "from scratch" without clear boundaries. The more custom it becomes, the harder it is to keep up with upstream improvements, security fixes, and new hardware support. You can end up maintaining two systems: your product and your operating system. That's a significant engineering burden, and it requires dedicated resources.

So the challenge is balance: building enough control to eliminate deployment surprises, without building so much that maintenance becomes its own full-time product. Know what you're signing up for before you start.

The Lesson

Still, the experience taught me something important about working with AI.

AI tools are incredible accelerators. They can help you explore unfamiliar territory, generate starting points, debug obscure errors, and move faster than you could alone. But safe engineering comes from guardrails, modularity, testing, and understanding what you are building. AI can help you move faster, but you still have to drive. You still have to verify. You still have to think.

The combination of AI assistance and disciplined engineering practices is powerful. Neither alone would have gotten me to a working operating system. The AI provided speed and breadth of knowledge; the guardrails provided safety and reliability.

And now, after starting with zero experience in operating system development, we have ScalpelOS: our own custom operating system layer running on our hardware. It's still being tested, still being refined. But it represents something bigger than a technical achievement:

We learned the hard way, and we built something stronger because of it.

The cameras, for the record, now work in every timezone.