Agents took a real enterprise app to pilot in 26 days. The build was the quick part. What took longer, and what it means for anyone selling AI.
In January, I spent five intensely focused days building a live application for a gathering of more than 200 leaders, and very nearly lost control of it. Agents working in parallel drifted apart, each optimizing its own corner of the code, until reviewing progress felt like archaeology. I threw away roughly two days of work, reset the codebase, and finished with every agent working one at a time through a single orchestrator. The application performed on the day. I wrote about it at the time, and I concluded, with some conviction, that agents belong on a short leash.
That was a sprint. I wanted to know what a marathon would teach me.
So in late August I started a build meant to last: a real application for running a global services portfolio, built with AI agents inside a large company's IT and security rules, and kept alive long enough for real people to depend on it. Since October I have written it up as it happens, in a weekly LinkedIn newsletter called Agents in the Building. This piece gathers its first four issues in one place, with the numbers, for anyone who has not followed it there. It also draws out what they mean for the people who build and sell AI solutions for a living, which is the side of the table I sit on.
Software creation has moved in quick succession from Agile to AI-assisted and now to agentic, and most of the companies I talk to are, candidly, lost on how to do it right at scale. The easy answer, getting rid of the software engineers and letting the machines "vibe code" everything, has failed in public more than once. So the market has settled into a classic analysis-paralysis moment. Some are waiting for the AI bubble to burst. Others are moving as fast as they can and quietly rewriting what their industries expect of everyone else. Most sit uneasily in between, and, I think, understandably so.
Rather than preach from an ivory tower, I chose to bring this home to my own, quite complex problem. The column draws on three builds: mine, and two we built as a small team, an AI-risk planning tool and a workbench of AI agents for software quality engineering. Three builds are a small sample, and I will say where the evidence stops. Each of them has failed at least once in a way worth writing down, and those failures carry most of what follows.
The first commit went in on August 27. On September 2, the seventh day, the demo identities came out, and from then on the only way in was by invitation. On September 22, 26 days after the first commit, the application was formally classified as a pilot "serving real users and real portfolio content", and by mid-October it had 40 to 45 real users. Agents co-wrote 94% of its commits. Of 436 pull requests in its first seven weeks, the median merged 2.2 hours after it was opened. Every change passes a fourteen-job automated gate before it reaches the main line, and nothing reaches production without a human release approval.
The speed, frankly, was the least surprising part. What I had not anticipated was how completely my own job would change. I wrote almost none of the code. I wrote the rules the agents work under instead: twenty of them so far, most composed in the aftermath of something going wrong.
Each rule has a story behind it. One agent, deep in a long session, dated my decisions a day in the future, having inferred the date from how long we had been talking rather than looking at the clock; looking at the clock is now a rule. A safety check once reported zero failures every single time, because the search tool it depended on existed only inside the agent's own shell and quietly vanished in a background script. Empty was read as zero, and zero was read as success.
The most instructive moment came at the end of September. With several agents working in parallel, a fix landed beneath work that depended on it, a shared test environment locked up, the laptop's disk filled, and one agent reported a status it had never checked. I stopped all of them, and for the best part of a week I let a single agent work at a time, much as I had in January. Then I let them back, on written terms: a hard limit on how many could run at once, an isolated workspace for each, and an obligation to verify before claiming anything.
In January I would have shortened the leash. In October I lengthened it, because the scaffolding had grown strong enough to hold.
Half of that scaffolding is the automated checking every change passes through on its way to release, and it turned out to deserve less trust than it appeared to. In all three builds, a check everyone trusted was quietly checking nothing while it reported green.
On the morning of June 11, we committed a test that was written to fail. Its one purpose was to prove that a new gate in the workbench's pipeline would refuse it. Nine minutes later, we took it out again. The day before, we had discovered that the workbench's unit tests had not been running in the pipeline at all: a missing plugin made the step fail within a second, and a setting that allowed failures hid it. For the first 1,671 commits, no failing test or code check could stop a change. A blocking gate over 10,580 tests went in that same day.
In the risk-planning tool, an agent found on September 17 that one test file still referred to a deleted component, which made the test runner abort before running anything. None of its tests had run since at least June 10. Fourteen quiet weeks. Once repaired, the suite showed 518 passed, 22 failed and 7 skipped, and the stale tests expected weaker behavior than the code by then had. An agent told simply to make the tests pass would have weakened the code. This one rewrote the tests instead.
In my own build, on August 29, the third day, the automated scan in a security audit reported zero findings. The agent verifying the audit checked why. None of the three scanners was installed on the machine where the scan ran, so nothing had been scanned. Once they were in the pipeline and set to fail the build, their first real run found a High-rated weakness that the audit itself had missed, since fixed.
And back in the workbench, a run in which every AI reviewer had failed reported, with perfect composure, "0 defects found."
A seasoned engineer will say, with some justice, that all of this is ordinary pipeline hygiene. I agree, with one amendment: agents make it compound faster, and report it with more confidence. They read output the way a hurried person does. In my build, a review agent and the orchestrator, the agent that coordinates the others, both read a code checker's "80 problems (0 errors)" as a pass. The automated gate caught every false green, at the price of extra pipeline runs and rebuilt work. The rules since: judge every tool by its exit code, the plain pass-or-fail signal a program returns, and check firsthand whenever a monitor goes quiet.
If a check has never stopped anything, assume it is not running.
I expected the road through the company's IT approval process to outlast the build, because earlier this year a review through the group's IT, legal and cybersecurity functions had taken two months of iteration. It is a necessary process, and one that is not yet optimized for AI speed: there was no single checklist of every reviewer and checker involved. So I did as much of the work in advance as I could, starting security on day three.
As of November 10, the security approval for production is in. The data-protection approval is still pending, and a couple of steps follow it, so the November 15 production target will likely slip by at least a week. That puts the approval road past two months, against 26 days for the build to reach its pilot.
The approval road is only the start. When you build, the maintenance, the support, the security accreditation and the key-person risk all become yours. The 417 changes merged in those first seven weeks were quick to make, and every one of them is now something somebody owns. Enterprise IT policy, security accreditation and support models were all written on the assumption that you buy, and that assumption will take longer to change than the technology did.
Even so, fit-for-purpose economics have flipped on their head (and software companies selling thin wrappers around a model should be nervous, in my opinion). Every license renewal for a fit-for-purpose internal tool is now a build decision. I did not build everything. The models reach my application through an AI gateway that someone else runs. What I built was the control around them: who pays for each call, a record of what each call uses, and safety rules held in code, out of the prompt's reach. I would buy the models and the platforms, and build the control and knowledge layer, because that is where your advantage and your risk both sit.
For the next renewal, here are the five questions I would ask, in order:
If the answer to the first or second question is yes and to the third is no, and you can answer the last two with a name and a plan, ask what it would take to build. Otherwise the renewal stands, and the question comes back next year.
On August 29, two days after the first commit, an AI security-review agent audited the code against the OWASP Top 10, the industry's standard list of web-application security risks. A second agent checked every finding against the running code and changed four of the severities. With what the scanners added and two findings that surfaced a few days later, the record held 11, two Critical and four High among them. Ten were fixed by September 2, and the eleventh, contained in the meantime, closed on September 17. Because the audit came before the first real user, the worst of it cost a few days of work rather than an incident.
When the group's own security scan ran in early October, its report held nothing rated High or Medium, and I accepted its six Low results as a risk, in writing. An agent drafted the security approval pack for production, and before it went anywhere an adversarial review checked its claims against the code and found eight gaps. The pack states every one of them as a condition for go-live. An approver reading it gets to see the sausage being made before deciding.
In mid-October the agents also built the application its own security testing, run from the pipeline. Its final full run, against a throwaway copy of the application, sent 465,720 requests in under 13 minutes. Of the 189 commits on the main line that mention the security work, 183 carry the mark of an AI co-author, which records participation rather than share.
What the agents could not do was accept the risk. People kept the decisions: holding production while the group's scan ran, accepting the six Low results, setting the depth and timing of the attack scans, and choosing which findings gate production. A skeptic will say the wait for approval is bureaucracy. In my view it is assurance, and an auditor will ask, fairly, who checked the agents' own security work. On this build, part of the answer is a scan the agents did not control.
Agents have moved the long pole. If they shrink your build to weeks, the assurance will set your date, so it deserves the planning and the staff you would give the build, from day one.
Most of what these builds taught me applies as much to the firms that sell AI solutions as to the companies that buy them. If a customer can build a fit-for-purpose tool with agents in weeks, a services business that sells the hours of building it is exposed. It has to move closer and closer to productization to differentiate, and its own intellectual property becomes the hedge against a customer who simply builds it themselves with agents. In practice that means moving the cost upstream, into reusable assets built once and used on every engagement after.
Five threads run through the column for anyone building that kind of business. First, discovery starts with one real, messy problem the customer already owns, run end to end inside the customer's own rules, and the customers who already trust you are the ones who will hand it over. Second, delivery risk has moved. If agents shrink the build to weeks, the customer's approval road sets the date, and with it the day the revenue lands, so it belongs in the plan and the price from the first conversation.
Third, the repeatable part is the control around the models: the rules, the checks that have been proven to fail, the security testing that runs on every change, and an approval pack that lists its own gaps. Partner for the models and the platforms, and package the control. Fourth, every license renewal is now a build decision, which turns a routine renewal into a discovery conversation for whoever is in the room when it comes up. Fifth, the business case needs numbers a board can defend: a pass bar with enough samples behind it to prove it, and a cost recorded for every call.
The craft itself has moved, too. The scarce skill in an agentic team lies less in writing code than in writing the conditions under which machine-written code is safe to ship, and in having the judgment to stop the machines when those conditions are not met. Most organizations have not yet staffed for that craft. They budget for licenses and pilots and assume that someone, somewhere, is reading what the machines produce. If I were hiring for it, I would ask to see the rules a candidate wrote after something broke, and a check they once made fail on purpose. Where those people sit in a job architecture, and how juniors will learn judgment once agents do the grunt work, is the subject of a coming issue.
The track runs further than most people expect. I think software delivery is on a path that will eventually lead to fully autonomous delivery, what manufacturing calls a dark factory: a plant that runs with the lights off because nobody needs to be on the floor. The next agents in the building may well run the building itself. My own build stops well short of that. The release approval and the acceptance of risk both stayed with a person, and on today's evidence I would not hand them over. What stands in the way for now is mostly structural: approval processes, contracts, job architectures and liability, all written for people doing the work.
Agents in the Building runs every week on LinkedIn: a longer article on Tuesdays, and on Thursdays a short Site Note built around one artifact from the build. Coming up are whether the production date held, and why not if it slipped; why most AI sign-offs prove less than they appear to, our own included; where the money for AI actually goes; and the twenty rules themselves.
A taste of the sign-off question, from a recent Site Note: fifty correct answers out of fifty support an accuracy of at most 94.2%, at 95% confidence, and showing 95% with no failures at all takes 59 perfect runs. So a perfect pilot on 50 cannot prove a 95% bar, however well the system performs. It's like gambling when you think you can't lose, but you don't really understand the odds.
For more details, feel free to read the four published issues so far:
In January I wanted to know whether I could build something quickly. Now I want to know whether it holds up for months, with real people depending on it. I would rather find out in the open.
If you have approved an AI program this year, what evidence were you shown?
I’m a technology executive with 30 years at IT solutions providers and in enterprise IT, combining sales and P&L leadership with hands-on AI delivery, from edge AI on industrial sensor data to agentic systems in production. I was born in Switzerland, grew up in Germany and Austria, and I’m fully bilingual in English and German; for the last three years I’ve run a global role from Houston, in Europe about every six weeks. As Chief Portfolio Officer for Sogeti, part of Capgemini, I own the global, AI-led portfolio across 13 countries, with more than €250M in business influenced. Earlier I grew two Texas markets’ revenue and margin, sold or influenced deals upwards of $60M, and built an advisory practice from scratch to about 20 people and $5M a year.
Please sign in to leave a comment.
No comments yet. Be the first to comment!
