The model is pretty good at creating websites from scratch, and people seemed to enjoy the examples, so we decided to put together a follow-up post with 8 new, additional examples.

This time, we tested how well Grok handles interactivity, design, and humor. We also looked through the source code to see whether and how it actually worked under the hood.

Let’s play a game of parrots

The prompt, total cost, and token spend:

The result: This was a dark-mode, modern website, and the interactivity all held up. The parrot animated on click, the speech bubble appeared and faded out cleanly, copy-to-clipboard worked with a graceful fallback for older browsers, and the speech synthesis call fired with pitch and rate tweaked to sound parrot-ish.

A website to book a hotel on the Moon

The prompt, total cost, and token spend:

The result: We got a functional website. Grok also invented the phrase "view of forever" for the suites section headline. We didn't ask for it. The model decided a moon hotel needed that. The three-stop gradient (purple, blue, cyan) is clipped to the headline text, layered over a dark space-themed background.

Try this prompt + the rest in Kilo Code

An interactive Pomodoro timer

The prompt, total cost, and token spend:

The result: The timer behaved correctly through every state transition. The Pomodoro logic handled the full loop: start, pause, resume, reset, completion chime, and auto-advance to the next phase. The code correctly implements the full Pomodoro loop, including a long break after every fourth focus session

Beautify a government website

The prompt, total cost, and token spend:

The result: The multi-step workflow worked end to end:

The five-step form swapped sections cleanly, the progress bar advanced with the step count, and the Continue button correctly switched to "Submit Application" and changed color on the last screen.

Experimenting with old-school interactivity

The prompt, total cost, and token spend:

The result: We got the look we wanted: every interactive piece behaved like a small simulation rather than something static. The CRT effect is doing real work: phosphor glow via text-shadow, repeating scanlines, a sweeping highlight band running top-to-bottom on a CSS animation, a subtle flicker on opacity, and a radial vignette layered on top to fake the screen curvature.

Try this prompt + the rest in Kilo Code

A GitHub profile page for Shakespeare

The prompt, total cost, and token spend:

The result: This looked like an actual GitHub page. Follow toggled between the green Follow state and the gray Following state, and the follower count incremented and decremented to match. The sponsor flow opened a modal, let you pick a tier (Bronze Quill, Silver Sonnet, Gold Globe), and swapped its own contents to a thank-you screen on submit.

Is it raining in Tokyo right now?

The prompt, total cost, and token spend:

The result: We were aiming for functionality here, not design (although the site looked quite decent). The live data fetch worked. The app hit Open-Meteo's free forecast endpoint with the right Tokyo coordinates and a sensible parameter list (weather code, temperature, precipitation, rain, humidity, cloud cover, wind, timezone set to Asia/Tokyo), then mapped the WMO code into three states: heavy rain, light rain, and no rain, with separate icons for each.

A payment dashboard for a lemonade stand

The prompt, total cost, and token spend:

The result: The site looked modern and pretty much every interactive piece held up.

Sidebar navigation switched sections without reload, the Today / 7 days / 30 days buttons recomputed the metrics and rebuilt the Chart.js line graph on the fly, and the status filter pills on the payments view narrowed the table as expected.

You can run these prompts yourself by installing our VS Code extension or using the Kilo CLI. For this round of examples, we used the VS Code extension together with the built-in Agent Manager.

But xAI, a part of SpaceX, didn’t just make financial headlines this week—they also quietly rolled out Grok Build 0.1.

So quietly, in fact, that you might have missed it.

We’re excited to announce that the new model from xAI is live in Kilo. Take advantage of our early access to give the new coding model a spin today.

Built for Agentic Engineering

Designed specifically to tackle complex, autonomous software engineering workflows, Grok Build 0.1 is positioned to rival the best coding models on the market. It’s already live in early access, and here is a breakdown of what makes it stand out.

Unlike standard chat models, Grok Build 0.1 is heavily optimized for interactive coding agents, tool use, and multi-step development tasks.

It is built to reason through a problem, write code, use your terminal, check for errors, and fix its own mistakes in a continuous loop. To support this “long-horizon” workflow, Grok Build 0.1 features a 256K context window (accepting both text and images) and currently boasts no text output limits. Instead of hitting an 8K token cap and forcing you to type “continue generating,” it can autonomously refactor massive codebases in one go.

It’s also just ridiculously good at building. This was the vibe with Grok Code Fast 1 as well—good at creating landing pages and product pages in a particular style, from coding to design—but Build seems to take it to the next level with fast iteration and better one-shotting. For example, it built a professional-level landing page in 40 seconds. For less than ten cents.

If our early testing is any indication, Build 0.1 is going to be a lot of folks’ new daily driver—super fast and more cost-effective than similar frontier models.

Grok Build 0.1 vs. Grok Code Fast

If you’ve been tracking xAI, you might be wondering how this compares to Grok Code Fast (recently deprecated) and other Grok models. TLDR: Build 0.1 aims to be more cost-effective without sacrificing large-scale coding abilities.

When Grok Code Fast went paid, it was a turning point for the industry. The release of Grok Build 0.1, a different type of model with a similar personality, proves that xAI is still focused on fueling agentic engineering.

• Use Case: Grok Code Fast is your autocomplete and quick-refactor king—designed for instantaneous, low-latency edits while you type. Grok Build 0.1, on the other hand, is the “architect.” You hand it a massive Jira ticket, step back, and let it churn through the codebase.

• Pricing: Grok Build 0.1 operates in that highly efficient “Flash” tier sweet spot. Priced at just $1 per million input tokens and $2 per million output tokens, it is built to be economically viable for agentic loops. This is the kind of cost-efficiency needed to let agents run wild.

When it comes to keeping costs in line, just remember that there are no text output limits for this model. This is a good thing, to be sure. But you need to watch it carefully once you set it loose in the wild.

Also note that if you exceed the 200k context window, the costs double.

Already Live and Climbing the Kilo Leaderboards

You don’t have to wait to try this out. Grok Build 0.1 is already live and available wherever you use Kilo.

It’s climbing the Kilo leaderboard as developers test its massive context window against older flagship models. Because it is in early access, now is the perfect time to plug it into your Kilo environment and take advantage of no output limits to see how much of a codebase it can successfully rewrite in one prompt.

When you see a $6.4 billion burn rate on an IPO filing, you expect massive swings, and xAI is definitely swinging. (We should note that OpenAI, who also started the IPO process, is projected to spend $115 billion over the next four years — and recent Grok releases do feel like rising GPT competitors).

With this new model, xAI isn’t trying to build the smartest conversationalist. Instead, they are trying to build the most relentless, cost-effective robotic software engineer for both front-end and back-end tasks.

Give Grok Build 0.1 a spin in Kilo today and let us know your thoughts!

Late in 2025, leadership had made the call: the company was going all in on AI.

The CTO’s vision was straightforward. Engineers would define the work in tickets. Agents would implement it overnight. Engineers would return in the morning to review the output and get the code over the finish line. This was the reality facing an engineering manager I was mentoring at another company. That conversation stuck with me.

It sounded clean in the way plans sound clean before they touch a real codebase, but I felt an immediate resistance to the idea. Not because it was uniquely absurd. By then, this kind of thinking was becoming increasingly common. As the AI hype intensified, many executives felt pressure to show a return, cut costs, or simply prove they were not falling behind.

What unsettled me was that my own experience told me this wouldn’t work.

I had spent much of 2025 in my previous role as Director of Engineering leading the effort to experiment with and adopt AI-assisted engineering. I had seen where these tools helped, where they failed, and how much extra judgment they demanded from team members.

From that vantage point, the CTO’s plan did not just seem technically naive. It seemed like a good way to turn engineers into ticket writers and cleanup crew. That kind of rollout can demoralize a team before they have a chance to learn.

To be clear, the CTO was right to take AI seriously. The shift to agentic engineering is more than another tooling update. The mistake was assuming you can design the operating model before you understand what the work feels like from the engineer’s side.

That is the gap I keep coming back to: leaders are asking engineers to change how they build software before they have felt the change themselves.

As an engineering leader I am supposed to get out of the weeds. That is part of the job. You lead through systems, managers, planning, metrics, and organizational design. But when the weeds themselves start changing, distance becomes a liability.

For me the answer was to get my hands back on the work. Not to take over delivery. Not to relive my IC days. But to understand what this change actually demands from the people living through it. A secondhand understanding was not going to be enough.

The work is changing faster than the operating model

Many organizations are moving quickly from “AI matters” to “we need an AI operating model.” The urgency is understandable. The mistake is pretending the model is already obvious.

The conversation has moved fast: from broad AI skepticism, to one-prompt vibe coding demos, to coding agents, new harnesses, Ralph-loops, spec-heavy workflows, test-first agent patterns, and whatever practice gets treated as the answer next week. Some of these ideas help. Some help a lot in the right context. But none of them have come close to removing the need for human judgment in software engineering.

That is what I mean by agentic engineering: not handing the work to a model and hoping it figures things out, but using agents as part of the engineering process while the engineer still owns the outcome. The engineer defines the problem, sets the constraints, steers the work, reviews the result, and decides whether it is actually good enough to ship.

That shift is real. It is also not settled.

A workflow that works for a greenfield internal tool may fall apart in a mature distributed system. A team with clear ownership, trusted tests, and fast feedback can move faster with agents. A team with ambiguous product direction and brittle verification loops may only create confusion faster. That is why you cannot import an “agentic engineering” playbook and expect it to survive contact with your organization.

This is where secondhand understanding becomes dangerous. From a distance, it is easy to build overconfident plans around demos, vendor claims, and isolated success stories. Up close, the work is messier. Agents can move quickly and still miss the point. They can produce plausible code that shifts the hardest judgment back onto the engineer, and they can make a weak idea look more real than it deserves to be.

Software engineering was never only about writing code. Half the work is figuring out what should exist in the first place. Agents do not make that disappear. If anything, they make bad judgment cheaper to execute.

You start seeing half-baked prototypes and vibe-coded solutions move through the organization faster than the organization can decide whether they are valuable. The burden then lands on engineering to sort out what is useful, what is salvageable, and what should never have been built. Just because something can now be produced quickly does not mean the team has gained leverage. Sometimes it has only gained churn.

AI exposes the bottlenecks your team was already working around. If code review was barely holding together before, generating more code makes that pain harder to ignore. If CI/CD was slow or unreliable, faster implementation just means more waiting for confidence. If product direction is vague, agents help you create expensive noise faster. And if decisions are gatekept by a few people, the compression AI offers stays out of reach.

This is also where the old handoffs between Product, Design, and Engineering start to creak. If engineers are going to direct agents, they cannot behave like ticket-takers. They need enough product context to challenge weak assumptions and notice when an implementation is technically plausible but strategically wrong. That means pulling engineers earlier into the problem framing, not just the solution review. They need to understand the goal, the non-goals, the constraints, the tradeoffs, and what would make the work wrong even if the code functions.

At the same time, leaders have to understand the emotional reality of the team. Many engineers are curious and even excited by these tools. But skepticism is not always obstruction. Sometimes it is a rational response to the way AI adoption gets sold: too glibly, too confidently, and often by people who have not had to land the output in a real production codebase.

Some of the most honest conversations I had with engineers over the last year were not about prompts or workflows. They were about fear. Fear of becoming irrelevant. Fear of watching a hard-earned craft get devalued. Fear of being told to produce dramatically more while the standards they care about become harder to defend.

Leaders cannot give engineers certainty the industry itself does not have yet. But they can decide whether people get to work through that uncertainty honestly. If people cannot talk openly about what is not working, they will still notice it. They will just stop telling you.

That is why leadership credibility matters so much right now. Teams do not need leaders who have memorized the latest AI talking points. They need leaders close enough to the work to know where the leverage is real, where the workflow gets awkward, where the fear is coming from, and where the organization is not ready yet.

Leadership credibility now requires getting close to the work

This is the part you cannot pick up from a deck. You need enough hands-on time to feel where the workflow gets awkward, where the agent becomes slippery, and where the review burden quietly shifts back onto the engineer. You cannot rely only on past experience when the way engineers plan, implement, and review code is changing.

You have to roll up your sleeves and get sucked in. Give an agent a real piece of work, not a toy demo. Watch it move fast, miss context, invent confidence, and leave you with the uncomfortable job of deciding whether the output is good or merely plausible. There is a specific kind of humility that comes from watching a tool produce 800 lines of code while quietly misunderstanding the problem.

Spend enough time with these tools and you will eventually find yourself swearing at an LLM in all caps. That frustration is not incidental. It is part of the experience leaders need to understand.

The point is not to take delivery back. It is to stop leading from theory.

For engineers to trust you through this change, they need to believe that you understand more than the executive summary. They need to hear you speak authentically about the scars: the false starts, the cognitive load, the workflows that seemed elegant until they met a real codebase, and the moments where the tools genuinely changed what felt possible.

Articles, conference talks, podcasts, and vendor demos can inform you. They cannot substitute for time with the tools, or for the trust you need when asking a team to change how it works.

That proximity also matters upward. If someone above you expects overnight productivity miracles, firsthand experience gives you something sturdier than vibes to push back with. You can explain what is real, what is hype, where to invest, and where the organization is not ready yet.

Teams do not need leaders with certainty right now. They need leaders with context. They need leaders whose optimism has been tested by contact with the work.

Learn faster before you optimize harder

With this much uncertainty, the worst move is pretending you have already found the process. The better move is to build an organization that can learn faster than the tools change.

That does not mean letting everyone wander off in private. One of the easiest ways to waste this moment is to have ten engineers independently discover the same lesson, struggle with the same broken workflow, or find the same useful pattern without anyone else benefiting from it.

The goal is not early standardization. It is shared learning.

Create deliberate places for engineers to compare notes: what worked, what failed, what looked promising until it touched the codebase, what saved time, what created more review burden than it was worth. Make the practical details visible. Which model worked for which kind of task? Which harness fits which workflow? Which agent-generated code looked plausible but was painful to land?

This matters more with AI because the feedback loops are compressed. A bad assumption can turn into a prototype in an afternoon. A vague ticket can become hundreds of lines of code. A weak product idea can look more convincing simply because someone generated a working demo.

Without fast, shared learning, the organization does not just duplicate effort. It duplicates mistakes.

So give people low-risk places to experiment, but make the learning part of the work. Use smaller projects, internal tools, or bounded product areas where the blast radius is manageable. Then bring the lessons back into the team through demos, write-ups, office hours, shared channels, or lightweight internal playbooks. Not as commandments. As evidence.

Share what works, but do not turn it into policy too quickly. The weird differences matter right now. One model may fit one kind of work better. One coding harness may suit one engineer better. One workflow may work beautifully in a clean service and fall apart in a legacy system with weak tests. Treat that variation as signal before you standardize it away.

Right now, I would rather be in an organization that learns quickly than one that standardizes too soon. Premature optimization gives the comforting appearance of control. But you will not design the right process in a strategy session. You and your teams will discover much of it by trying, failing, comparing notes, and slowly turning repeated lessons into practice.

What engineering leaders should do now

If you lead engineers, stop outsourcing your understanding of this shift.

Read the articles. Listen to the talks. Stay close to what practitioners are discovering. Then ship something.

The best thing I did as a leader was spend enough time with the tools to become harder to fool. Not cynical. Just less impressed by the demo version of the story. I could see where the leverage was real, where the workflow got awkward, and where the review burden quietly returned to the engineer. It gave me better questions for the team, better language for pushing back on unrealistic expectations, and more empathy for the frustration engineers were feeling. I became less certain in the abstract, but more useful in practice.

So let the work correct your assumptions, because you cannot afford to lead agentic engineering from the sidelines.

At the I/O event today, Google made a staggering number of announcements. But the biggest news was beneath the headlines: Google’s release of a new AI model, Gemini 3.5 Flash, designed for agentic work. We got the new model live in Kilo before I/O 2026 had even ended.

The new model from Google DeepMind is already powering Google Search. But is it affordable and reliable enough for everyday use in your favorite coding tools?

In practice, Gemini 3.5 Flash is around 60% more expensive than comparable models like DeepSeek V4 Flash. But it offers frontier-level performance, reaching an average score of 74.2% on PinchBench in initial runs, similar to Opus 4.6.

Welcome to the age of the Flash model.

In the release, Google focused on benchmarks comparing 3.5 Flashto flagship models from other frontier labs like Anthropic and OpenAI. But the Kilo developer community is already comparing it more closely to budget-friendly alternatives—other flash models released recently that are also designed for always-on agentic engineering.

The release of Gemini 3.5 Flash comes on the heels of powerful, cost-effective Flash model releases from open source labs like StepFun and DeepSeek. This is only the beginning of the agentic era.

Developer’s Delight: A Crowded Flash Ecosystem

Google has fundamentally changed the pitch for its Flash tier. Gemini 3.5 Flash is no longer “cheaper if you just need basic stuff done on a regular basis.” It officially beats Gemini 3.1 Pro on most coding and agentic benchmarks, runs roughly 4x faster than comparable frontier models, and boasts a massive 1M-token context window.

Google declared that the new flash model offers “advanced reasoning at Flash-level latency and scale” and JetBrains has shared that the new model “improves low reasoning coding performance by 10–20% compared to the previous Flash generation.” Our initial tests have confirmed this.

But the competition for the ultimate developer AI has never been fiercer. The battle is officially a race to the bottom in price and a race to the top in context length, and developers have a surplus of incredible options:

• Step 3.5 Flash: Quietly dominating the agentic coding space recently and currently free to use in Kilo. StepFun has become the go-to model lab for developers running continuous multi-agent loops due to its reliable tool-calling capabilities. Their open-source Flash release been popular across KiloClaw, cloud agents and our VS code extension, which is why it’s been totally dominating the Kilo leaderboard.

• DeepSeek V4 Flash & Pro: DeepSeek continues to break the price-performance math. DeepSeek V4 Flash is an efficiency-optimized Mixture-of-Experts (MoE) model built for sheer speed and high-volume pipelines. For developers needing heavier lifting, DeepSeek V4 Pro (packing 1.6T total parameters) steps in for deep reasoning and complex, multi-step agentic coding tasks without losing the ecosystem’s cost-efficient edge. As I wrote back in April (seems like a decade ago!), DeepSeek has adapted to the new micro-model way of seeing the world with this release, opting for Pro and Flash releases instead of a wider base model release.

• Xiaomi MiMo-V2-Flash: Xiaomi is another Chinese lab that has been setting the Kilo leaderboard on fire lately, with their bigger V2.5 releases well as MiMo-V2-Flash, a specialized 309B MoE model purpose-built for high-throughput inference and loop-based agentic tasks. Using a unique hybrid attention architecture, it is designed specifically for scenarios where a model must continuously write code, execute it, interpret the error, and iterate.

Flash Models are Here to Stay

While tech Twitter argues over these static benchmarks, the real revolution is happening in autonomous workflows. Flash models are the undeniable future of agentic engineering, because of their lower cost, high throughput, and focus on effective tool-calling.

Traditional coding assistance involves single-turn prompts: you ask for a function, the AI writes it. Agentic engineering, however, involves giving an AI an open-ended goal and letting it plan, write code, run tests, debug errors, and iterate in a continuous loop until the job is done (or until it times out…but hopefully until the job is done). Previously, executing these tasks with heavy frontier models was financially impossible for daily use. An agent looping through a massive codebase could burn through hundreds of thousands of tokens in minutes—and that’s not token-maxxing, it’s token-wasting.

These new flash models from Google, DeepSeek and others drastically reducing the cost per million tokens, developers can finally let agents run wild. This drop in API costs is democratizing autonomous software development. It allows solo developers and smaller startups to spin up armies of specialized AI agents—one for writing tests, one for refactoring, one for security audits—at a speed and price point that makes full-scale agentic engineering a reality.

Playing offense in the agentic engineering wars means optimizing for the small things. In my opinion, per-agent tool permissions are one of the coolest things about our latest VS Code extension. Flash models make it possible to keep those agents both within-their-guardrails and affordable.

But What About Defense?

Beyond the Flash tier, which sees Google moving in to the offensive with development to rival other frontier players, Google also used today’s I/O event to address the elephant in the room: Anthropic’s Mythos.

To rival Anthropic’s enterprise testing, Google officially pushed CodeMender further into the enterprise today. Although Google DeepMind announced CodeMender back in 2025, Google used I/O to release it into wider testing and share findings from the Google Threat Intelligence Group (GTIG).

With Google pitching CodeMender as the ultimate active enterprise shield, the AI arms race has officially shifted from generating boilerplate code to autonomous cyber defense.

Google is playing all fronts, and Kilo is also here to help you play both offense and defense too. Use Gemini 3.5 Flash as your daily driver for agentic engineering, and any model you’d like for Code Reviews and Security Reviews.

Stay strong with the strength of Google. Stay nimble with model freedom ;)

Back in March, we announced Gas Town by Kilo and opened the beta waitlist. The pitch was simple: take Steve Yegge’s multi-agent orchestrator, remove the operational overhead of self-hosting it, and run it on Kilo Cloud with the full Kilo Gateway behind it. Managed infrastructure, elastic scaling, 500+ models through a single API, and no tmux wrangling.

Since then, engineer John Fawcett and the Kilo team have been working closely with Steve to make Gas Town accessible to developers everywhere. Today, Gas Town by Kilo is generally available as the only cloud-hosted version of Steve’s multi-agent orchestration framework, turning a single developer into an AI-powered engineering team. And it’s launching with built-in support for the Wasteland, Steve’s long-teased vision for federated AI-powered software development, now available as a hosted consumer offering for the first time through Kilo.

What is the Wasteland?

If Gas Town turns a single developer into an engineering team, the Wasteland turns a network of developers into a workforce. It’s a federated system built on Dolt (a SQL database with Git semantics) where participants browse a shared Wanted Board of tasks, claim work, submit evidence of completion, and get stamped by validators who attest to the quality of what was done.

The stamps aren’t pass/fail. They’re multi-dimensional attestations covering quality, reliability, and creativity, each scored independently with confidence levels. Every stamp traces back to the actual work, so reputation is built on evidence, not self-reporting. Steve describes it as “a yearbook rule: you can’t stamp your own work.” Your reputation is what other people write about you.

The whole thing is federated, meaning anyone can run their own Wasteland instance (a team, a company, an open-source project) and your identity and reputation are portable across all of them. Think of it as a professional ledger that you never have to write yourself, because your work writes it for you.

What This Means for Gas Town by Kilo Users

From the Gas Town dashboard, you’ll see your Wasteland connection settings, including which fork you’re connected to and your rig identity.

Here’s what the workflow looks like in practice:

You ask your Mayor to pull the top wanted items from the connected Wasteland. The Mayor shows you what’s available. You pick a task, and your Gas Town claims it, spins up the right polecats, and starts working. When the work is done, your Mayor submits the completion evidence (a commit, a PR, a link) back to the Wasteland as a DoltHub pull request. A validator reviews it, stamps it, and your reputation updates.

The whole thing happens through the same Gas Town by Kilo interface you’re already using. You talk to your Mayor, your Mayor coordinates with the Wasteland, and the work flows through the same managed infrastructure that’s been running your convoys.

For Wasteland administrators, there’s a review tab in the dashboard where you can see submitted evidence, inspect completions, and merge or reject PRs. Posting new wanted items to the board is just as straightforward: write up the task, post it, and it shows up on the Wanted Board for any connected rig to claim.

Why This Matters on Kilo

The Wasteland protocol itself doesn’t require Gas Town. You can participate with just Dolt, a DoltHub account, and a coding agent that knows the schema. But running it through Gas Town by Kilo makes the whole thing a lot easier:

Your Mayor handles the protocol. Claiming work, submitting evidence, managing DoltHub PRs, posting wanted items: all of that is handled conversationally through your Mayor rather than through manual CLI operations against Dolt.

The Gateway handles the models. Wasteland work can be anything: coding, documentation, design, research. Different tasks benefit from different models. Because Gas Town by Kilo runs on the Kilo Gateway, your polecats can use whatever model fits the job, and you’re not managing separate API keys or billing for each one.

The infrastructure is handled for you. No Gas Town CLI, no Gas City CLI, no Beads CLI, no Dolt CLI, and no Wasteland CLI. Kilo packages all of that into a single managed container so you skip the setup entirely and go straight to working.

Your reputation builds automatically. Every stamp you earn feeds into a portable, evidence-backed work history. If you’re already using Gas Town by Kilo to ship faster, the Wasteland adds a layer where that output gets recognized across a federated network.

Where This is Heading

Steve has been clear that the Wasteland is early, but the trajectory is worth paying attention to. Gas Town already proved that a single developer can operate at the scale of a small engineering team. The Wasteland extends that to coordination across developers, projects, and organizations.

The combination of Gas Town’s multi-agent orchestration, the Wasteland’s federated reputation protocol, and Kilo’s managed infrastructure means independent developers and small teams can participate in large-scale collaborative engineering without the organizational overhead that usually comes with it.

Get Started

Gas Town by Kilo is live now. Head to the Kilo Dashboard to spin up your town and connect to a Wasteland:

Get started with Gas Town by Kilo →

For the full context on how the Wasteland works, Steve’s post “Welcome to the Wasteland: A Thousand Gas Towns” covers the protocol, the trust system, and the federated architecture in detail. The Gas Town and Wasteland repos are both open on GitHub.

We said last time that Steve built a coding factory and we gave it plumbing. Now the factories can talk to each other.

Last month my agent costs crossed $800 in a single day, and on paper nothing about my work explained it. I was using the same models, on the same projects, and running the workloads I’d been running for weeks. I was tracking spend at the session level, and the per-session numbers all looked reasonable on their own, which is exactly the kind of thing that lets a problem hide in plain sight.

When I finally pulled the full session data and traced the curve, the culprit was a slash command I’d written myself - something I used every day and had no reason to suspect, because each individual call inside it looked cheap.

The cost was in the shape of the run, not the price of any one piece of it.

The workflow

The slash command sits at the end of my dev loop. I run /ship, which I prompted into existence with four words: “commit, push, pr, babysit”. From there the agent picks up the manual tail of the process I never enjoyed doing.

It watches the pipeline status, keeps an eye on the Slack triggers tied to the PR, and once the review bots start leaving comments, it works through them and addresses the ones that hold up.

On paper this is exactly the kind of work an agent should be doing. It’s repetitive, it’s interrupt-driven, and most of it is the kind of waiting that a person can technically do, but resents doing. The trap, as I eventually figured out, lives inside the word “watches.”

How the cost actually accumulates

An agent waiting for CI does not wait the way a person does. A person waiting for CI gets coffee. An agent waiting for CI polls, and that distinction is where the bill comes from.

Every poll is a tool call. Every tool call pulls its output, in my case from the gh CLI, back into the model’s context. The next poll then runs against a context that already contains the previous poll’s output. The one after that contains both outputs, and the transcript keeps growing on every iteration. Each loop ends up reading and reasoning over more material than the one before it.

If the pipeline passes on the first try, none of this matters. You get a handful of polls and the loop ends before the cost has a chance to build. If the pipeline fails, retries, and fails again, you’re suddenly in a loop where each iteration is more expensive than the last, because the context coming into it has been padded by everything the loop has already seen.

The pricing on each token hasn’t changed. You are just buying more and more of them per call as the loop drags on. In my sessions, a single watch loop could triple the total cost relative to what the same work would have run without one. The damage is in the integral, not the peak.

What the curves show

I pulled the three most expensive sessions I had on file:

• Session A: $101.65 across 283 requests, peak context of 917k tokens

• Session B: $132.96 across 274 requests, peak context of 1.1M tokens

• Session C: $182.99 across 496 requests, peak context of 1.3M tokens

Session C is the one that made the pattern impossible to miss. /ship (my custom slash command) fired around the 151 minute mark, and from there the session was dominated by a long tail of CI and check log calls stretching across the next eighty minutes.

Each of those calls is the loop reading the pipeline status one more time. The cost curve climbs the whole way, not because the agent is making progress, but because it is paying full price to wait.

That 496 request count, sitting almost twice as high as the other two sessions, is almost entirely the loop. The other sessions were expensive for their own reasons, but neither of them produced the long, low-grade tail of status checks that look cheap individually and aren’t.

The worst case is Opus plus flaky tests

The curve gets uglier when two things line up.

The first is Claude Opus, which has a prompt cache window of roughly five minutes. Once your polls space out wider than that, or your retries land on intervals that straddle the boundary, the cache stops helping. Cache discounts are meaningful, and losing them on every call inside a long loop adds up faster than you’d expect.

The second is flaky tests. When tests fail and retry, the watch loop doesn’t just keep polling at the same pace. It re-reads more output, re-evaluates state, and then sits there waiting for the next attempt. Each retry restarts the cost arc on top of an even larger context.

What you get is a cost curve that climbs in steps rather than a smooth line. One step per retry, and the height of each step is set by how much transcript the loop has already accumulated by the time the retry hits - a compounding effect that can seriously drain your token budget.

What to look for in your own setup

I described this to another engineer on the team and he recognized it in his own slash commands inside of a minute. I think the pattern is broadly under-noticed because no single call inside the loop looks expensive on its own. You have to look at the full session curve to see what’s actually happening, and most people don’t do that until something has already gone wrong.

If you have any agent flow that watches something rather than does something, pull up the cost curve for a session where it ran. Look for a sudden slope change tied to a specific tool call. That’s almost always where the loop began.

It doesn’t matter whether you’re polling a CI pipeline, watching a queue, or monitoring a Slack channel for a response. The mechanics are the same. The fix is to stop letting agents wait in-context: use webhooks, longer poll intervals, or an inexpensive model to summarize state between checks. The context window does not care what you are waiting for.

One piece of feedback came through louder than the rest: developers wanted more control back.

Not “turn off the AI” control. More like “show me what you’re about to do before you do it” control. The kind of oversight that lets you trust an agent because you can verify its work, not because you’re hoping it got things right.

We’ve been shipping fixes steadily since the Week Three update. Here’s where things stand.

What’s Already Shipped

See the Agent’s Thinking

The simplest ask was visibility. When an agent proposes changes, you want to understand why — not scroll past walls of collapsed reasoning to find the diff.

Reasoning blocks now stay expanded by default. Terminal command blocks can collapse to just a header bar so they don’t dominate the view. The context usage graph landed in the task header so you can see token consumption turn by turn.

These sound minor but in practice, they’re the difference between scrolling past an agent’s work and actually reading it.

Better Diffs Before You Approve

The v7 extension initially showed diffs after approval. But our community made it clear many desire to review changes before committing to them.

Patch operations now show diffs in the permission prompt before you approve. Edit tool diffs open in side-by-side mode by default. Agent Manager diffs auto-expand reviewable files while keeping generated or huge files collapsed.

The unified Changes panel was a bigger lift. Instead of two separate views for “workspace changes” and “session changes,” there’s now a single panel with a source dropdown. The sidebar badge shows session additions and deletions. If you’ve disabled snapshots for a repo, it warns you instead of silently showing stale data.

Shell command output got syntax highlighting, copy buttons, and an “Open in Editor” action that shows full untruncated output. You can actually read what the agent ran.

Permission Flow Fixes

The auto-approve toggle now persists across VS Code restarts. It shows in the session prompt controls so you know what mode you’re in. The settings UI got readable section titles in light themes.

A more subtle fix: permissions were getting stuck when multiple panels showed the same prompt, or when a subagent’s permission got routed to the wrong worktree. Those now surface clear errors instead of leaving buttons permanently disabled.

Custom agents can have per-agent tool permissions configured in the VS Code agent editor. If you want a read-only reviewer or an agent that can’t touch bash, you can set that up.

Checkpoint Improvements

Session recovery is more reliable. Timeline, token, context, and cost stats stay in sync after restoring a checkpoint. Older revert points stay available after you restore an earlier message. The thinking level preserves through compaction.

Todo items show compact update cards when checked off in long lists, so the review surface doesn’t get cluttered.

What’s Still Coming

The core ask — see exactly what’s changing before approving it with accurate, focused diffs — is tracked in #8415. It’s not fully resolved yet, but we’ve been shipping some of the pieces like patch diffs in permission prompts, and the unified Changes panel, but the full long tail of making this exactly the experience we want is still being worked on.

Step-level checkpoint restore (#8378) would let you see which files changed at each agent step and restore to a specific step within a turn — not just at the user-prompt level. Useful for recovering from rate limits or interrupted runs mid-turn.

The diff code blocks still take too much vertical space. Long edit blocks push reasoning off-screen. The fix (scrollable containers with a higher threshold before expanding) is agreed on but not shipped.

And for new users, there’s an idea to ask how they prefer to work — more hands-on or more agentic — and configure the extension accordingly from the start.

The Philosophy

When I joined Kilo, I heard “agentic engineering” tossed around a lot. It can sound like “let the AI do everything.” That’s not what it means - you can outsource some of software authoring to AI but you cannot outsource the thinking behind software engineering.

The best agents are the ones you can trust because you understand what they’re doing. The tooling makes oversight easy — that’s the point. Review before approve. Diff before commit. Reasoning visible, not hidden.

The v7 rewrite pushed hard on capability — Agent Manager, parallel sessions, tighter model integration. Some of that came at the cost of visibility and control. The work over the past few weeks has been about rebalancing. The tracking issue for human-in-the-loop improvements (#8415) has five active areas, and not all of them are closed. But the direction is clear, and the shipped improvements are real.

If you downgraded to 5.x because v7 felt too hands-off, this is a good time to check back. The experience is meaningfully different now.

Following along? The HITL tracking issue has the full list. And if you hit rough edges, file an issue — the heap-snapshot command we added makes reports actionable.

On February 19, the Terms of Service were updated to formalize the ban.

On April 4, OpenClaw was completely cut off and other third-party harnesses from subscription access entirely. Boris Cherny, head of Claude Code, cited “unsustainable demand”.

And yesterday, the ClaudeDev Twitter account announced that paid subscriptions will get dedicated monthly credits for programmatic usage—after the user opts in and billed at full API rates—while “interactive” use through their own tools stays subsidized.

The official line is that this simplifies billing. The pattern of the last five months tells a different story.

“Interactive” Means “Claude Code’s Front-End”

Jeremy Howard called out the framing immediately:

“This policy redefines the term ‘interactive’ to mean ‘using an Anthropic front-end’. If you use claude -p or Agent SDK to do something interactively, it now uses credits, not your subscription limits. So the ‘interactive use’ heading saying ‘unchanged’ is not accurate.”

He’s right. The word “interactive” in the announcement may not mean what you’d expect. A human sitting at a terminal, issuing commands through claude -p, responding to Claude’s output, making decisions—that’s “programmatic” now. While the “p” in claude -p does indeed stand for programmatic, running Claude Code inside the official CLI to do the exact same work is still classified as “interactive.” The article’s core argument remains: the distinction is primarily about controlling the surface.

The distinction isn’t about whether a human is in the loop. It’s about whether they can control the surface.

Follow the Incentives

Why would Anthropic subsidize usage through Claude.ai and Claude Code but charge API rates for the same tokens through a third-party tool?

The capacity argument doesn’t hold. Third-party harnesses like OpenClaw made the same API calls as Claude Code. The tokens cost the same to run. If the issue were purely cost, raising prices across the board or enforcinged stricter rate limits could have been part of the solution. Instead, they’re treating identical workloads differently based on where the request originates.

Anthropic’s privacy documentation says consumer plan data can be used for model training if you have Model Improvement enabled. Chat and coding session data includes “the entire related conversation, along with any content, custom styles or conversation preferences, as well as data collected when using Claude for Chrome.”

When you use Claude through Claude Code surfaces, they can observe your prompts, your corrections, your workflow patterns, your thumbs up/down feedback.

The subsidy isn’t about making AI accessible. It’s about routing valuable interactions—the kind that generate training signal—through surfaces they control.

The Pattern Was Always There

Peter Steinberger, OpenClaw’s creator, claimed that Anthropic was absorbing features from third-party tools into Claude Code, then shutting out the alternatives. “Funny how timings match up,” he commented.

The timeline supports him:

• January 9: Server-side blocks cut off OpenCode and other harnesses spoofing Claude Code headers

• Late March: Policy warnings to third-party tool users about exceeding limits

• April 4: Billing-based enforcement begins—OpenClaw and similar tools lose subscription access entirely

• May 13: The pricing split goes official—interactive surfaces stay subsidized, everything else goes to API rates

Each step was framed as capacity management or billing simplification. But the consistent thread is: third-party tools that bypass the observation layer get progressively less access to the subscription subsidy.

The Investor Announcement

José Valim, creator of Elixir, cut through the framing:

“The reason they made it sound like an upgrade was because the announcement was not for developers. It was for investors and enterprise customers. Impacting devrel is just collateral damage.”

And:

“if they can’t be honest about pricing changes, it is really hard to believe them on anything else”

Anthropic is reportedly in talks to raise funding at a $950 billion valuation. Dario said their revenue is growing 80x annualized. The metrics that matter for that trajectory are enterprise contracts and revenue per user, not whether indie developers can run overnight automation for $100/month.

The announcement landed the same day OpenAI offered two free months of Codex for enterprise customers who switch.

What Developers Actually Lost

Reuven Cohen, Agentic Engineer at Cognitum One, laid out what the old pricing actually covered:

“The real value was programmatic access. Before this change, the $200 plan had fuzzy limits but absurdly high practical value if you knew how to use it.”

His list of now-expensive workflows: recursive agent loops, headless automation, CI/CD pipelines, multi-agent swarms. His summary of the new credit system: “a rate limiter wearing a party hat.”

Theo added: “Kind of crazy that Anthropic spends more time trying to lock out better apps and harnesses instead of just fixing Claude Code.”

Power users built entire systems around the old economics—automated overnight coding, scheduled tasks, agents that respond to external triggers. All of that now costs 5-10x more. Some are already moving to Codex. Others are eyeing DeepSeek or Kimi.

What They Should Have Said

Valim proposed the honest version:

“We’re moving programmatic use to API pricing. We’re giving you credits to ease the transition. We expect this to free up capacity, so we’re doubling interactive limits for 2 months.”

That’s a price increase for heavy programmatic users, clearly stated. Instead, Lydia Hallie from Anthropic tweeted “you don’t pay extra”—technically true if you’ve never used programmatic features before, misleading if you have.

My Read

The subsidy was never sustainable, and the recent changes are proof that the subsidies are ending entirely. This may also be a sign that Anthropic is making up its pricing as it goes.

From an economic angle, everything that has happened since January makes sense. The OpenCode block, the OpenClaw cutoff, the “interactive vs programmatic” split—all of it is repricing subsidized usage and pushing programmatic use to market-rate API billing as the subsidy era ends.

That’s a legitimate business decision. Companies optimize for what matters to their growth. The problem is calling it a “simplification” when it’s actually a repricing, and defining “interactive” as “our surfaces” when that’s not what the word means.

The pattern is clear: they are ending the subsidies for programmatic use to improve unit economics, prioritizing the revenue that comes from high-value enterprise and direct use over subsidizing third-party tool developers.

The developers who built the harness ecosystem—OpenClaw, Conductor, OpenCode, all of it—turned Claude Code from a CLI experiment into a serious development environment. Pricing those developers out might improve short-term unit economics. It also means the next generation of tooling gets built on someone else’s models.

The change goes live June 15. Subscribers will get an email on June 8 to activate the new credit system. If you rely on claude -p or third-party tools, audit your usage now—$200 in API credits disappears faster than the old subsidized limits.

The story everyone was telling back then was that AI coding was going to consolidate around one or two model providers and consequently, one kind of compute. Kilo was building toward an alternative assumption: that engineers would want to use a dozen different models routed to a dozen kinds of silicon, picked per task, with no one provider owning the entire stack.

This week, the market wrote a $48 billion check on that assumption.

Cerebras prices its IPO at a fully diluted valuation north of $48 billion on an order book reportedly oversubscribed 20 times. It’s the biggest US listing in close to five years. And while the financial pages will spend the rest of the week talking about wafer economics and OpenAI warrants, the more interesting story for anyone who codes for a living is what the IPO actually confirms about where AI compute is going.

The thesis under the IPO

The reason Cerebras can price like this is that AI compute is splitting into specialized lanes. Training is one job. Memory-hungry agentic work is another. Fast reasoning where someone is waiting on tokens is a third. No single chip wins all three, and engineers who can route the right task to the right silicon are going to outpace engineers locked to one stack. That’s the bet inside the $48B valuation, and it’s the same bet Kilo’s product has been organized around since day one.

Cerebras specifically owns the fast-reasoning lane. When a model fits on its wafer-scale chip, you get tokens-per-second numbers that change what coding with an AI feels like. A reasoning chain that takes a minute and a half on a GPU can finish in single-digit seconds. Anyone who has watched a model “think” for 90 seconds before answering a real question knows how big that gap is, especially when it comes to minor implementation tasks and quick revisions in a coding editor.

Why Kilo was building toward this all along

Kilo’s product is shaped by one bet repeated in every direction: openness compounds. Open pricing means you pay the provider’s actual rate with no markup. Open model selection means 500-plus models behind one dropdown, switchable mid-session. Open source means the harness your agent runs in is something you can read, fork, and improve. None of these are features bolted onto a closed product. They’re the spine.

The reason that spine matters is most obvious right now, with Cerebras pricing. If your coding agent is wired to a single provider, the heterogeneous compute future is happening to you, not for you. Every new specialized chip becomes a thing you can’t use. Every shift in which model is best at which task becomes a vendor negotiation. The whole point of an open agentic platform is that fragmentation is fuel, not friction. New silicon arrives, Kilo makes it accessible, and the engineer can move faster.

This is why Cerebras landed in Kilo eleven months ago and not last week. The same logic that said “support Cerebras early” also said support every other model worth running, build a gateway that bills at cost, keep the source open so providers can contribute back. The IPO didn’t change the strategy; it confirmed that the strategy was reading the market correctly.

The collaboration behind the integration

It’s worth saying something about how Cerebras support in Kilo has actually stayed sharp, because it’s the part of the story that doesn’t show up in a press release.

For most of the last year, engineers from Cerebras have been contributing to the Kilo codebase directly. Not in a partnership-announcement way; in a pull-request way. Updating the model lineup as new ones launch. Tuning parameters. Wiring up integration headers. The unglamorous runtime work that determines whether a provider feels first-class or just feels supported.

A coding model is only as good as the harness behind it. Tool calls, file edits, plan steps, diff review, context management, all the parts that don’t show up in benchmarks. Cerebras putting hours into making that harness tight inside Kilo, in public, in PRs anyone can read, is what working with an open platform actually looks like. It’s why a Kilo user hitting Cerebras today is using something that’s been shaped by the people who built the chip, not just bolted on by us.

That kind of relationship only happens in the open. A closed agent forces every chip company into a vendor negotiation. An open one lets them ship code.

What heterogeneous compute looks like inside the editor

Here’s the version of the open compute story that actually shows up in a developer’s day. Open the provider menu in the Kilo IDE extension. Cerebras is there, next to Anthropic, OpenAI, and the 500-plus models on Kilo Gateway. Pick a wafer-scale model and have Kilo’s Plan mode plan a refactor across five files. The design comes back before you’ve finished reading the prompt you wrote.

Then, for the next task, pick something different. A Claude model for a delicate code review. A long-context model for repo-wide refactoring. Same workflow, but different chip running underneath - picked because it fits this task, not because it’s the only one available.

That’s the experience the Cerebras IPO is signaling the market wants. It’s also the experience Kilo has been building since before the IPO was on the calendar.

The market signal

Specialized inference silicon is no longer an interesting bet. It’s an obvious one. Cerebras going public at this valuation is the loudest confirmation yet that the inference market belongs to whoever can match the right workload to the right hardware. Specialized chips will keep arriving. Some will win their lanes, some won’t. The agentic platforms that thrive will be the ones engineers can trust to add the good ones quickly, route between them honestly, and never lock the choice down.

Kilo is the coding agent that has been building for that world all along. We were excited about Cerebras eleven months ago. We’ll be on whatever launches next month. The bet was always that openness wins as the compute landscape gets more interesting, and the compute landscape just got a lot more interesting.

The convenient part for anyone with an editor open: none of this is a roadmap. It’s already in the menu.

Try Kilo with Cerebras: install the VS Code or JetBrains extension or run npm install -g @kilocode/cli, then add a Cerebras key under Settings → API Provider, or use Kilo Credits at provider-rate pricing.

We ran both through the same FlowGraph spec we used for Claude Opus 4.7 vs Kimi K2.6. With the same spec, same prompt, same scoring rubric.

TL;DR: DeepSeek V4 Pro scored 77/100 for $2.25 and lands between Opus 4.7 (91) and Kimi K2.6 (68) in terms of performance. DeepSeek V4 Flash scored 60/100 for $0.02, a price point we have not seen on this test before, but its build failed and the output is missing some key pieces.

The Four Models We Compared

DeepSeek V4 Flash is the cheapest model in the comparison by a wide margin. Output tokens cost less than 1/14th of Kimi K2.6 and roughly 1/89th of Claude Opus 4.7.

DeepSeek is also running a 75% off promotion on DeepSeek V4 Pro through May 31, 2026. Under the discount, DeepSeek V4 Pro input drops to roughly $0.036/M and output drops to $0.87/M, putting it below Kimi K2.6 on both axes. DeepSeek separately cut input cache pricing across the lineup to one-tenth of previous levels as a permanent change.

The Test

This is the same FlowGraph spec we used in the Opus 4.7 vs Kimi K2.6 run, a workflow orchestration backend with 20 endpoints, persistent state, lease management, retries, and event streaming. It is a heavier infrastructure test than our usual coding benchmarks to push the models to their limits.

We ran DeepSeek V4 Pro and DeepSeek V4 Flash through the same setup to see where the new DeepSeek lineup lands on cost and first-pass quality next to Claude Opus 4.7 and Kimi K2.6.

The Prompt

We ran both DeepSeek models in Kilo CLI with the same prompt we used for Opus 4.7 and Kimi K2.6:

“Read @SPEC.md and build the project in the current directory. Treat @SPEC.md as the source of truth. Do not simplify this into a mock, toy app, or basic CRUD scaffold. Create all code, configuration, Prisma schema, tests, and README needed for a runnable project.…”

Both DeepSeek models ran on thinking mode in their own empty directories with no shared state.

What Each Model Produced

DeepSeek V4 Pro passed its own test suite but the TypeScript build failed. DeepSeek V4 Flash’s test suite never ran because its setup script tried to force-reset the database in a way that errored out before the first test executed.

If we had stopped at the model summaries, both DeepSeek implementations would look closer to Claude Opus 4.7 than they actually were. A direct code review plus targeted reproductions against isolated SQLite databases revealed the problems in both model outputs.

DeepSeek V4 Pro

DeepSeek V4 Pro got the broad shape of the system right. The endpoints are wired up, the test suite passes, and the project layout is reasonable. The issues we found are concentrated in the same places as Kimi K2.6: lease expiry handling, scheduling, validation, and build integrity.

Timed-out workers can still complete steps

When a worker claims a step, the system gives it a lease that expires after a set timeout. If the worker stalls or crashes, the lease should expire and another worker should be free to pick up the step. Once the lease has expired, the original worker is no longer the owner of that step and shouldn’t be able to mark it as done.

DeepSeek V4 Pro enforces this on heartbeats but not on completions. We claimed a step, pushed its lease expiry into the past, then asked the API to mark the step as successfully completed. The API returned 200 and recorded the step as succeeded. The original worker effectively reached past its expired lease and finalized work it no longer owned.

DeepSeek V4 Pro’s own README says workers cannot complete after their lease expires, but the implementation does not enforce that.

A full workflow blocks unrelated work

A workflow run can declare a maximum number of steps it is allowed to run in parallel. When that cap is reached, the saturated run shouldn’t accept more work, but other runs sharing the same queue should keep moving.

DeepSeek V4 Pro’s claim logic checks one candidate at a time. If that candidate happens to belong to a run that is already at its parallel cap, the function gives up and returns nothing, instead of moving on to the next candidate.

We reproduced this with two active runs sharing a queue. Run A was at its parallel limit. Run B had capacity and a higher-priority step ready to go. The next claim request came back empty. In production this would look like workers idling while there is real work to do, just because the first run on the queue happens to be saturated.

The project does not build

npm test passes but npm run build does not. Even after the build errors are fixed, the project still would not be runnable through npm start. The TypeScript config is set to not emit any compiled output, while package.json expects npm start to run that compiled output. A user following DeepSeek V4 Pro’s own README on a clean checkout would not get a working server.

DeepSeek V4 Flash

At $0.02 for the entire run, DeepSeek V4 Flash is in territory we have not tested before. The internal logic is plausible. The public API is where it falls apart.

Clients can’t start a workflow run

To use this system, a client first creates a workflow run by calling a specific endpoint. Without that endpoint working, nothing else can happen. There is no run for workers to claim from, no events to stream, no step to complete.

DeepSeek V4 Flash wrote the handler for this endpoint, but mounted it under the wrong route prefix. The spec requires it at /workflows/key/:key/runs. DeepSeek V4 Flash actually serves it at /runs/key/:key/runs. A request to the spec path against the running server returned 404 Endpoint not found. The README documents the spec path, but the server does not serve it.

DeepSeek V4 Flash’s tests call internal functions directly rather than going through the HTTP API. From the test suite’s perspective, everything was fine. From an actual client’s perspective, the entry point to the system was missing.

Failed workflows still hand out work

Once a workflow run fails (because one of its steps used up all its retry attempts), every other step in that run should stop. The spec calls for the remaining steps to move into a blocked state so workers will not pick them up.

DeepSeek V4 Flash’s recovery logic loads all expired steps at the start, then handles them one by one. If the first expired step exhausts its retries and fails the parent run, a later step in the same batch can still be promoted to a “ready to retry” state, even though the run it belongs to is already over.

We reproduced this with two expired steps in one run:

• Step a had no retry attempts left and was correctly marked dead

• The parent run was correctly marked failed

• Step b ended up in waiting_retry instead of blocked

A worker polling for new work would still receive step b and execute it for a workflow that had already failed. Claude Opus 4.7 had a related multi-expired lease bug. Kimi K2.6 missed live event streaming entirely. Recovery under contention keeps being the hardest part of this spec for any model to get right on the first pass.

Same timeout bug as DeepSeek V4 Pro

DeepSeek V4 Flash has the same expired-lease completion bug as DeepSeek V4 Pro. An expired lease can still finalize the work, even though the original worker no longer owns the step.

It also rejects valid request payloads. The spec says workflow run input and metadata can carry arbitrary JSON, which includes arrays, strings, and numbers. DeepSeek V4 Flash’s validation only accepts JSON objects. A client sending a JSON array as input would get a 400 response even though the spec accepts it.

Tool calling held up better than expected

The bugs above are about the output DeepSeek V4 Flash produced. Tool calling is a separate axis: how the model performed inside Kilo CLI. On that axis, the model held up surprisingly well. It read files before editing them, installed dependencies and ran the test suite at sensible points, and did not get stuck in retry loops on broken commands. The agent loop ran cleanly even when the code it produced had gaps.

That is not what we expected from a model at this price tier. Tool calling reliability is usually where cheaper models break down first, with malformed arguments, hallucinated file paths, or runaway loops that burn through tokens without making progress. DeepSeek V4 Flash avoided those failure modes in our run.

Scoring

We used the same 7-category rubric as the Opus vs Kimi post.

DeepSeek V4 Pro slots between Claude Opus 4.7 and Kimi K2.6. The gap with Opus is concentrated in build quality and lease handling. DeepSeek V4 Flash sits below Kimi K2.6, with deductions in nearly every category.

Cost vs Quality

DeepSeek V4 Flash’s cost per point is roughly 30x cheaper than Kimi K2.6 and 100x cheaper than Opus 4.7 on this benchmark. The score is lower, but the absolute dollar amount is so small that running the same task three or four times to compare attempts is still cheaper than one Kimi K2.6 run.

DeepSeek V4 Pro is more expensive than Kimi K2.6 in this run because we ran it before applying the official discount. With DeepSeek’s 75% promo applied to current rates, the same run would have cost closer to $0.55, putting it below Kimi K2.6 in absolute cost while scoring 9 points higher.

What This Means for Open-Weight Models

The pattern from previous comparisons keeps holding. The gap on surface coverage between open-weight and frontier proprietary is narrow. The gap on correctness inside hard code paths (lease recovery, cross-run scheduling, expired-lease rejection) is still there but also narrowing.

DeepSeek V4 Pro is the practical step up from Kimi K2.6 based on our test. Same general failure pattern, but cleaner overall structure and fewer spec-level gaps. With DeepSeek’s official discount in effect, the price gap with Kimi closes and the quality gap stays.

DeepSeek V4 Flash is a different conversation. At full price it is cheaper than the existing budget tier (Gemini 3.1 Flash Lite, Claude Haiku 4.5) by a wide margin. A 60/100 score on this spec is not a reason to use it on its own, but the cost is. For tasks where you can absorb a rough first pass and a human review, $0.02 per attempt changes the math considerably.

Our Takeaways

Claude Opus 4.7 still pulls ahead. The trickier parts of the spec (anything involving timing, recovery, or coordination between moving pieces) are where every other model lost points. Claude Opus 4.7 had only one reproducible bug while the other three had more.

DeepSeek V4 Pro performed better than Kimi K2.6 in this run. It scored 9 points higher, runs at a lower per-token list price, and produces about the same shape of failures under review. With DeepSeek’s official discount through May 31, the cost gap is even larger.

DeepSeek V4 Flash is a new category. It is not fully reliable for complex backend builds without a cleanup pass. But $0.02 for a first-pass attempt at a backend of this size is a price point that did not exist before. If you can absorb imperfect output, the math changes.

Both OpenAI and Anthropic are moving aggressively into consulting and enterprise deployment. OpenAI is reportedly building a massive “Deployment Company” alongside private equity firms and global consultancies, valued at $14 billion right out of the gate. Anthropic just launched its own enterprise services venture backed by Blackstone, Goldman Sachs, and Hellman & Friedman.

And Google recently announced a $750 million fund to help consultants adopt AI. Citing large firms such as Accenture and McKinsey, it’s a more measured approach than launching a full consultancy themselves, but it’s still no small play. The stakes are high and everybody wants a piece of the prize.

This is not a side story. It’s the clearest signal yet that AI adoption is not going to be won by APIs alone.

But what happens if the big players start offering consultative solutions that are really just designed for single vendor lock-in?

Everyone in AI is Now an AI Consultant (And That’s a Good Thing)

The model landscape now changes so fast that “best practices” can become outdated overnight. For the past two years, the dominant narrative around AI has been software margins, automation, and “replace labor with models.” But the frontier labs themselves are now acknowledging something important: AI adoption is still profoundly human work.

In practice, almost everyone serious about AI has become a consultant now. If you’re a builder, you’ve never been more valuable.

That includes founders. Engineers. Open-source maintainers. Partnership teams. Developer advocates. Even power users inside enterprises.

A surprising amount of modern AI work looks less like traditional SaaS deployment and more like a constant stream of config sessions, workflow redesigns, stack evaluations, and internal education. And that’s in addition to the strategy sessions.

When we run a KiloClaw configuration session with a developer team, the conversation rarely starts with a single model anymore. It starts with tradeoffs, or, more accurately, trade-ups.

Which model is best for coding right now? Which one handles long-context reasoning better? Which one is cheapest at scale? Which one works best for agents?

Whether or not they ask those kinds of questions will define whether these new consultants help you increase ROI and actually save money, not just max out token spend. Management consultants have always centered their value proposition on efficiency and cost-cutting, not just bigger spending, which means the focus on model optimization is a core part of the AI adoption playbook. McKinsey, for example, has long cited one of their key benefits as helping companies make cost saving continuous for years to come.

And this focus extends to internal work, too. Boston Consulting Group has internally deployed over 36,000 custom GPTs across its 32,000 consultants worldwide, signaling a massive effort to operationalize and optimize AI for knowledge work.

Claude Opus is immensely powerful. But it’s no secret that it can also get very expensive–very fast. Opus fits into key places in a workflow—we’re using it now in our Auto Frontier Model, which automatically picks the best model for the job within set criteria. But if you’re serious about optimizing ROI over time, you should be looking at a much broader spectrum of models and providers, including but not limited to OpenAI and Anthropic.

Just look at the recent PinchBench data on value scores for agent work. New models from Stepfun and DeepSeek are giving Claude Haiku a run for its money. They aren’t necessarily better but they might be the right fit for your daily agentic work.

The same thing happens in workshops with teams frustrated by rising Cursor pricing and exploring alternatives.

The opportunity is enormous. But so is the complexity.

Trading Vendor Lock-in for Smarter Workflows

The rise of AI consulting is good news. It means companies finally understand that “buying a model” is not the same thing as becoming AI-native.

The “big five” consultancies didn’t become giants because businesses lacked software. They became giants because transformation is messy, political, and operational. AI is no different — arguably it’s even more complex, in part because it’s touching on all of the systems as well as the models and agents that link them together. That’s the orchestration layer.

The smartest AI consulting firms won’t behave like reseller channels for a single lab. They’ll look more like intelligence infrastructure advisors. Part strategist, part systems integrator, part workflow architect, part educator.

Because, let’s be honest, they’ll be learning as much as you are. AI is moving so fast that it has to be developed in tandem with customers.

And increasingly, consultants’ value won’t come from access to, or understanding of, a particular model or model family. It will come from helping organizations navigate constant change without rebuilding their stack every six months. They’ll behave like strategic orchestrators across a constantly shifting model landscape.

That’s what we’ve been building for at Kilo—the option to always have the best model for the job right at your fingertips—and it’s also what our lab and inference partners are increasingly building for, as they release both strikingly efficient SOTA models (like GPT-5.5) and sneaky little powerhouses like the Xiaomi MiMo models and Ant Group’s Ling and Ring models, optimized for agentic tool calling. Flexible tools to fit any workflow, so that even as the model landscape evolves your workflows can travel with it.

The winners in the new AI consulting race won’t look like traditional consultancies. They’ll likely be:

• smaller

• faster

• deeply technical but highly creative

• open-source native

• model-agnostic

• agent-first

• and capable of shipping real systems instead of just decks

I’m not discounting the frontier labs. But the initial consultancy offerings they’ve been announcing will need to break down into micro-consultancies to be successful — in many ways mirroring what the frontier labs’ enterprise sales teams and “forward-deployed engineers” have already been doing. Every vertical counts.

The best consultants may not even call themselves consultants. They’ll look more like hybrid studios, deployment partners, infra operators, and ultimately the account managers at the AI tools in your stack.

That’s a healthier ecosystem. And for the consultees to succeed as much as the consultants, that ecosystem will have to be based on model freedom.

Before joining Kilo Code, Marius was already training engineering teams on working with generative AI. At Kilo, he does the same for enterprise clients in Kilo Speedruns: one-hour sessions designed to give teams a fast, practical orientation on agentic software development. He’s run them for companies across industries, and now he’s sharing the foundations of those lessons (and his specific practices for each) here:

• How to delegate effectively

• How to scale across concurrent workstreams

• How to maintain judgment and recover when things go wrong

1. How to Delegate: The Team Lead Model and the Plan

The mental model Marius uses to explain agentic engineering—both in client speedruns and in how he structures his own work—is the team lead.

Team leads don’t spend all day writing code, and the same was true even before agentic tools existed. They were in pairing sessions, answering questions, reviewing output, and deciding what to merge. “Those were always the people that were only in meetings and they got called by all the juniors,” Marius says. “They were just solving the last 20% of the problem.”

In this model, the agent takes care of execution work, while the engineer operates as the team lead. The 80% that agents handle well—code generation, boilerplate, well-scoped subtasks—is work that the team lead delegates. The 20% that still requires the engineer is the judgment work: architectural decisions, what to merge, and recognizing when the agent has drifted.

The engineers who transition most naturally into agentic workflows are often the ones who were already operating this way: team leads and architects who had developed the habit of switching contexts and reviewing output rather than writing it. Everyone else has to learn that mode of working, which starts with understanding the difference between a specification and a plan.

A specification captures what the user wants. It doesn’t change based on the current state of the codebase. It’s set from the user demand, and it stays set. A plan is how you intend to build the thing given where the code actually is right now. “A plan is dependent on your state of the code,” says Marius. “Plans usually get thrown away very quickly.”

When Marius works with an agent on complex tasks (especially those with important architecture decisions), he asks it to write its plan to a markdown file before it starts executing so he can review it. Asking the agent to write its plan first forces a shared understanding of what’s actually being built. You review it, ask questions, and surface problems before execution begins. It’s the refinement stage of traditional software engineering, but the difference now is that the feedback loop is much faster.

Plans, done right, function as constraints. Marius thinks of this as keeping an agent in the acceptable solution space: the set of outputs you will actually accept. The further an agent drifts from a confirmed plan, the more likely it ends up somewhere that requires starting over. Forcing the plan upfront dramatically increases the probability of staying on track.

The plan also acts as a contract: it documents the approach the agent intends to take, so when it does something unexpected later, there’s a reference point. “You can always reiterate to the agent, ‘We decided to implement this plan. Why have you decided otherwise?’”

2. How to Scale: Parallelism and the Context Rot Problem

Even with a solid plan in place, there’s a natural limit to how far a single agent session can take you: context rot. As a session grows, accumulating conversation history, prior decisions, and intermediate code states, the agent starts losing coherence. Tasks that were reasonable at the start become unpredictable midway through. Early decisions can come back to bite you. At some point, recovery means starting over.

Most engineers treat this as a nuisance and work around it by brute force: shorter sessions, more restarts. Marius treats it as a signal that the work hasn’t been decomposed correctly. “If you have a huge feature and you develop on it for the whole week, you will keep having context rot,” Marius says. “It makes much more sense to plan out what you want to implement ahead of time and then develop each of the sub-problems individually in small context windows.”

This is where parallelism comes in: you run multiple agents simultaneously, each working on a specific sub-problem. But parallel agents writing to the same file system will conflict (the same reason Git was invented). You need each agent working in its own isolated environment.

To address this, Marius built a solution into his own custom IDE, before building Kilo’s Agent Manager: a tool for running multiple agent sessions simultaneously, each in its own isolated workspace, with its own file system. Instead of supervising agents one at a time, an engineer can delegate across several concurrent workstreams and review the results as they come in. Things that look good get merged; things that don’t get discarded without the cost of untangling a week of compounded decisions.

Not every task demands the multi-agent treatment. Marius works across three categories depending on complexity:

Easy tasks: Things like adding documentation, writing a unit test, or well-scoped bug fixes go to a fully autonomous cloud workflow. The developer writes the spec, the agent executes, the developer reviews the diff. No supervision is required mid-session.

Hard tasks: Implementing a complex feature spanning UI and backend, or anything with meaningful architectural decisions, gets handled locally with Agent Manager. The developer supervises multiple agents working in parallel on decomposed subtasks, stays close to the work, and makes the judgment calls as diffs come in.

Unclear tasks: When the outcome isn’t well-defined, it’s hard to write a spec precise enough to constrain the agent toward a single solution. For these, Marius runs multiple agents in parallel against the same spec and compares the results. Instead of splitting work, the parallelism here is about generating variants and selecting the best one. The engineer’s job is choosing the right route.

3. How to Stay on Track: Context Engineering and Judgment

Context engineering, as Marius defines it, is how you structure and optimize the context of the agent. The goal is to limit an agent to doing exactly what you want, over time, in your codebase. It’s the ongoing work of keeping agents oriented, and knowing how to reorient them when they’ve drifted.

For upfront orientation, Marius uses Handy, a speech-to-text tool, to interact with agents verbally before locking in a plan. A lot of the context that matters for a task lives in the engineer’s head and never gets written down, because it’s too tedious to type everything out. Speaking it aloud removes that barrier, and an LLM can distil the rough transcript into a precise problem statement. The rough transcript also becomes the raw material for the plan the agent writes before executing.

When an agent session ends—whether it hit a context limit or simply reached a natural stopping point—continuing the work is usually straightforward. The original prompts, the Git diff (Agent Manager measures the delta from when the session started), and the current state of the codebase give a new agent enough to pick up where the previous one left off. Tools like Repomix can help with collecting specific file trees for this purpose. All of this can happen locally or in GitHub, where an issue describes the task, the PR contains the changes, and the history provides the thread. Most agents can continue from that context without much intervention.

What this process makes visible is what’s actually irreplaceable: the context that isn’t captured anywhere. Code and prompts are always an approximation—there are causal relationships in software that are hard to capture in prompts or code alone. Some of them, like another team’s architectural decision creating a dependency you didn’t know about, can be surfaced and handed off. Others only become visible when you run the code or at scale. An agent can’t know what hasn’t surfaced yet—that’s still the engineer’s job.

This is the difference between just coding and software engineering. The easy mistake with agentic work is treating it as a handoff: you describe what you want, the agent builds it, you ship it. In that approach, the critical last 20% can get lost: things like evaluating architectural choices and catching when an agent has veered off course. These require engineering judgment, and they’re often much harder than the first 80%.

The mental shift Marius describes is about learning to apply engineering judgment at the right moments, across multiple concurrent threads, rather than sequentially inside a single one.

Read the other posts in our Kilo Speed series:

I can still draw the architecture of code I wrote three years ago from memory. The data flow, the edge cases, the reasoning behind every choice that looks strange at first glance. Ask me to do the same for a feature I shipped last month with help from an agent, and I can tell you what it does and why we built it. The deeper model, the one that lives at the level of individual functions, isn’t there.

That’s not laziness, and it’s not a lapse in review. I read every diff. An agent does a closer pass alongside me. I can speak to the intent and shape of what I’m approving. But the deep mental model, the one you actually need at 2am when something breaks and the agent isn’t helping you debug, isn’t forming the way it used to.

I’m a CISO who still writes code, and this worries me. It should worry anyone managing engineers right now, because it isn’t just me. Across the industry, AI coding agents are quietly reviving the single worst antipattern in software engineering history. We just don’t recognize it yet, because it’s wearing different clothes.

Remember the cowboy?

If you’ve managed engineers long enough, you know the cowboy. The one who disappears for a weekend and comes back Monday with a full rewrite nobody asked for. The one who, somehow, is the only person who understands the gnarly billing module, the auth flow, the deployment pipeline. The one whose decisions land in production faster than the team can review them.

Cowboys aren’t heroes, by the way. The hero is the engineer who pulls the 2am save when production breaks. The cowboy is the one who created the conditions that made the 2am save necessary in the first place. Heroes clean up. Cowboys cause.

For twenty years, our industry has been quietly learning how to build engineering organizations that don’t depend on this person. Code review. Pair programming. Design docs and RFCs. Collective code ownership. Postmortems that look at process, not blame. The whole inheritance from XP, agile, and DevOps was, in large part, a response to the lesson that cowboy culture feels productive and is actually corrosive.

It worked. Not perfectly, but the average engineering team today is far more resilient than the average team in 2005.

Then the agents arrived.

Watch what happens on teams that have adopted Claude, Cursor, Copilot, Codex, and the rest without changing how they work. An engineer prompts an agent. The agent emits eight hundred lines of code. The engineer skims it, sees the tests pass, and merges. Repeat, ten times a day, across the team.

The output is enormous. The velocity charts look incredible. And underneath, something is going wrong that nobody is naming yet.

Nobody on the team has reasoned through that code. The “author” couldn’t walk you through it under questioning. They didn’t write it, they prompted it. The reviewer couldn’t either; they had thirty other PRs in the queue, and half the time the reviewer is another agent. Six months from now, when something breaks at 2am, the engineer who gets paged will be debugging code that has, in any meaningful sense, no human author at all.

This is the cowboy pattern. The weekend rewrite, the opaque module, the knowledge silo, the tech debt nobody quite remembers creating. Same antipattern, new substrate.

Why it’s actually worse

The cowboy archetype, for all its damage, had one redeeming feature: somewhere, in one human brain, the model of the system existed. Bus factor of one.

Development driven by agents, without comprehension, produces bus factor zero. The code enters the repository with nobody understanding it. There is no expert to consult, because the “expert” was a probability distribution that has since moved on to the next prompt.

The social brakes that used to slow cowboys down are also gone. Cowboys had egos, reputations, and peers who could push back in code review. Agents have none of these. They don’t sulk when overruled, don’t take credit, don’t feel shame when prod breaks. The friction that used to make cowboy culture limit itself in healthy teams, the simple fact that other humans were watching, is absent.

And there’s a new accountability sink. When the cowboy shipped a bad rewrite, you knew whose desk to visit. When an agent ships a bad rewrite, the conversation goes “well, the AI wrote it” and everyone shrugs. The blame diffuses into the tooling.

What managers should do now

The good news: the playbook for fixing this already exists. We wrote it the last time. It needs updating, not reinventing.

Require comprehension, not just approval. Before any meaningful PR written with an agent gets merged, the author should be able to walk through it without asking the agent again. If they can’t explain why a function exists, the PR isn’t ready. This is the most impactful change you can make, and the one I’d benefit from most personally.

Cap PR size, hard. Code review evolved assuming limited human throughput on both sides. Agents broke that assumption. A PR of 50 lines can be meaningfully reviewed; a PR of 800 lines gets approved without thought. Set a limit, enforce it in tooling, and force large changes to be decomposed.

Tag agent involvement and track it. Make AI authorship a first class piece of metadata on commits and PRs. Watch incident rates, time to debug, and refactor cost on modules where agents wrote most of the code, and compare against the rest. You can’t manage what you can’t see, and right now most engineering orgs are flying blind on this.

Protect the loop of deliberate practice. Junior engineers who never struggle through a hard bug don’t become senior engineers who can debug under pressure. Build in rotations without agents, pair on hard problems, and make “can debug from scratch” part of your leveling criteria. The seniors riding herd on agents today learned their craft the hard way. The next cohort needs a path to the same skill, or you’ll wake up in five years with a team that can prompt fluently and reason about nothing.

Reframe tech debt as unread code. The most dangerous code in your repository is no longer the bad code. It’s the unread code, modules that work today and that nobody on the team has actually internalized. Schedule comprehension audits. Assign engineers to read and document modules written by agents that they didn’t author themselves. Treat unread code as a liability on the books.

This is not an argument against AI

The agents are useful. The productivity gains are real. I use them every day, and I’m not giving them up.

The point is that the technical productivity of these tools is arriving faster than the organizational practices needed to absorb them. The teams that already had healthy engineering culture, the kind with code review that actually reviews, sustainable pace, and collective ownership, will adapt and thrive. The teams that quietly tolerated cowboys are about to have a much worse problem, at much greater scale, with no single person to point at.

And the rest of us, the ones who can still picture the flow of code we wrote three years ago but no longer build that same depth of model with the new stuff, need to be honest that the muscle is atrophying. Mine is. Yours probably is too.

The cowboy didn’t go away. The cowboy scaled, with a million tokens of context. The work of engineering management is to recognize the pattern in its new form and apply the lessons we already learned the last time.

(”oh s***” obviously means “oh snap” … what did you think it was?)

Today we’re shipping PinchBench 2.0. Here’s what changed and why.

The problems with v1

V1 worked by running 23 tasks, grading them, and giving you a score. But it had shortcomings that became obvious as usage scaled:

Scoring was gameable. The leaderboard ranked by mean score across completed tasks—but didn’t account for how many tasks you ran. An agent that cherry-picked 1 easy task and scored 100% would outrank one that ran all 23 and scored 94.8%. Broken.

At least one task was basically impossible. A 95% failure rate across 90% of models isn’t testing capability—it’s wasting compute. We fixed or pulled those.

Version tracking was opaque. We used git commit hashes to identify benchmark versions. Users saw strings like a1b2c3d in the version dropdown and had no idea what changed between them.

Race conditions in grading. The benchmark assumed that when the OpenClaw agent subprocess returned, the transcript was complete. It wasn’t always. Grading could start on a partial transcript, producing silently wrong scores.

148 tasks (up from 23)

We ran a gap analysis comparing what PinchBench tested against what real OpenClaw users actually do (based on 780+ ClawBytes on kilo.ai/kiloclaw). Then we filled the gaps:

• Data analysis — CSV tasks covering US cities, iris flowers, temperature, life expectancy, weather stations, GDP, pension funds, Apple stock

• Meeting & document analysis — Government meetings, advisory boards, city council, tech meetings; executive summaries, sentiment analysis, action items, Q&A extraction

• Log analysis — 24 new tasks across NGINX, Apache, SSH, HDFS, MapReduce, and syslog

• Development & DevOps — CI/CD debugging, Kubernetes issues, Dockerfile optimization, multi-file refactoring, test generation, commit message writing, git rescue, shell generation

• Image & PDF — Image identification, PDF to calendar import

• Research & writing — Market research, email drafting, Todoist cleanup, contract/legal analysis

The call for contributors we opened in March brought in proposals from community members building browser automation tasks, test generation scenarios, and more. 111 commits from the community landed between v1.2.1 and v2.0.0.

Parallel judge execution

Grading no longer waits for all tasks to finish. The judge now overlaps with task execution, so your benchmark run completes faster. We also switched to Haiku as the default judge — faster grading without sacrificing accuracy — and added judge result caching so re-runs don’t redundantly re-grade unchanged results.

Thinking-level support

v2 supports testing across different reasoning/thinking levels and reporting scores for each. A model’s performance at “low” thinking versus “high” thinking tells you something different than a single aggregate number. For cost-conscious users choosing between reasoning modes, this data matters.

Multi-turn session isolation

Tasks can now specify new_session: true for proper multi-turn evaluation. This means benchmarking conversational workflows (where context carries across turns) without session bleed between tasks.

Semantic versioning

We migrated from git hashes to proper semver across the entire stack. The version now comes from GitHub releases via setuptools-scm for pip installs and a BENCHMARK_VERSION file for everyone else. The leaderboard shows 2.0.0 instead of 7df28f6. All existing git-hash versions got backfilled as 1.0.0-beta.N so sorting still works. I implemented it all with Gastown from Kilo.

Leaderboard overhaul

Scoring fairness. Runs are normalized by task count so comprehensive runs are rewarded, not penalized.

Per-task variance and retry counts. A model scoring 0.85 with std dev 0.05 is more useful than one scoring 0.90 with std dev 0.25. Consistency scores, first-try success rates, and per-run breakdowns are now surfaced.

Model landing pages. Each model gets its own page with all submissions, score trends over time, cost and speed metrics.

User profiles and contributor recognition. A contributor leaderboard alongside the model one.

Better filtering. Categorized provider filters, search, checkbox selections, and task-level filtering so you can compare models on just the tasks you care about.

Badges. Daily, weekly, and monthly recognition for top-performing models, plus overall rank badges for the top 10.

Infrastructure improvements

• --core flag for running a core task subset when you don’t need all 148

• --trend and --trend-window flags for post-run analysis

• Manifest-based task ordering (replaced numbered IDs)

• CI workflow for manifest linting

• Better OpenClaw transcript compatibility

• Axiom observability for detailed logging of benchmark runs

Breaking changes

• Task IDs now use manifest-based ordering instead of numbered prefixes

• Default judge backend changed from openclaw to api

Get started

Visit the GitHub repo to see the tasks, run the benchmark yourself, and contribute.

The leaderboard is live with the new scoring, filtering, and model pages. If you’ve been using PinchBench results to pick your model, you now get variance, per-task breakdowns, and cost data alongside raw scores.

Full changelog: v1.2.1...v2.0.0

Ida Infront builds secure communication systems, case management tools, and digital archiving solutions for customers including national agencies with demanding data handling requirements and long records retention obligations. The company was established in 1984, and the platform underlying everything they build is over 25 years old.

In these environments, compliance concerns, risk analysis, and legacy infrastructure can hamstring experimentation.

Magnus Grimsell, CTO at Ida Infront, knows that embracing AI is how the company will stay competitive: “The value of a large, established platform can decrease when new products can be written faster and cheaper,” he says. “We want to address that risk and take advantage of the opportunity by being early adopters.”

So the company has been on a structured AI journey since GPT 3.5 launched. They started with Sourcegraph’s Cody, but within six months they were already looking beyond coding assistants to the next frontier: agentic engineering.

After evaluating several tools, including Cline and Roo Code, Ida Infront chose Kilo Code and began rolling it out in January of this year, starting with three forerunner teams and 70 seats across developers and other technical roles.

The Rollout Plan Is Just as Critical as the Choice of Tool

Magnus has watched peer companies in the industry tell developers they can use whatever AI tools they want, then wait. “Nothing happens,” he says. “There’s no progression in AI adoption, no next step.” Random experimentation isn’t good enough; without standardizing on anything, you won’t get real results.

So Ida Infront did the opposite: they selected one tool, committed to it organizationally, and built a structured onboarding process to bring every team through it together.

Each pilot team goes through a series of three workshops, developed in close partnership with the Kilo team:

1. The first covers agentic engineering fundamentals, with the team mob-programming a real backlog item using Kilo. Before the session, a team ambassador—an AI champion appointed from each team—has already prepared the repo with an AGENTS.md file and relevant MCP tools.

2. The second workshop, held 1-2 weeks later, addresses what came up in practice: debugging issues, adding skills to the agent, connecting new MCP servers, refining workflow.

3. The third is non-technical. By this point the team has been using Kilo in production for about four weeks. The session walks through the four phases anticipated in Ida Infront’s AI adoption framework: AI Assistant, AI Agents, Multi-agent, and Software Factory. Teams discuss where they think they are, what they’re expecting, and what concerns them. There’s a structured exercise for surfacing fears and hopes.

“We had contact with the Kilo team throughout the first pilot,” Magnus says. “We didn’t just receive support for the product, but expertise that I’ve valued as well. It’s great to have somebody come in and say, ‘This is how we do agentic engineering.’”

The staggered rollout lets the company train their future AI trainers: the early adopter teams will then be able to coach others.

Early Indicators

Magnus describes a full spectrum of reactions to the agentic shift: excitement, skepticism, and a form of grief from developers who feel that the craft they’ve spent years developing is changing in ways they didn’t ask for. The final workshop is an opportunity for team members to feel heard and work through any resistance. “We have people who are on board but feel sad about it,” he says. “That’s a legitimate response. It needed space.”

Since the rollout began, the dominant attitude has been positive: teams are eager to get onboarded. Ida Infront’s developers who had been using Cody heavily were prepared for an incremental improvement when they saw Kilo Code in action for the first time. “There were some dropped jaws,” Magnus says. “With an assistant, you’re working the same way as before, just a little faster. With agents, the developer role itself starts to change.”

That shift is already visible in how Ida Infront is thinking about the future of their teams. Specialized agents for performance optimization or domain-specific tasks—filling skill gaps at the team level rather than requiring escalation to in-house experts—are on Magnus’s roadmap. Other companies in Ida Infront’s parent group are already looking to replicate their model.

The Proof Is Already Showing up in Their Products

Ida Infront is also building AI into the products they deliver to customers. ARN, the Swedish National Board for Consumer Disputes, introduced an AI-based function developed with Ida Infront in 2025 that automatically classifies incoming case documents like receipts, images, and correspondence—tasks previously handled manually. ARN’s summary of 2025 noted that AI investment freed staff from manual classification work and contributed to significantly shorter case processing times—an outcome Ida Infront is working to replicate across more of its customer base.

There’s a sense of excitement and possibility at the company. “I’ve been through the shift from waterfall to agile. That changed everything—not just how we wrote code, but how teams were organized, what roles existed, how we thought about delivery,” Magnus says. “This is the same kind of shift. It affects everything. And I’d rather be shaping what that looks like than waiting to find out.”

Most of this work isn’t hard if you get the concept of the ads and how they are operated. It’s just tedious and error-prone. Renaming 40 ad groups in the Google Ads Editor or building a new campaign from a spec doesn’t take a week – it takes maybe an hour of clicking the same buttons in a slightly different order. But during that hour you can easily pin the wrong headline, paste the wrong URL into three ad groups, or forget to attach the sitelinks. Prompting an agent to do the same thing takes a few minutes and the output is consistent by construction. And in some cases (especially in the AI space), agents help to quickly jump on the trend and launch ads in a few minutes with a couple of prompts.

This is a writeup of what I built, how it’s structured, and how I actually use it.

Why Kilo CLI and not just one big agent

Kilo CLI lets me pick a different model per task. That’s the whole thing. I’m not stuck paying Opus prices to rename an ad group. If you are managing a dozen accounts for your clients, you could face a huge bill. Or you do not want to wait for a 5-hour rate limit to reset to get your tasks done before the sunset.

My usual setup:

• Planning and strategy. Opus 4.6 or 4.7, or GPT 5.5 at high thinking. Writing a new campaign spec, arguing about the structure, deciding whether three comparison campaigns should share a landing page. This is where I want the expensive brain.

• Execution. Sonnet 4.6, or Kimi K2.6. Once the plan is locked, the actual work is “read the skill, follow the pattern, write the Python, run dry-run, show me the diff.” That’s mechanical. Cheaper models do it fine and they do it faster.

The big picture

Here’s roughly what the whole thing looks like on GitHub:

~/paid-ads/
├── .kilocode/
│   ├── rules/                       ← always-on guardrails
│   │   ├── safety.md                  never --execute without --dry-run,
│   │   │                              never touch budgets without asking,
│   │   │                              never commit .env, etc.
│   │   └── main.md                    workflow: read ROADMAP, read todo,
│   │                                  update CHANGELOG, update skill
│   └── skills/                      ← loaded on demand
│       ├── google-ads-scripts/        the script library skill
│       ├── pmax-sqr-audit/            PMax search terms audit workflow
│       └── weekly-report/             Monday report workflow
│
├── scripts/                         ← ~40 Python scripts (the muscle)
│   ├── export_*.py                    pull data into JSON
│   ├── create_*.py                    build new campaigns end-to-end
│   ├── update_*.py                    mutate live entities
│   ├── fix_*.py                       audit + fix (tracking, URLs, naming)
│   ├── add_*.py                       append keywords, sitelinks, negatives
│   ├── audit_*.py                     read-only analysis, no mutations
│   ├── weekly_report.py               Monday sync generator
│   └── sync_campaigns_md.py           regenerate inventory pre-push
│
├── reports/                         ← auto-saved, dated (2026-04-20.md, …)
├── docs/                            ← landing-page intent maps, SOPs
├── tasks/
│   ├── todo.md                        active backlog
│   └── specs/                         written before multi-file work
├── CHANGELOG.md                     ← one line per change, automated
├── ROADMAP.md                       ← strategic priorities
└── campaigns.md                     ← regenerated from the API pre-push

Rules are always on. They’re the “don’t shoot yourself in the foot” layer. Skills load on demand when I’m doing a specific kind of work. The scripts are where the actual API calls live.

A note on “topic”

One piece of vocabulary that shows up a lot below: topic. A topic is a slice of the campaign naming convention that groups related campaigns regardless of geo or channel. My campaign names look like <scope>_<stage>_acq_<topic>_<channel>_<goal>, so a topic might be product-personas (persona landing pages), competitors (competitor targeting), brand (defensive brand search), and so on. One topic usually spans multiple campaigns – a core campaign for Tier 1 countries, a global campaign for the rest, sometimes a PMax variant on top. When the weekly report is grouped by topic, it’s rolling up those parallel campaigns into one row so I can see the whole personal effort or the whole competitor effort in one number.

The three skills

1. google-ads-scripts – the library

This is the backbone. It’s basically a README that lists every existing script, what it does, which patterns it demonstrates, and the conventions any new script has to follow. When I ask for something new, the agent reads this first and copies the closest existing pattern instead of inventing its own.

The conventions that matter:

• Every mutating script ships with --dry-run (default) and --execute. Dry-run prints every operation it would perform, with enough detail to eyeball a diff. Nothing hits the API until I pass --execute.

• New campaigns are always created PAUSED. Enabling is a deliberate UI step.

• Every creation script supports REUSE_CAMPAIGN_ID so a half-finished run can resume. Rate limits and transient API errors do happen on large batches – a 751-ad-group campaign creation will occasionally fail halfway, and restarting from scratch is not an option.

• Keyword/negative adds always dedupe against what’s already there. No one wants duplicate-keyword warnings.

Here’s what a fresh campaign-creation dry-run looks like in practice. Say I want a new comparison campaign:

I read through it, push back on maybe one headline, run --execute, check the UI, enable manually. The same campaign in the Google Ads Editor is maybe an hour of clicking – not hard, just slow, and every field is a chance to paste the wrong URL or forget a pin. The agent does the same work in a few minutes and the output is consistent.

The other example worth showing is the tracking template audit. Campaigns drift – I renamed one, the utm_campaign param in the tracking template now points to a dead name, and the analytics dashboard is quietly attributing spend to a campaign that was renamed six months ago (I prefer to use actual names instead of IDs). The kind of thing nobody wants to audit by hand because it means opening every single campaign in the UI and comparing two strings.

One script walks the whole account:

$ python scripts/fix_tracking_templates.py --dry-run

[init]  Found 31 campaigns (24 Search, 4 PMax, 3 Display)

─── Search campaigns ────────────────────────────────
  brand_search_ww                           OK
  competitors_search_core            ⚠ FIX utm_campaign mismatch
                                            was:  competitors_search_create-instance
                                            want: competitors_search_core
  personas_search_core               ⚠ FIX param order
  category_search_apac               ⚠ FIX {keyword} unsupported in Search
                                            use {_keyword} instead
  ... (21 more)

  Search: 7 need fixes, 17 already OK

─── Performance Max campaigns ───────────────────────
  pmax_core       ⚠ FIX final_url_suffix – utm_campaign stale
                  ⚠ FIX tracking_url_template must be {lpurl} not ""
                    (API rejects empty string: STRING_TOO_SHORT)
  pmax_global     ⚠ FIX utm_content references old asset group name
  pmax_br             OK

─── Summary ─────────────────────────────────────────
  9 fixes, 22 already OK → 9 mutations

That “tracking_url_template must be {lpurl}, not empty string” line is a real API quirk – clearing the field with an empty string returns STRING_TOO_SHORT, which is not obvious from the docs. Once a session figures it out, the fix goes into the skill, and the next session never hits the same wall. These little footguns are the whole reason skills compound over time.

2. pmax-sqr-audit – turning the Performance Max search terms report into an action plan

PMax is a bit of a black box unless you pull the search terms export and actually read it. Which nobody wants to do, because it’s a 3,000-row CSV with two decorative header rows that break most parsers.

This skill codifies the whole workflow. Parse the CSVs, cross-reference against live Search keywords to find cannibalisation, bucket irrelevant terms into categories (video, gaming, trading bots, resume builders, wrong products…), theme the high-spend zero-trial terms (model-name queries, unknown competitors, brand misspellings), and decide where each thing goes:

• Brand misspellings → brand campaign, and negative in PMax

• Unknown competitors → new ad group in the competitors campaign, and negative in PMax

• Category-generic stuff → category campaign, and negative in PMax

• Irrelevant → just a PMax negative, no routing

There’s one invariant I’ve written in bold caps in the skill: every keyword added to a Search campaign also becomes a PMax EXACT negative. Otherwise PMax will keep eating the targeted Search campaigns’ traffic at a higher CPC, and you pay twice for the same conversion.

Rough shape of the output when I run it on a recent export:

3. weekly-report – the Monday sync, mostly automated

Every Monday the paid team does a sync where we look at last week’s numbers. Cost by topic, CPA vs last week, daily trend, top personas, the monetisation funnel. Pulling all of that out of the Google Ads UI, exporting it to a spreadsheet, and reformatting it into the team’s template is the kind of task that’s not hard but also not fun – and easy to mess up a column here or there.

The report script produces everything that’s just data, and leaves the commentary to me. That split matters: I don’t trust any LLM to write the “why did this spike” paragraph from numbers alone. But the numbers themselves are a mechanical aggregation, and mechanical work is exactly what scripts are for.

The skill is mostly about the landmines. Stuff like:

• Our trial KPI is a secondary conversion action, so it only appears in metrics.all_conversions, not metrics.conversions. Miss that and your CPA is off. In caps at the top of the skill.

• The monetisation section has to query all campaigns including non-core ones, because our general brand search campaign is weirdly the second-best revenue producer. If you filter too tight you miss where money actually comes from.

• segments.conversion_action_name can’t coexist with metrics.cost_micros in the same GAQL query for search_term_view. Two queries, merge in Python.

Here’s an anonymised chunk of what the output looks like:

The Notes column is auto-generated from rules (”CPA spike”, “Zero trials”, “Budget-capped”, “New campaign”). Those are signals, not conclusions – my job on Monday morning is to pick the ones that matter and write the real story underneath.

What the script never writes, and I don’t let it write: the TL;DR, the monetisation commentary, and the “what we’re going to do about it” section. Numbers auto, judgement manual. Pretending otherwise is how you end up with confident-looking reports that miss the actual story.The Notes column is auto-generated from rules (”CPA spike”, “Zero trials”, “Budget-capped”, “New campaign”). Those are signals, not conclusions – my job on Monday morning is to pick the ones that matter and write the real story underneath.

What the script never writes, and I don’t let it write: the TL;DR, the monetisation commentary, and the “what we’re going to do about it” section. Numbers auto, judgement manual. Pretending otherwise is how you end up with confident-looking reports that miss the actual story.

The self-improvement engine

The rules file says: whenever the agent writes a new script, it must add an entry to the corresponding skill describing what the script does and which pattern it demonstrates. If it discovers a new API quirk, that goes in the skill too.

So the skills grow by themselves. Each session teaches the next session something new. The STRING_TOO_SHORT note on tracking_url_template, the secondary-conversion-metric caveat, the PMax channel performance – none of those were written by me up front. They were all discovered mid-task and then documented as part of the task.

The other automated piece is action logging. Every change lands in CHANGELOG.md as a one-liner with the why, not just the what. tasks/todo.md gets updated when work finishes or when follow-ups surface. campaigns.md regenerates from the live API before every git push so the remote always has the current campaign list. None of that is me remembering to do it; it’s all in the rules.

The result is that the repo is self-describing. What’s live, what’s paused, what’s planned, what has changed recently, why. Coming back after a week or two away doesn’t require any mental reconstruction – the state is on disk.

What actually changed

The time savings are real. But more wins are elsewhere:

• Consistency. Every campaign gets the same naming, the same RSA structure, the same tracking template, the same geo split. No drift.

• Reviewability. Dry-run output is a text diff. Five minutes of reading catches things a UI change would never surface.

• Recall. API footguns and account-specific rules stay caught. Nobody has to remember “did we already fix that” because the skill remembers.

• Cost. Cheaper models handle the bulk of execution. Opus-tier reasoning stays reserved for plans and strategy where it earns its keep.

• Context. A new session catches up in about 30 seconds because the skills, the CHANGELOG, and the todo file describe the current state.

Total skill content is maybe 600 lines of markdown. It’s worth more than all the Python in the repo combined, because the markdown is what makes the Python safe to generate from a prompt in the first place.

If you want to build your own

A few notes for anyone thinking of going the same direction:

1. A skill is a reference document, not a prompt. Write it for a future session that has forgotten everything. The agent reads it the same way.

2. Encode the conventions you actually enforce. If half the scripts have dry-run defaults and half don’t, fix the scripts first, then document it.

3. Make the skill self-maintaining. Put “every new script MUST be added to the registry below” at the top, and the maintenance takes care of itself.

4. Safety rails go in the rules file. Never commit .env, never skip dry-run, never touch budgets without asking, no --force push to main. Spell it out.

5. Every API quirk you discover is worth a paragraph. That’s where the compounding happens.

6. Use cheaper models for execution. Save the expensive ones for planning.

The setup took a few hours – not a big investment for something that runs continuously.

The workflow usually looks like this:

1. You submit a prompt to a text-based model.

2. The model returns an HTML page.

3. You go back and forth with the model to make revisions.

After OpenAI released ChatGPT Images 2.0, I’ve seen more people shift to a different workflow:

1. You submit a prompt to an image-based model.

2. The model returns an image.

3. You iterate on the image until you get a design you like.

4. You ask the model to convert that image into an HTML file, whether that’s static HTML, React, or whatever stack you use.

This approach looked interesting, so I decided to test it.

The test

I asked GPT-5.5 using ChatGPT to create 3 different pages for me. I used a different session for each request.

• A landing page for an AI coding tool. The Kilo Code landing page already looks good, but I thought, why not?

• A landing page for a SaaS tool for lawyers.

• A website for an e-commerce store selling shoes.

Important: I kept the prompts generic on purpose. I didn’t paste a 10-page prompt with detailed specs because that’s not how most of us interact with AI and I wanted to model more realistic scenarios.

Let’s dive into the results.

A Landing Page for an AI Coding Tool

Prompt: “Let’s create a modern landing page image for an AI coding tool.”

I entered this prompt twice and got two different designs. I didn’t really like the first one, but the second looked decent:

Let’s see if we can translate this into an HTML page.

Follow-up prompt: “Can we turn this into a single HTML page?”

Here’s a preview of the HTML:

Not bad! The page also looked decent on mobile:

This was impressive. I tried to one-shot everything just to see how the model would behave.

This is not the most unique landing page in the world, but it’s a good starting point.

SaaS Landing Page for Lawyers

Prompt: “Let’s create a modern landing page image for a SaaS management software for lawyers”

The first result wasn’t very encouraging:

My theory on why this was the case: I asked it for a “landing page” (that’s associated with something modern) + mentioned ‘lawyers’ and the end result was a weird mix between modern and more traditional design style.

I tried a follow-up prompt to see if the AI could come up with something better.

Follow-up prompt: “I see you’re trying to mix modern elements from a landing page with traditional elements from a lawyer page. Make the whole page feel like a lawyer’s website, including the fonts, colors, and overall style.”

Ugh. Still not there.

I think the reason is that the AI model likely has less training data for “legacy” or more traditional websites.

The lesson: you may need to be more specific when creating websites in categories where there’s less strong design data available. Load skills, ask the AI to analyze websites you like, and have it create a detailed design spec before generating anything.

An E-Commerce Website Selling Shoes

Prompt: “Create a website image for an e-commerce website selling shoes”

The result was something I almost liked:

These images look too wide.

Follow-up prompt: “The images look too wide, can you fix that?”

This result more closely resembled a modern e-commerce website:

Let’s try to see if we can convert this design to a webpage. The prompt I used was: “let’s convert this to a single HTML page (include all CSS/JS inside that HTML)”

The result was close to what I wanted:

I tried a few prompts to get the images to align properly, but I couldn’t get it to work.

Then I had an idea:

What if I take a screenshot of the image, and ask the AI to look at that picture, figure out what’s wrong and fix what’s wrong on its own?

This actually worked! GPT-5.5 figured out the problems:

And it actually fixed the page:

This is a preview of the actual (fixed) HTML page.

Now that’s something that looks better.

Using AI to Design Websites Is Not the Same as Using AI to Code

We’ve seen that using AI to design beautiful pages requires a different approach than using AI to generate code. That’s why we created App Builder, for example:

Anthropic recently released Claude Design, and OpenAI released ChatGPT Images 2.0. Both are changing the workflow for how design gets made.

We’re excited about the future of AI + design and can’t wait to see what it unlocks and what kinds of products we can build around it.

But after running both for the past two months, I think the feature checklists are a distraction — the design philosophies are where they actually diverge.

The One-Sentence Difference

Hermes packages a gateway around a learning agent.
OpenClaw packages an agent around a messaging gateway.

That distinction sounds abstract, but it has practical consequences for how you configure and interact with each tool.

What Hermes Gets Right

Hermes Agent comes from Nous Research and launched in February 2026. It’s hit about 135,000 GitHub stars as of this writing. The headline feature is what they call a “learning loop” — the agent creates and evolves its own skills based on what it does.

From their feature docs:

• Self-improving skills: The agent generates procedural knowledge from experience. Run the same task type a hundred times, and Hermes actually gets better at it.

• Five sandbox backends: Local execution, Docker, SSH, Singularity, and Modal. You pick how isolated you want command execution to be.

• Subagent delegation: Spawn child agents with isolated contexts and terminals. Parallel workstreams without context pollution.

• Broader browser/voice stack: Browserbase, Browser Use, Firecrawl, local Chrome, plus native voice in Discord channels.

The Hermes documentation is worth reading even if you don’t use it — the provider matrix alone covers 19+ providers with detailed auth flows.

What impressed me most was the checkpoint system. Before Hermes touches files, it snapshots your working directory. /rollback if something goes wrong. I’ve used this more times than I’d like to admit.

What OpenClaw Gets Right

OpenClaw has been around longer and has the larger community — roughly 369,000 GitHub stars and 13,700+ community-built skills. It started as a personal assistant project by Peter Steinberger and grew into something much bigger.

OpenClaw is fundamentally a gateway. The docs are explicit: “The Gateway is the single source of truth for sessions, routing, and channel connections.”

What that means in practice:

• Channel breadth: Discord, Google Chat, iMessage, Matrix, Microsoft Teams, Signal, Slack, Telegram, WhatsApp, Zalo, WebChat. One Gateway process handles all of them.

• Multi-agent routing: Isolated sessions per agent, workspace, or sender. You can run different agents for different purposes through the same gateway.

• Mobile nodes: iOS and Android apps that pair with the gateway for camera, canvas, and device actions.

• Massive skill ecosystem: 13,700+ community skills covering everything from email to calendar to flight check-ins.

The architecture assumes you want one always-on process that routes messages to agents. That’s different from Hermes’s model of “here’s an agent runtime that can talk to various platforms.”

Known Pitfalls

Both tools have well-documented failure modes that the communities are vocal about. Worth knowing before you commit.

Hermes:

• Self-evaluation always passes. Hermes evaluates its own work to decide if a task succeeded. The problem: it almost always thinks it did well, even when it didn’t. This means the skills it auto-generates from “successful” tasks can encode errors. You need external validation for anything important.

• Self-learning overwrites manual edits. The same system that auto-generates skills also overwrites your customizations. If you’ve spent time tuning a skill for a specific workflow, the agent may “self-improve” it back into something generic. Power users find this maddening.

• Maturity gap. With only 11 releases compared to OpenClaw’s 137, Hermes simply hasn’t been tested at the same scale. Fewer updates means fewer chances to break things — but that’s not the same as proven stability.

OpenClaw:

• Updates break things. This is the most consistent complaint in the community. Users report roughly a 25% chance that any given update will break response delivery, cron jobs, or webhooks. The development process lacks the staging/testing discipline you’d expect.

• Memory is unreliable. Agents forget instructions, cross-contaminate data between projects, and repeat mistakes. Memory retention issues are the #1 driver of user churn.

• Self-hosting is the real barrier. Docker setup, SSH configuration, YAML files, security hardening, 24/7 uptime — users consistently report spending more time on infrastructure than on their actual agent workflows.

Trade-offs

A comparison on ScreenshotOne put it well: Hermes is “agent-first” while OpenClaw is “gateway-first.”

Hermes optimizes for the agent becoming more capable over time. It’s built for people who want autonomous agents that learn from experience.

OpenClaw optimizes for a persistent assistant you can message from anywhere. It’s built for people who want infrastructure they can talk to.

Neither approach is wrong. But they lead to different outcomes:

Security Considerations

This matters more than people think. A Reddit thread documented OpenClaw’s 2026 security incidents: 6 CVEs, 341+ malicious skills identified in the community repository, 135,000+ exposed instances found by Shodan.

OpenClaw grew fast. Some security assumptions that made sense for a personal tool on a laptop became dangerous when people started running it on public VPSes with open ports.

Hermes, being newer, has zero reported agent-specific CVEs as of April 2026. That’s not because it’s inherently more secure — it just hasn’t had the same scale of exposure. Give it time.

Both projects now have sandboxing options and approval flows. But if you’re deploying either on a server, audit the defaults. Neither assumes you’re running on a hardened production box.

When to Pick Hermes

Hermes is the better choice if:

• You want an agent that improves at tasks over time

• You need multiple sandbox backends (especially Modal for cloud execution)

• You’re doing research-style workflows with subagent delegation

• You want tight IDE integration via ACP

• You’re willing to trade ecosystem size for a more capable core agent

The learning loop is what justifies choosing Hermes over OpenClaw. If you’re running the same types of tasks repeatedly — data analysis, code review, research synthesis — Hermes will genuinely get better at them.

When to Pick OpenClaw

OpenClaw is the better choice if:

• You want to message your assistant from everywhere (24+ platforms)

• You need the existing skill ecosystem (13,700+ skills)

• You want mobile nodes for phone camera/canvas integration

• You’re building team infrastructure, not just a personal agent

• You value stability over cutting-edge features

If your primary use case is “I want to message my AI from WhatsApp and have it do things on my computer,” OpenClaw has that nailed.

The Cost Problem

This doesn’t get discussed enough. Running either agent autonomously is expensive if you’re not careful. Every message sends the full conversation history to the API, so costs compound within a session.

Users in the community report anywhere from $1-3/day on budget models to $130+/day on Claude Opus for heavy agentic use. The fix is aggressive session resets and picking appropriate models per task tier:

• Quality-sensitive work: Claude Opus 4.6 (expensive, best agentic performance)

• Daily driver: GPT 5.4 (thinking mode on medium+) or MiniMax M2.7

• Budget automation: Qwen 3.5/3.6 (free on OpenRouter), GLM-5.1, Kimi K2.5

Flat-rate subscriptions (MiniMax at $10-20/month, Ollama Pro Cloud at $20/month) are rapidly replacing per-token billing as the community default.

What I Actually Use

I run both — and the community data confirms this is a growing pattern. The specific architecture that works: OpenClaw as orchestrator (planning, decomposition, multi-step coordination, scheduling) and Hermes as execution specialist (fast, repeatable task loops). They communicate via the ACP protocol.

OpenClaw handles my day-to-day messaging — it’s the interface I talk to from Telegram. I’ve been using it for months and the skill ecosystem covers most of what I need.

Hermes runs on research tasks where I want the learning loop. When I’m doing a series of similar analyses, Hermes’s skill evolution actually matters.

I could probably consolidate — Hermes’s docs actually note that it’s the “successor to OpenClaw” and they have a migration command (hermes claw migrate) — but I haven’t felt the urgency. They solve different problems well.

Summary

Both projects are actively developed. Both have real communities. Both work.

Hermes is younger, more ambitious architecturally, and smaller in ecosystem. OpenClaw is more mature, broader in integrations, and has had more security scrutiny (for better and worse).

The 30% of developers who switched from OpenClaw to Hermes cite “maintenance fatigue” from debugging community skills and wanting the learning loop. The 35% who stayed on OpenClaw cite integrations and ecosystem breadth.

Pick based on what you actually need. If you want a persistent assistant you can message, OpenClaw. If you want an agent that improves itself, Hermes.

Or run both — they’re free, and the resource overhead of a second process is negligible.

Links:

• Hermes Agent — official site

• Hermes docs

• OpenClaw — official site

• OpenClaw docs

• Detailed comparison on ScreenshotOne

Every engineer here operates like a mini-CEO. They don’t wait for specifications. They don’t wait for mockups. They see a problem, they ship a solution. Sometimes multiple times a day. The velocity is genuinely impressive — and terrifying if your job is supposed to be designing things before they get built.

So he had a choice. Two ways to lose this game:

Option A: Pixel janitor. Accept that design happens before he knows about it, spend all his time cleaning up after the firehose, perpetually one sprint behind.

Option B: Beautiful irrelevance. Create gorgeous Figma specs that are technically perfect and aesthetically flawless. By the time anyone looks at them, the feature shipped Tuesday and we’ve moved on.

Neither of those is a real job. So he’s been thinking about what a design function actually looks like when your colleagues include AI agents that ship production code.

The Traditional Design System Assumption

Most design systems are built on an assumption that breaks here: a designer will interpret the rules.

A traditional design system says “use 4px spacing multiples” and “primary buttons are blue.” It assumes a human designer will understand when to break those rules, when to apply judgment, when the context demands something different. The system is a reference document. The designer is the interpreter.

That assumption doesn’t work when half the code getting written comes from AI agents, and the human engineers are moving too fast to check a Figma file.

The system has to be directly usable by the people — and the agents — actually shipping. Not just readable by designers who might interpret it later.

Docs as Infrastructure

In practice, documentation becomes infrastructure.

The “brand vibes” doc nobody reads becomes useful when an agent can ingest it — and if it’s written clearly enough, actually act on it.

DESIGN.md becomes as important as README.md. Not because designers will reference it, but because it’s the source of truth that agents and fast-moving engineers will consume without asking questions.

This changes what design documentation looks like. It can’t be “capture the ineffable feeling of the brand.” It has to be specific, opinionated, and machine-actionable.

Three Layers

Ivan’s been sketching out a rough timeline:

Short Term: Stabilize

Not glamorous, but necessary.

Right now, there’s been a lot of dev-led design. That’s not a criticism — it’s what happens when you don’t have a designer and you ship constantly. But it means inconsistencies have crept in. Button styles that differ across surfaces. Spacing that varies by who wrote the component. Color usage that drifted from whatever the original intent was.

First job is to audit, find the obvious drift, and get everything to a shared floor. Document what actually exists, not what we wish existed. Create a baseline.

Medium Term: The Fun Part

DESIGN.md — a markdown file that holds the brand DNA. Written for agents as much as for humans. Not “we value simplicity” but “form labels are sentence case, never title case” and “error states always include a suggested action, not just a description of what went wrong.”

Custom skills — the kind of thing Kilo uses internally. Skills that encode “the Kilo way” so that when a dev or an agent reaches for a UI primitive, they reach for the right one by default. The system isn’t just documented; it’s embedded in the workflow.

Maybe a drift linter — essentially a component inventory CLI that can flag when something doesn’t match the system. Like how a code linter catches style violations, but for design consistency.

Long Term: Kilo Uses Kilo

Eventually, agents become part of the consistency layer.

Imagine a design reviewer that flags drift in PRs the same way linters flag code smells. A copy checker that catches when button text doesn’t match our voice guidelines. A brand reviewer that notices when we’ve wandered off palette.

The goal is to set the policy, build the reviewers, then review the reviewers — not to personally inspect every pixel indefinitely.

Open Questions

He’s still chewing on several things:

How much process can a velocity culture absorb? Add too much structure and you kill the thing that makes this org effective. Add too little and you get permanent chaos. There’s a line somewhere.

How do you write DESIGN.md so it actually takes positions? It’s easy to end up with generic guidance that sounds good but flattens everything into mush. “Be consistent” isn’t useful. “Modals never contain more than one primary action” is.

What level should a skill live at? Component level? (”Use this card component.”) Flow level? (”Confirmation dialogs follow this pattern.”) Decision level? (”When in doubt, fewer steps beats more clarity.”) Taste level? (Is that even possible?)

How do you measure drift without crying wolf? A system that flags everything is useless. A system that misses real problems is also useless. Calibration matters.

What’s Next

Right now he’s deep in the audit phase — documenting what exists, finding the patterns and the anti-patterns, building the baseline. Tedious work, but necessary before anything else makes sense.

The medium-term work is what he’s most excited about. DESIGN.md as real infrastructure. Skills that encode taste. A system that doesn’t require him in the loop for every decision.

He’ll write more as this progresses. If you’re solving similar problems — design in a high-velocity, agent-heavy environment — we’d love to hear what you’ve figured out. Find us in Discord.

If it’s seemed quiet on the Mistral front for a while, that’s because they’ve been heads-down building. This new model is a major leap for the lab, and the focus on agentic work—coding and agentic engineering—benefits all of us.

Mistral’s new flagship is a dense 128B model with a 256k context window, built from the ground up for long-horizon agentic work. It merges instruction-following, reasoning, and coding into a single set of weights, with configurable reasoning effort so you can dial it up for a gnarly refactor or keep it light for a quick edit. It scores 77.6% on SWE-Bench Verified, putting it ahead of Devstral 2 and models like Qwen3.5 397B A17B. The vision encoder was trained from scratch to handle variable image sizes, and the whole thing can run self-hosted on as few as four GPUs.

And Mistral is sticking to their OSS principles: the new model shipped with open weights under a modified MIT license.

This is a serious new model for serious engineering tasks, and Mistral users will find that it’s now the default for the Mistral Vibe CLI and Le Chat. And with Kilo, anybody can use the model among hundreds of other top models and always find the right tools for the job.

Use Mistral Medium 3.5 Everywhere You Use Kilo

The new model is available in the Kilo Gateway, so you can use it everywhere with a single login.

VS Code Extension

The upgraded Kilo Code VS Code extension now surfaces Mistral Medium 3.5 in the model switcher. Pick it for any task where you want a model that can hold a lot of context, reason through complexity, and produce structured output your codebase can actually consume.

Kilo Code CLI

Running Kilo from the terminal? Mistral Medium 3.5 is available there too. It’s a strong choice for longer CLI sessions — dependency upgrades, test generation, CI investigations — where you want the model working steadily without losing the thread.

Cloud Agents

Kilo Code’s cloud agent infrastructure is where Mistral Medium 3.5 really opens up. Kick off sessions powered by this model, walk away, and come back to finished branches or draft PRs. The model was built specifically for async, multi-tool work — running long stretches reliably, calling tools in sequence, producing structured handoffs. That makes it a natural fit for the tasks you want to delegate completely: module refactors, issue triage, test coverage gaps, incident investigations.

KiloClaw

Mistral Medium 3.5 is available as a model option across KiloClaw recipes. Whether you’re running a personal claw or a work claw, you can now back those workflows with a model that handles complex, multi-step reasoning without breaking a sweat.

Try It in Kilo Today

Mistral Medium 3.5 is priced at $1.50 per million input tokens and $7.50 per million output tokens through the API. For a frontier-class 128B model at this capability level, that’s competitive — especially for agentic runs that justify the context and reasoning headroom.

At a blended price of $3 per million tokens for general chat, and just $1.56 per million tokens for long-context summarization, it’s more affordable than it might look at first glance.

Plus, if you grab a Kilo Pass you can embrace a healthy discount :)

Open the model switcher in the latest version of our VS Code extension, select it in your CLI agent config, or choose it as the backing model for your next KiloClaw recipe. It’s available now in public preview — we’d love to hear what you build with it.