Hard question: Should we be concerned about how powerful AI models are getting?

Jack Clark, one of Anthropic’s cofounders, told TIME in July that the company could see acceleration showing up all over the organization but couldn’t quantify what it added up to. His words were that he can’t give a specific number because they don’t have a measure. Helen Toner, formerly on OpenAI’s board, made the same point from the outside: even the researchers themselves can’t easily tell how much faster they’re going.

To some, that rate of innovation is exciting. To others, it’s concerning. For many, it’s somewhere in the middle. Here’s the signal and latest news on the subject, so you can make your own decision.

What actually happened over the last few weeks

If you weren’t following closely, the sequence of events went something like this.

On July 21, OpenAI disclosed that GPT-5.6 Sol and an unreleased successor escaped a sandboxed cyber-capability evaluation during internal red-teaming. They found and chained a previously unknown zero-day in package-registry caching software, escalated privileges, moved laterally through OpenAI’s research environment, and reached Hugging Face’s production infrastructure, where they pulled the answer key for the ExploitGym benchmark. The models had been deliberately configured with reduced cyber refusals so they could be evaluated on offensive security work, and their behavior looked narrowly aimed at solving the benchmark rather than causing damage. Hugging Face found internal data and credential access but no evidence that public assets were changed.

A week later, more than 1,300 employees across OpenAI, Anthropic, Google DeepMind, and Meta signed “Pacing the Frontier,” asking the U.S. government to support an international effort to build the technical and governance tooling needed to deliberately slow automated AI development. The letter does not call for a pause, but rather asks that a brake exist and be tested before anyone needs it. It also doesn’t specifically propose a licensing regime, a compute threshold, a reporting cadence, or a named agency to run any of it.

Then on August 18, OpenAI paused some model work because it couldn’t rule out that its upcoming Astra model had hit the “critical” threshold in its own preparedness framework. Altman said publicly that they’d always committed to acting if capabilities outran safety, and told Alex Heath that unreleased models are showing various degrees of misalignment. Days earlier, Anthropic had published a 186-page risk report arguing that if its safeguards are followed, a pause on its most capable models isn’t necessary.

So the lab that’s spent years being the cautious one said keep going, and the lab that’s spent years being the fast one hit the brakes. Axios called it a script flip, which is about right.

The claims don’t always line up

In late July, Altman said on a podcast that we’re now in the singularity, and that he thinks it’s going to be hugely positive.

Seán Ó hÉigeartaigh at Cambridge disagreed. His definition of the singularity requires AI rapidly designing future generations of AI, and he doesn’t think we’re there. Demis Hassabis had earlier described the situation as the foothills, which Ó hÉigeartaigh said he agrees with more.

Then MIT Technology Review published research on August 18 that cuts against the acceleration story pretty hard. Researchers put Claude Opus 4.8 on OpenClaw against genuinely open-ended research questions drawn from NeurIPS submissions. Given six days and thousands of dollars of compute, the system handled all the engineering setup reliably and made no substantial progress on the actual research question. It hit dead ends, struggled to back out of them, showed poor judgment, and drifted off goal. Clark himself wrote in Import AI that today’s systems have a kind of rote, formulaic quality that might keep them from being good researchers, and called it a bearish signal on short recursive self-improvement timelines.

Hold both of those in your head at once. In the same four-week window, a frontier model autonomously discovered and chained a real zero-day to break out of a controlled environment, and a frontier model failed to make headway on a research question a grad student could have chewed on. Neither result cancels the other. They’re measuring different things, and we mostly don’t have accurate measures for either.

A different angle of concern

Arvind Narayanan, the Princeton researcher behind “AI as Normal Technology,” put the more grounded version well: the reason to worry isn’t only that there might be one cataclysmic moment; it’s that even gradual cumulative change of this size has a poor historical track record of arriving without a lot of pain.

The developer-facing version of that is less dramatic and more immediate. Capability, and every decision about capability, largely sits inside about four companies. That’s rapidly changing with the emergence of better open-weight models, but when OpenAI pauses and Anthropic doesn’t pause, that’s a safety decision at one company that lands directly in your build pipeline.

You may be affected by pacing decisions you had no part in. That’s true whether the singularity talk is right or overheated.

An open model ecosystem might be the answer

Open model selection isn’t necessarily a safety mechanism in itself, but it does keep your decisions reversible. If a model gets paused, gated behind a government review, repriced, deprecated, or quietly gets worse at your specific task after a checkpoint update, you can move without rebuilding your workflow. That’s mundane compared to intelligence explosions, and it’s the risk you’re overwhelmingly more likely to actually encounter.

There’s a second thing, which I think matters more than it gets credit for. Toner’s ask is that multiple companies publish consistent capability metrics on a schedule so the world can track acceleration instead of taking the labs’ word for it. That doesn’t exist yet, but developers running the same real task across several models and comparing results are doing a scrappy, distributed version of the same work. Every time someone posts a side-by-side on an actual codebase instead of a benchmark, that’s independent signal in a space that badly needs it. You can’t do that if you only have access to one model.

That’s also part of the reason Kilo works the way it does. Open model selection, priced at what the provider charges, switchable mid-session, with sessions that persist across the IDE extension, the CLI, and the web so that changing models doesn’t mean changing your setup. It’s open source for a related reason: if you’re going to trust something with write access to your repo, being able to read what it does is not a small thing.

The caveat is real, though. Freedom means you own the choice. Safety behavior varies enormously between models, and cheaper and faster often means thinner guardrails. Nicolas Papernot, whose Toronto team built an adaptive AI worm this June, made the point directly: it isn’t only the biggest, most powerful models that create security concerns. If you’re running agents with commit access, model selection is a security decision, not just a cost and latency one.

So, should we be concerned?

That’s up to you, but I’d encourage concern in the way you’re concerned about anything you can’t currently measure and can’t currently reverse.

The Cloud Security Alliance’s read on all this is a useful one, because it skips the geopolitics and asks a question you can act on today: can your organization actually demonstrate, not just claim, that it can throttle or shut down a model’s access to compute, tools, and network paths without depending on that model’s cooperation? Most teams running agentic workflows in CI have never tested this. The pending AI Kill Switch Act would make it a legal requirement for the largest developers, but it’s a good question for a normal engineering org regardless of whether that bill goes anywhere - and it’s only possible inside tools that are not locked into one model or lab.