For most of the past year, open-weight models lived in the experiment column. Teams tried them, benchmarked them, and kept their production traffic on closed APIs. That is changing fast.
We recently co-hosted an evening panel with Friendli AI and MiniMax to talk about that shift. Each panelist represented a different layer of the stack: the agent harness, the model, and the inference engine. The conversation kept returning to one idea. No single layer decides the outcome. They all depend on each other.
The panel
Ari Messer, AI Partnerships, Anaconda (Kilo)
Victor Su-Ortiz, Product Marketing & DevRel, MiniMax
Yunmo Koo, Founding Engineer, Friendli AI
Alex Campos, Moderator, Friendli AI
Key Takeaways
Adoption is real now. Banks, fintech, and biotech are running open models in production, and the reason has moved from cost to performance and control.
Think in systems, not models. A reliable endpoint can still produce an unreliable agent once calls are chained together.
Architecture gains need engine work. New attention and cache designs only pay off when the inference layer is built for them.
Treat your inference provider as a partner. Then test on your own workload, not just public benchmarks.
Build evals and observability in from day one. It is the only way to keep up as models and hardware change.
Theme 1: The Conversation Moved From Cost to Control
Ari opened with what he sees across Anaconda and Kilo customers. Interest in open models has been around for a while, especially in regulated sectors that care about visibility and governance. What is new is actual production use.
“It’s really just in the last couple of months that we’re seeing everyone from our biotech clients to really big banks and fintech starting to lean in this direction. Some of the initial conversations were about cost. But now it’s actually a lot more about performance and control.” — Ari Messer, Anaconda
Victor agreed, with a twist. Cost was “all the rage over the summer,” he said, but the performance teams care about is not general benchmark scores. It is how well a model does in one domain, like legal or healthcare. More teams are fine-tuning and post-training open models for exactly that.
Cost still matters. Victor pointed to public usage data from OpenRouter and OpenCode, where fast, low-cost Flash models lead. Speed and price are still top priorities for most users.
Theme 2: From Model-Centric to System-Centric Thinking
Yunmo framed the biggest change teams face when they go to production.
“The biggest shift is moving from model-centric thinking to system-centric thinking.” — Yunmo Koo, Friendli AI
In experiments, you check a model card, plug the model into a coding tool, and see if it handles your task. In production, the question is whether the model can complete a complex agent pipeline reliably, within your latency and cost budget. He gave a simple example of why that is harder than it sounds.
95% → 60%: An endpoint that is 95% reliable drops to about 60% end-to-end reliability across a 10-step agent chain. Define success for the whole workflow, not one call.
He also warned against a common assumption. Open weights do not automatically bring better economics or performance. You take on responsibility for model choice and inference configuration. That is where an inference provider can carry the load so your team can focus on the product.
Victor added that cost per token is a useful number, but your own internal benchmarks on your own use cases matter most.
Theme 3: Models Built for Long-Horizon Agent Work
Victor explained how MiniMax’s design shifted between its M2 and M3 series. Long-horizon tasks got much more weight. One-million-token context windows are now standard, because agents prompt over and over across many turns.
“A common practice I used to set for myself is to stay below 30% of the context window. But nowadays, up to 70 to 80%, I can’t stop myself.” — Victor Su-Ortiz, MiniMax
To handle that load, MiniMax uses a sparse architecture and its own KV-cache optimizations, built for users who will push toward the full window when refactoring a whole codebase.
Yunmo added the inference view. Labs now design for intelligence and efficiency together, using hybrid attention, cache compression, and sparse indexing. But a better design on paper is not the same as a faster system.
“Theoretical improvement in the architecture does not always translate into real-world benefit.” — Yunmo Koo, Friendli AI
Linear attention, for example, does not use a standard KV cache. Without engine work tailored to how it manages state, it can even slow inference down. That is where model labs and inference providers need to work closely.
Theme 4: Safety, Governance, and Hybrid Deployment
Open weights give enterprises more control, and more responsibility. Ari said requirements vary by client, but safety testing and red-teaming come up again and again. You have to test the model and also how the agent behaves once it has access to it.
Anaconda does this with Enkrypt AI, which it acquired this year. It lets teams red-team models and agents in a similar way: the agent using the model, or the model placed in different agent environments.
Ari also described a growing interest in local AI. Picture running a model on a laptop for something very private, then moving that session into a shared cloud environment when the team needs it. Open models fit that movement well. Ari was candid that hybrid is still more vision than settled pattern.
“A lot of people don’t want to deal with on-prem. That’s what’s being pitched to them, but they’re like, there must be a different option.” — Ari Messer, Anaconda
Theme 5: The Human Layer Still Matters
Asked what it looks like when every layer works well together, Ari’s answer was about people, not technology.
“There can be a tendency in any org where people think that their individual failures, like getting a lot of tool call errors, are their fault. And they don’t really share it.” — Ari Messer, Anaconda
The most successful teams he sees share both wins and failures across groups. Victor agreed and closed with a line that got a laugh.
“This knowledge sharing is just as important given how early we are. Don’t just chat with ChatGPT.” — Victor Su-Ortiz, MiniMax
Advice for Your First Production Agent on Open Models
Start with a Flash model. Ari’s pick: “Flash models have come along really fast. They’re really efficient, and they’re fun to work with.”
Go specialized. Victor suggested moving past generalist models and working with labs and providers on fine-tuning, post-training, or on-prem needs for your domain.
Embed evals and observability everywhere. Yunmo’s top tip: build them “into every single step of your agent workflow,” so you can quickly compare models and providers as things change.
Define success for the full workflow. Track reliability, latency, throughput, and cost end to end.
Share what you learn. Tool-call errors and odd traces are team knowledge, not personal mistakes.
Thanks to Friendli AI and MiniMax for co-hosting, and to everyone who came out and asked great questions!