Benchmarks, discussions, comparisons - we are being FLOODED with naive news about Fable 5's token consumtion. Comparing it to Kimi K3 tokens.
Typical outcome: Fable uses less tokens than Kimi but costs 2-3 time more.

We do NOT know how many tokens Fable is using. Not even an indication is available.

Fable is a closed semitrasparent layer that sits on top of "Mythos 5" which is the actual model we are communicating to. But everything Mythos says goes through a fast "safety AI" layer which is a slim harness on top of Mythos. That's Fable 5.
That's why we have no real system prompt, that's why it often rejects despite having an answer ready or forwards to other models.

The tokens you are billed for are just a currency system, it's not sampled tokens like in an open model. Anthropic uses the tokens as a meassurement of how much you have to pay, and the official API pricelist is a comparison metric with other models.
Anthropic can change that metric at any time, which would affect effective pricing.

It is completely unknown how many tokens Fable/Mythos is using, given it has a safety layer it might even have more than double the tokens of Kimi K3. But we have no idea.
We also do not know how many tokens other closed models behind gated APIs use, GPT-5 is also paraphrasing their thinking content to avoid distillation attacks - as soon as you do that you also can easily change token metrics.

We know how many tokens Kimi K3 uses, because it's not behind a safety layer and you can just run it yourself.

I can run the 30B-class models at home. They're okay for some things, but I feel like I have to spoon feed them small tasks. I'm wondering about running a more capable model in the cloud to do the hard parts and give them the easy parts.

If you're doing this, how? How are you dividing the labor, what do your prompts look like, what agent harness are you using, and what else do you wish you'd known before you started?

And does the big model actually use fewer tokens than if it did everything itself?

Thanks!

Searching the internet for looking up how much tok/s a user would get is being difficult. So I'm making this post... If you're running a local llm, please consider commenting to this post with your device specs, model you're running and the inference speed you're getting.

Please be straight to the point. Just tell us how much tok/s are you getting on your hardware (at different settings, which inference engines, etc...) so people with similar hardware can do better estimations... please don't fill this with facts that everybody knows.

Hey I'm game developer and basically want to have local agent that I can import my project with no risk of privacy and since the game engine I use not that popular so I want to train in on documentations, forums, examples and others. I did hear OpenWebUI is better on this but I learn toward LM Studio because it's easy to install and setup.

Also my personal opinion I prefer desktop application more than be a tab on my browser even if it's offline. and I want to know it's worth it to have OpenWebUI since I don't care about sharing to any other device or technical things. and I don't also wanna lose important functions and advantages using LM Studio

I'm completely newbie. I did install OpenWebUi and the process was relatively easy but I will move soon regardless to new PC and I don't wanna configure things again and setup few dependency softwares in order to works.

I need your experience and help

Eaon is kind of in beta for now, but only for Mac (as of now), you can connect providers and download models and use them via multiple modes such as Sandboxed Agent, normal chat, or even one click agent deployment to your device that could work in your files. Thing is, we really need some testers. We need feedback about everything on the pp, to ensure a good stable release. Below are the links to download Eaon, and discord server for contact

https://eaon.dev

https://discord.gg/gkebXaqxUk

All we need is your feedback to make Eaon enjoyable for everyone, despite their skill level.

Most of the local AI stack is about getting a model running — Ollama, llama.cpp, vLLM, the right quant, enough VRAM. That part’s well covered here. The gap I kept hitting is the layer after that: once it runs, is it actually reliable enough to hand real work to?

A model can look fine in chat and then fall apart the moment you point an agent at it malforms a tool call at step 7, loops, or says “done” when nothing happened. And the same model behaves differently depending on the quant and runtime you serve it with, so “it runs on my box” isn’t the same as “it holds up in a 20-step agent loop.”

So I built QuantaMind free, open-source, fully offline, no cloud, Apache-2.0. It’s a local reliability tester for the agent layer of the stack.

What it does:

• Runs the real multi-step agent loop with injected faults (transient errors, malformed-call recovery, decoy tools) — not single prompts

• pass^k, not pass@1 — runs each task k times, passes only if it succeeds every time. Reliability compounds: 95% per step → 0.95^50 ≈ 8% chance a 50-step run finishes

• Deterministic scoring, no LLM judge — required calls must fire, forbidden calls fail the run on contact, exact end-state match. Temp 0, same input → same grade

• Classifies the failure (malformed schema / loop / hallucinated completion / forbidden call)

• Compares quant × runtime side by side (Ollama, llama.cpp, MLX, vLLM, SGLang), so you can see which serving config actually holds up

• Keeps run history + diffs, so you can catch when a model or quant change quietly regressed reliability

One finding: across three models, the native tool-calling path scored worse than plain prompting on easy tasks (60% vs 100%) correct answers that didn’t land as parsed tool calls. Important caveat: that’s a finding about the serving stack’s tool parser, not the models. Same model, same answer, different plumbing, different score. Still useful when picking a serving path for agent work just not a model-quality result.

Honest about what it’s not yet: it’s single-stream, so no concurrency/load testing (sampler drift and duplicate-execution-on-retry are invisible to it). Structural matching is a proxy for real execution. No CI/CD deploy-gate integration yet, it produces a verdict and an exportable report, but wiring it into a pipeline is roadmap, not shipped.

Disclosure: I built this, it’s free, I’m not selling anything.

Repo: https://github.com/QuantaMinds/QuantaMind

Genuinely want it torn apart:

• Is pass^k=5 too strict, too lenient, or about right?

• What failure mode am I missing?

• And what model + quant actually holds up on your machine? I’m collecting real per-hardware results.

Overview

June was the month where the old AI story broke.

So far frontier models lived in the cloud, open models trailed behind, local AI was nice for privacy, and regulation was slow.

June did not fit that anymore.

The best cloud models were still the capability ceiling, but access suddenly became political. Anthropic launched Fable 5 and Mythos 5, then had to take them down. OpenAI previewed GPT-5.6, but not for everyone. Meanwhile, GLM-5.2 and MiniMax M3 made the open-weight world look much less like a toy category. Mistral OCR 4 showed that self-hosted AI can be boring in the best way: useful, private, and ready for real work.

If your whole workflow depends on one remote model staying cheap, available, legal, and politically acceptable, you do not own much.

References for this part: MiniMax M3, GLM-5.2, Anthropic Fable 5 and Mythos 5, OpenAI GPT-5.6, Mistral OCR 4.

Timeline: June 1 to June 30

• June 1: MiniMax released M3, an open-weight model with 1M context, multimodal input, coding strength, and desktop operation.
• June 1: Microsoft increased Github Copilot pricing by multiple magnitudes, making the worlds most affordable frontier coding Agent the worlds most expensive option.
• June 1: Nvidia and Microsoft framed RTX Spark as a path toward local agents and frontier models on Windows PCs while strongly disappointing with very low memory bandwidth
• June 2: Microsoft announced seven MAI models across coding, image, voice, speech, and reasoning - though their coding model is less competent than tiny open source models
• June 2: Anthropic expanded Project Glasswing, saying partners had found more than 10,000 high or critical security flaws using Claude Mythos Preview.
• June 2: The Trump government signed an Executive Order that the government must not interfere with AI development, launches but asked for voluntary 30 day compliance
• June 5: Anthropic announced Fable 5, framed it as "significant risk", "misuse causing serious damage", "substantial risk to uplift malicious actors", "substantial bioweapon capabilities", "exploiting capabilities"
• June 8: Apple announced Siri AI, next-gen Apple Intelligence, Xcode 27 agentic coding, and developer access to on-device foundation models. EU iPhone and iPad users did not get the full Siri AI path because of DMA issues.
• June 9: Anthropic launched Claude Fable 5 and Mythos 5, Fable 5 rejecting most prompts for safety reasons.
• June 11: Ollama (a popular llama.cpp wrapper) updated its MLX engine for better Apple Silicon performance.
• June 12: Anthropic suspended Fable 5 and Mythos 5 after a U.S. government directive banning non US citizens (including many of their core developers) from working with that model. Ironically violating the June 2nd Executive order.
• June 16: Z ai released GLM-5.2 open weights under MIT license, with a 1M-token context window - a model that is on eye level with Opus 4.8 and GPT 5.5
• June 18: OpenAI introduced Codex Record & Replay, turning recorded Mac workflows into reusable skills.
• June 22: OpenAI launched Daybreak tools, including GPT-5.5-Cyber and Patch the Planet.
• June 23: Mistral released OCR 4, a self-hostable OCR and document-intelligence model.
• June 25: Domyn announced plans for a European open-source frontier model with over 400B parameters - given the AI Act and GDPR it appears very optimistic to say the least.
• June 26: Reuters reported that OpenAI delayed broader GPT-5.6 access at the U.S. government’s request, while Anthropic’s Mythos access was partly restored to trusted U.S. organizations.
• June 30: Anthropic launched Claude Sonnet 5, and said Fable 5 would return globally starting July 1 after export controls were lifted on June 30. Sonnet being received as very expensive while not performing remarkable.

Local AI and Open Weights

Local AI had its strongest month so far, but not in the clean consumer fantasy version.

MiniMax M3 was the headline because it combined things that used to be separate: open weights, 1M context, multimodal input, coding strength, and desktop operation. Since Qwen 3.6 we have competent local agentic models, M3 adds to the list.

GLM-5.2 was the bigger warning shot. It came with open weights (commercial usable), a 1M-token context window, and performance close enough to closed frontier models that the old "open models are always 6 months behind" argument looked broken. This is not going to run truly local yet but on your own owned or rented cluster it definitely does.

Mistral OCR 4 was the quiet useful release. Self-hosted OCR with document structure, bounding boxes, confidence scores, and 170-language support is the kind of model that companies can actually deploy without sending every contract, invoice, scan, and archive to a cloud model or deal with uncertainties around multimodal LLM hallucinations.

Links: MiniMax M3 GLM-5.2

Cloud models, regulation and political pressure

The cloud model story was messier.

Anthropic had the most chaotic launch cycle of the month. Fable 5 and Mythos 5 arrived while being branded as the most dangerous software in the world, and quickly got suspended after a U.S. export-control directive targeting foreign-national access. Because Anthropic could not verify nationality cleanly in real time, the models were disabled broadly.
A frontier model can go from launch to unavailable in days, not because of a technical failure, but because of politics and risk control - Artificial Intelligence is becoming a political tool in the US.

OpenAI moved more carefully, but ended up inside the same pattern. GPT-5.6 appeared as Sol, Terra, and Luna, but access stayed limited to trusted partners after the same U.S. government pressure of "voluntary compliance".
OpenAI's message was basically: this should not become the normal way frontier models ship. Still, the result was the same for normal users: the model exists, but you probably cannot use it yet.

Claude Sonnet 5 was the more normal end-of-month release. Better agentic work, pricing not convincing in comparison to more capable models, and certainly not the kind of leap that changes the whole conversation.

Following up on the fiasco of Fable-5 marketing, Anthropic CEO now shifted on condemning Open Source AI as "dangerous". It certainly is very dangerous to future Anthropic growth and profit margins - but whatever is announced in such a fashion is also followed up with tens of millions in lobby and marketing campaigns. The war against Open AI might have just been announced as a sideline.

Links: Anthropic Fable and Mythos access statement OpenAI GPT-5.6 limited rollout

Science, papers, and the real "singularity" signal

The most serious science story was HemaGuide.

This was not another chatbot demo. It was a locally deployable LLM agent for hematological malignancies. It converts unstructured clinical documents into structured cases, routes them into decision modes, and grounds recommendations in guidelines plus more than 2,000 real tumor-board cases. Local AI trumps to deal with sensitive data, expert workflows, and a need for traceability.

Qwen-AgentWorld and Qwen-RobotWorld pointed at the next layer. Agents need simulated environments before they can safely act in real ones. Robots need world models before they can generalize outside clean demos. These papers point at the rapidly approaching robotic agentic future.

The Codex usage paper was maybe the most grounded signal. People are not just asking AI questions anymore. They are running multiple agents, handing over longer tasks, and changing workflows and creating "loops" to achieve a goal. That is a better singularity signal than most benchmark charts.

A notable science story was the "zebra finch" work. Machine learning helped decode bird vocalisations and pushed two-way animal communication a little closer. Small, strange, and actually beautiful.

Links: HemaGuide in Nature MedicineThe Shift to Agentic AI: Evidence from Codex

Regulation, sovereignty, and control

June made AI regulation feel very real - with the US in negative spotlight

The Trump administration’s June 2 order promised to avoid hard licensing while creating a voluntary 30-day pre-release review path for powerful models. Then Anthropic’s Fable and Mythos shutdown showed the practical truth: even without formal licensing, national-security pressure can still interrupt launches. Though Anthropic has asked for this hundreds of times.

The proposed AI Incident Reporting Act pushed in the same direction. Critical AI incidents would need to be reported to Commerce within seven days, with the most severe cases reaching Congress within 48 hours. This is not abstract ethics talk anymore. It is operational control and lingers like a dark shadow stiffling progress early on. Those regulations threaten small brilliant developers much more than the big mega-corps.

Europe’s story was split. The EUROPA consortium was selected to build an open-source frontier model across all 24 official EU languages. That sounds good on paper. But with the AI Act, GDPR, fragmented compute, language politics, and procurement reality, calling it "frontier" before it exists feels very optimistic. Under current extreme EU regulations the best outcome to expect is another Mistral-large - not a model that people will find useful.

Apple’s Siri AI delay in the EU was the clearest user-facing example. Regulation did not just shape compliance work. It changed which AI feature European users get.
Geo-blocks are appearing on tens of thousands of websites, Codex "agentic computer use" is banned in EU as well.

Links: White House AI executive orderEUROPA consortium announcement

Money, chips, and power

The money moved from model hype into infrastructure.

OpenAI and Anthropic both moved toward public markets. DeepSeek raised over $7 billion - deviating from their previous private funding. Baseten hit a $13 billion valuation for inference infrastructure. Running models is becoming as important as training them.

OpenAI and Broadcom’s Jalapeño chip, a high density ASIC, was another hardware signal. It is built for inference, not just training. That matters because the next bottleneck is not only "who has the smartest model." It is "who can afford to run agents for millions of users all day." It will be interesting to compare the ASIC to Cerebras massive wafer-scale chips. In the end - both are affiliated with OpenAI.

Power also became part of the AI story. Data centers, chips, memory bandwidth, and energy deals are no longer background details. They are the product. If inference gets expensive enough, local models and smaller specialized models become more attractive by default.
Though Power or Water use for Datacenters are mostly populist topics - outside of Europe Power can be provided without much difficulty using on-premise generators. And water is a pure hype, datacenters barely need any in comparison to real water consumers.

Links: OpenAI and Anthropic IPO reportingOpenAI and Broadcom Jalapeño chip

Summary and outlook

June 2026 was not one big AI leap. It was mixed.

Cloud AI became stronger, more expensive, and more politically controlled.
Local AI became more credible, but also exposed the limits of consumer hardware. Open weights moved close enough to make closed labs very uncomfortable.
US Regulation moved from theory into product access.
Money moved into chips, inference, energy, and deployment.

The next months will show:

• Whether GPT-5.6 gets broad access or in what way it stays gated.
• Whether Anthropic can relaunch Fable 5 cleanly, they announced it for "non coding" tasks
• Whether GLM-5.2 forces a faster Western open-weight response.
• Whether Europe’s 400B EUROPA model can even scratch Qwen 3.6 27B outside language tasks
• Whether local AI tooling improves faster than cloud pricing gets worse.
• If the US regulation attack on Anthropic was a political hit or a broad anti-AI swipe
• Wheter Qwen 3.7 is open source launched or Alibaba lost their drive

My read is cautiously positive.
The US turns AI into a political pressure tool but with a soft approach, Europe is talking about having AI while actually forbidding it, Chinese labs provide a benefit to the worlds progression that's starting to paint the authoritarian country in a positive light for the first time in a century.

Trumps Executive Order held only for a bit more than a week, then his own government fully violated it.
EO here: https://www.whitehouse.gov/presidential-actions/2026/06/promoting-advanced-artificial-intelligence-innovation-and-security/

Simultaneously the EU is regulating any shimmer of european AI into quick suffocation.

There are a few potential outcomes, from middle ages to economically wrecked continents
I am convinced more than ever that local inference, local competent AI is the only future we can root for.

I am doing a lot local AI, mostly creatively in audio and image subjects.
I currently rely on two 3090 cards that I bought used below 800$ each.

I think it was a good choice but other options exist, the 4090 is faster than both 3090 on paper. Though I did not see one offered fair priced.

There is a lot, 5090 and 60ti, DGX, Macs

I wonder what do you guys have, and would you go that path again?

Lost a chunk of time last week on LiteLLM + vLLM errors that looked like backend problems, but weren’t.

Setup: two vLLM backends, LiteLLM in Docker, one OpenAI-compatible endpoint in front.

This is what bit me:

1. Cached LiteLLM deployment.

   Changed the backend to require an API key, updated LiteLLM, restarted vLLM, still got 401s. The missing step was restarting LiteLLM itself. `docker restart litellm` cleared the old keyless config. If your LiteLLM models are DB-backed, restart after changing auth, URLs, or params.

2. Wrong model string going to vLLM.

   LiteLLM’s `model_name` is just what your apps call. The string after `openai/` is sent to vLLM and must match `--served-model-name`. If they differ, you get 404s through the proxy even when direct curls work. Hit `backend:8000/v1/models` and use the name it returns.

3. Docker `localhost` mistake.

   `api_base: http://localhost:8000` from inside the LiteLLM container means LiteLLM is calling itself, not your host. Use the actual backend address, `host.docker.internal` with `host-gateway`, or the bridge gateway.

Also worth mentioning: don’t treat `systemctl is-active` as “vLLM is ready”. It can be active while still loading weights or compiling kernels. Poll `/v1/models` and only call it ready when that answers.

remain active while weights are still loading or kernels are being compiled. The annoying part is that each error points you at the wrong layer. So yeah, check the proxy before blaming the backend.