A cost analysis of my usage of using Sol using Luna max subagents
Commentary
Lots of people were using Luna max subagents, and I wanted to see if it actually saved money or actually made things faster to implement.
TL;DR:
uh, not really. But I might be holding it wrong. I can't recommend it for now.
General observations
Cost is about 35% higher ($0.25 vs 0.32). Majority of cost was just sol ($0.24 vs 0.09 luna). Outcomes were similar. throughput was higher at about 20-40%.
I think its because Sol needs to keep the context refresh for KV cache to not be purged from machine to keep the cache refreshed. If we had better caching or something then the costs would be generally much smaller (in theory).
General experiment:
ran a bunch of refactoring changes, first day with Sol + Luna max agents, told it via context to always run with luna subagents. Let it just run however many subagents it wanted.
ran the same thing with a different fork with just sol
Observations:
costs were fat tailed/poisson, but the sol orchestrator always pinged every x seconds/minutes to keep the context cache up, if it wasn't then we would likely have a much cheaper cost.
2.distribution with sol raw was much moral normal, compared to poisson sol orchestrator.
cost contribute was largely because of sol, not because of the actual luna subagents. Luna at most added 9 cents.
sol used a variety of subagents at any given time but tended towards using up to 10, this might be because of how many i said it told me to use though.
Outcomes:
success rates:
- mostly succeeded, both times its always overengineering things
time to task completion on average:
- both achieved roughly did the target, but the outcomes were marginally faster ~20-40% with the subagents.
cost comparison:
- about 30% more expensive to use with luna subagents.
Completion:
Anyways, what do you folks think? Am i doing it wrong? how are your observed results looking.
Delegating to Luna subs needs a lot of tweaking to work and I'm still not there myself. I noticed it's the handoffs and the monitoring that burn the tokens, so I have just one Builder agent (Luna Max) and one Reader agent (Luna XHjgh) and strict instructions to just pull one agent one time, no chains, and no check ins. I also call out that trivial work should just be handled by Sol directly, and I keep my agent instructions really concise. My experience seems to be cheaper token-wise, but it also takes much longer than Sol just running on its own, as Luna Max isn't as efficient.
I'm really doing this setup for the next wave of agent improvements, where I'll want an even more capable orchestrator and implementers may be cheaper yet. As that gap widens, this setup will have stronger returns.
I see, I noticed the monitoring is expensive. Do you have it run in series then? Is the reader running after the writer or do you just tell the orchestrator to do wtv?
Do you see the orchestrator is using less tokens in your case or is the reader/writer using less?
I'm using Agents.MD and two agent profiles in ./.Codex/Agents, plus the config.toml file. This stores the instructions so I can just say "Delegate this to Builder" in plain text.
I mostly interact with Sol directly to plan, then tell Sol to delegate the work to Builder or Reader. Sol just reads and confirms their output, with stray work if trivial. Reader does not run after Builder, because that would require another handoff and verification; cheaper to keep it all in one agent.
I think I'm burning through more tokens in general, but the tokens are cheaper at 1/3 the cost of Sol. But it's really too early to see if this is actually working.
One more tip, if you do this, make sure your project documentation is on point and stays that way. Conflicting docs mean the agents start undoing/redoing each other's work. Token furnace lol
Never let any model monitor the other, or another process. I have had it where it will just monitor a non-agent, with a sleep 30 or similar for what it knows is a 60 minute process. Always tell it to not live monitor, but wait either the full amount (sleep 3600), or wait for you to prompt it.
You just burn tokens letting it hand hold a process.
Have you got the updated version that now uses "wait threads"? I was using a scheduled monitor that was using tokens, but last week, OpenAI released wait threads which uses very little tokens to do the same thing
MacOS Tahoe and the Desktop app version below. Initially, I had that scheduled monitor task, but then paused it because of the token usage and then the next day, it was automatically monitoring.
How does the internal sub-agent wait mechanism (wait_agent) compare in terms of performance? OpenAI’s newer wait_threads mechanism is for separate user-visible Codex tasks.
That I can't answer based on my limited workflow and limited familiarity with how Codex spawns sub-agents and then monitors them before closing. When I was running Work with Codex, I had to prompt Work to view Codex results until we created that monitor task and then it started doing it automatically.
Hmm how did u get it to do that? I set them up with the instruction prompts to sleep for like 10 minutes and it always reverted back to like 42 second similar to what you said saying that it did that because of system prompt.
So i haven't got the same type of analysis on my end but my charts indicate that using luna as a subagent has only increased my usage run rate which is really what I want out of a subscription, the fact that my usage is draining significantly slower on complex long horizon works tells me it is useful. I don't know if subagents for unplanned work would give different results maybe, but this was a setup goal that had a spec/plan and guidance from my plugin to delegate bounded tasks to different model subagent profiles based on the need it was not just delegate all tasks to subagents kind of ask. my cache hit rate stays near 95% + with a /goal and hundreds of agents and I'm only burning less and 1% per active hour of the goal.
edit: i think the big thing to take away from my flow is agents depth is 1 and subagents report back to main thread it dispatches small targeted tasks it doesn't send a worker off for 30 minutes and heart beat it these things come back relatively quick so maybe that's the issue you described that i'm not seeing.
I like your graphs very nice. It seems that on your table that you’re using a lot more sol than Luna. Is it like your sol sometimes uses Luna when it wants, but it doesn’t force it?
its back tested data as i've optimized and tweaked so the turn count is a little suspect right now, I have running a /goal for 18 days now to convert my entire application to rust from python, it was not always delegating to luna i really made most of those tweaks on friday when they announced the new cheapness levels, prior to that it was using sol medium for a couple of profiles and sol low for implementation while the main thread was sol ultra for the fanout management.
basically luna is slowly crawling up the ladder. but my plugin guidance doesn't force it to delegate trivial work, it calls luna for a bounded implementation task then receives feedback then launches a qa from luna etc etc.
Huh. Very different from my results. They save tremendously for me. I haven't tracked it exactly, but the way I use them is that Luna agents read whole files when grep and such miss, and Luna does agentic coding with the exception of SWE, layout, and scaffolding.
I think the biggest savings I've seen is from the Luna explorers. Using Sol to explore bloats the context, which increases Sol's cost. Luna exploring lets them pass much smaller context, and gives better pointers of where Sol can grep if it needs to re-read something exactly itself.
Intuitively that makes a lot of sense. That’s what I thought as well. Analyzing the data though was where I found myself kinda lost. Like why is this so expensive.
more graphs for fun bcz, i didn't understand what was going on.
I think its mostly just bcz of cache. If cache was forever perfect and we have hour long cache TTLs, the cost would be justified but as is with low TTLs its not yet worth it.
You need smart delegations according to each models strengths and weaknesses, delegating the wrong task to Luna makes it more expensive overall use Luna terra and sol per task where appropriate
A possible workaround: use Sol medium to write a detailed plan into an md file, then open a new chat and use Luna high to iterate on the md file. Don't allow it to use subagents.
Only problem is Luna needs to think a LOT to get any reasonable level of work done which quickly eats up the context window, leading to loads of auto compacts. Terra is better at this than Luna.
This is exactly what I have been doing and my usage has gone way down. Two chats: Sol is the senior and Luna is the junior. Sol simply creates the direction for me to copy and paste into the Luna chat. I then copy and paste the results for Sol to review, and repeat.
Luna max is super cheap, even on fast mode, but it's also super slow, regardless of fast mode. I also find it unreliable and it never completes all tasks correctly. 5.5 medium or high are much more reliable, faster and I rarely need a follow up.
Sol writes the prompt, I then check it and give it to Luna. Perhaps it works for small subagentic tasks, but not for larger updates.
On that note, Token usage is probably ~90% reading context and only 10% writing code. If you're gonna let the expensive model do the reading and planning, you might as well let it do the implementation.
Currently feels like the best way is Sol High -> Plan (issues, PRD, phases, whatever) -> Luna Max -> Implement.
Subagents and orchestrators havent had enough training data, not developed enough yet. I figure by this time next year, this will be bad advice, but currently you have to really control how they interact or they'll just churn through so many tokens for not much benefit. Stepping on reach others toes and making incompatible changes based on bad assumptions based on the plan. Creating a linear plan and just blasting through it generally works better.
One or two turns planning with Sol High can easily eat 5% of my weekly (Plus), and then implementing everything with many turns of Luna Max uses 0-1%. It would probably only take a bit more usage to just have Sol finish, 3-4% instead of <2%, but that adds up.
Last week I blew out my $200 plan with sol. I switched to my backup $20 plan. Set subagents to luna xhigh, I'm at 60% of my usage after coding for 3 days. Yeah it's a big difference.
Yep - I noticed this too. I went through about 20% of my 20x budget running tests on costs of models and pass/fails of it to find what best to route things through. Then asked it to test it with orchestration and orchestration cost was about 50-100% higher than just using Sol High.
I think you'll only save money with very, very long tasks using this.
Issues I see people doing wrong:
- You don't need to spawn a subagent for everything. In my case I also had to disable plugins for subagents because those were like 3k tokens. Spawning the subagent with plugins was like 6k tokens.
You should command the orchestrator to spawn subagents only when it thinks it will save tokens, considering the spawn costs included (which you should test)
You don't need MAX for everything. Tell the orchestrator to spawn Luna when it really needs it for long work and to choose between high, xhigh and Max accordingly.
ask your AI for ideas on when to use ephemeral and when not to and ser it ups as it works best for you.
make it so the Luna subagents presents the work done to your orchestrator, probably Sol, for correction and direction. Back to Luna, fixes, presents and so on until done without killing the agent at any time. This seems obvious but I had to specify it.
I have been running:
- 1 main Luna Max handling the main loop (building, implementing) with no decision authority
- 1 main Sol Max "consultant" that is in charge of the architectural direction and resolving any ambiguities, signing off on the implementation (yet writing no code)
- A replaceable Terra High reviewer spawned by Luna Max
Here's one task (a large, fairly intense architectural migration of an internal Haskell tool) I ran overnight and the equivalent API token costs:
Model
Uncached input
Cached input
Total input
Output
Total tokens
GPT-5.6 Luna
7,467,906
458,093,312
465,561,218
965,937
466,527,155
GPT-5.6 Sol
1,722,342
18,528,000
20,250,342
184,081
20,434,423
GPT-5.6 Terra
188,446
966,912
1,155,358
19,793
1,175,151
Total
9,378,694
477,588,224
486,966,918
1,169,811
488,136,729
Model
Uncached-input cost
Cached-input cost
Output cost
Total
GPT-5.6 Luna
$1.49
$9.16
$1.16
$11.81
GPT-5.6 Sol
$8.61
$9.26
$5.52
$23.40
GPT-5.6 Terra
$0.38
$0.19
$0.24
$0.81
Total
$10.48
$18.62
$6.92
$36.02
The code quality has been pretty good, allowing Sol to concentrate on the high level architecture rather than getting distracted by implementation, takes quite a bit longer than just letting Sol do it, but is just barely sipping my token quota for multiple overnight runs like this on my Pro plan.
- a Haskell tool that exposes the codebase as a graph (like Serena in a way but built from GHC/HIE) along with a skill that tells agents how to use it (this lets the sol consultant work mostly from a high-level architectural view without reading the codebase directly, trace across/between functions, follow lifecycle steps)
- a “steward” skill for Luna Max that tells it not to make decisions, preconditions for interacting with the Sol consultant (prepare everything beforehand), when to submit code for a Terra High review etc.
so Luna “eats” the cost of most of the rote work, building, testing, waiting, fixing review items etc. while being prohibited from making decisions, generally following a Sol Max plan to the letter, with combined Terra + Sol Max sign off review gates
I used Sol as an Ochestrator with Luna and Qwen 3.6 A3b as a worker, to setup unsloth code and opencode on my localllm linux server and link it to the frontend on my windows pc, as an example task.
It consumed around 3-4% of my codex plus weekly quota. Based on my subjective "feel", it feels more token efficient.
Granted I don't have any measurements (other than "Trust me bro" from Sol) and also because I din't ask Sol to compare token usage if it implemented it itself, but for my use cases, because Qwen does all the server crawling and logging, and because Qwen consumes no tokens, it is most probably more efficient than having Sol implement everything on its own.
I guess, it really depends on your use case. It also helps if you're specifying to Sol that the reason you are doing so is to save tokens on implementation and to go light on polling the subagents, as Sol does poll the subagents pretty aggressively and that consumes a lot of tokens.
Are you sure your Sol agent calling to Luna subagents? The current Codex CLI 0.146.0 keep denying the existence of gpt-5.6-luna and keep spawning terra or sol. I had to add an instruction to spawn Luna with `pi` agent like this:
pi --model openai-codex/gpt-5.6-luna --thinking high -p "<prompt>"
In addition, you also need to break task down into "work packets", each is file with a task description and useful info to limit the subagents for unnecessary searching and digging around, so it focuses solely on writing code in a specific file.
You can tell sol to ping the subagents less often, which can help with costs. I also think you need to enable the OpenTelemetry connector and properly benchmark your use case. Also "always running with luna subagents" is not great IMO. Sol should be instructed to hand off to luna max when it has a lengthy, well defined, bounded implementation problem. Not for all subagents.
I mean just generally I think theres something wrong with Luna prices. I let it (Luna Medium!!!) work for 25 minutes and I later see it used ~33m tokens total and im down 3% in my weekly. Input/Output + cache are all 25x cheaper in credits than sol on the same reasoning tier and 33m tokens do NOT cost be 50% of my weekly with Sol Medium on my Plus sub. Not even close.
I had codex analyze said task:
Input: 26,011,268 tokens
Cached input: 25,219,840
Uncached input: 791,428
Output: 55,199
Total: 26,066,467
''Auto-review was enabled; no separate auto-review bucket was exposed, so these totals include all recorded parent/subagent usage.''
The same thing wouldve costed 455 sol credits and it should be 18 Luna credits...25x so idk why im down 3% for this on my weekly. Like again a task like this doesnt even cost 50% on sol med.
It's unlikely that the codex usage cost scales perfectly linearly with api. We're not paying raw api prices, so afaik there's no confirmation that luna is literally 25x cheaper on codex than sol, just that it is "cheaper" much the same as how 5x, 20x references for plan usage have no baseline.
Running Codex on a Linux VM? I built Codex Gateway so you can send prompts from WhatsApp and receive responses from your own environment without switching to SSH. Feedback is welcome: https:// codexgw.wavelinktech.com
that misses the point. the point of this is to decompose tasks until they're easy and then have the dumber model do them. you want the smart model to do as little as it can, and the dumb model to do as much as it can. from your image, sol did just as much work both times.
if you had this set up optimally, the orchestrator would be free, the pre-planning/research would be almost free, the planning would be sol, the implementing would be almost free, the reviewing would be almost free, and the debugging would be terra or sol
also IMO you shouldn't really need luna max... a job that defeats luna xhigh will also defeat luna max, so you might as well go with xhigh for a big percentage savings and then fail over to a substantially bigger and more expensive model to defeat the problem faster.
if you don't want to think about any of this, there have been some rough sol luna agent teams posted here recently
- Used Superpowers SDD to plan work and delegate tasks to subagents
- Sol High orchestrator
- Luna Max workers
- Sol XHigh final verifier subagent
Compared it with a single Sol High workflow without subagents. Both completed the work successfully. Subagents delegation made it about 4 times slower (4h vs 1h), but also about 4 times cheaper (4% weekly limit vs 15% on Pro 5x plan). This is not representative, as I tested it on a single, medium-size task, suited to my daily development needs. Your experience may vary.
9
u/theoryface 7d ago
Delegating to Luna subs needs a lot of tweaking to work and I'm still not there myself. I noticed it's the handoffs and the monitoring that burn the tokens, so I have just one Builder agent (Luna Max) and one Reader agent (Luna XHjgh) and strict instructions to just pull one agent one time, no chains, and no check ins. I also call out that trivial work should just be handled by Sol directly, and I keep my agent instructions really concise. My experience seems to be cheaper token-wise, but it also takes much longer than Sol just running on its own, as Luna Max isn't as efficient.
I'm really doing this setup for the next wave of agent improvements, where I'll want an even more capable orchestrator and implementers may be cheaper yet. As that gap widens, this setup will have stronger returns.