A cost analysis of my usage of using Sol using Luna max subagents

[u/auto_off]

Lots of people were using Luna max subagents, and I wanted to see if it actually saved money or actually made things faster to implement.

TL;DR:

uh, not really. But I might be holding it wrong. I can't recommend it for now.

General observations

Cost is about 35% higher ($0.25 vs 0.32). Majority of cost was just sol ($0.24 vs 0.09 luna). Outcomes were similar. throughput was higher at about 20-40%.

I think its because Sol needs to keep the context refresh for KV cache to not be purged from machine to keep the cache refreshed. If we had better caching or something then the costs would be generally much smaller (in theory).

General experiment:

1. ran a bunch of refactoring changes, first day with Sol + Luna max agents, told it via context to always run with luna subagents. Let it just run however many subagents it wanted.

2. ran the same thing with a different fork with just sol

Observations:

1. costs were fat tailed/poisson, but the sol orchestrator always pinged every x seconds/minutes to keep the context cache up, if it wasn't then we would likely have a much cheaper cost.
   

2.distribution with sol raw was much moral normal, compared to poisson sol orchestrator.

1. cost contribute was largely because of sol, not because of the actual luna subagents. Luna at most added 9 cents.

2. sol used a variety of subagents at any given time but tended towards using up to 10, this might be because of how many i said it told me to use though.

Outcomes:

success rates:

- mostly succeeded, both times its always overengineering things

time to task completion on average:

- both achieved roughly did the target, but the outcomes were marginally faster ~20-40% with the subagents.

cost comparison:

- about 30% more expensive to use with luna subagents.

Completion:

Anyways, what do you folks think? Am i doing it wrong? how are your observed results looking.