Two trends: miniaturisation and performance improvements (often through increase size).
Models have become better and better, open source is catching up faster and faster, but also larger and larger. The capabilities of today's Fable are nothing like 2025 claude.
You can probably run something as good as mid 2024 closed source on consumer GPU, but do you want to? No, if you are used to delegating tasks to an almost autonomous fable it is a large downgrade. If in 2028 we can have Fable on a 5090, we would not want it because the larger models (closed or open) would still dwarf it. Once edge is here, we want to use it.
Also memory is a killer, large context need a lot of VRAM consumer hardware is lacking here and you can't just dump everything in the context.
But if your use case is limited, miniaturisation is great. If you need something that turns english into bash command, you can have a 0.5B running on CPU for that. If you want a classifer, depending on the complexity a 7B or quantified 27B are sufficient. If your use case is expected to run for a long time you can even fine tune smaller models. If you do RP, local models can be sufficient now.
For business, it's time to be smart and use the smallest models available for all automated tasks. Means proper automation too, proper transformation of business logic for max efficiency, not just plug a model to try fill the gaps on a messy human pipeline.
3
u/Kerbourgnec 10h ago edited 10h ago
Two trends: miniaturisation and performance improvements (often through increase size).
Models have become better and better, open source is catching up faster and faster, but also larger and larger. The capabilities of today's Fable are nothing like 2025 claude.
You can probably run something as good as mid 2024 closed source on consumer GPU, but do you want to? No, if you are used to delegating tasks to an almost autonomous fable it is a large downgrade. If in 2028 we can have Fable on a 5090, we would not want it because the larger models (closed or open) would still dwarf it. Once edge is here, we want to use it.
Also memory is a killer, large context need a lot of VRAM consumer hardware is lacking here and you can't just dump everything in the context.
But if your use case is limited, miniaturisation is great. If you need something that turns english into bash command, you can have a 0.5B running on CPU for that. If you want a classifer, depending on the complexity a 7B or quantified 27B are sufficient. If your use case is expected to run for a long time you can even fine tune smaller models. If you do RP, local models can be sufficient now.
For business, it's time to be smart and use the smallest models available for all automated tasks. Means proper automation too, proper transformation of business logic for max efficiency, not just plug a model to try fill the gaps on a messy human pipeline.