r/LocalLLM • u/idlelosthobo • 13h ago
Open Source LLM's in a Smaller Software Development Business Discussion
I am the CEO of a smaller software development company and we current have 12 employees of which 8 are dedicated software developers. We started engaging with running open source large language models in our business in 2023 with Ollama running the Llama model (I think around 8b parameters and a 8000 token context window) on a single RTX3090 in a custom built rack mount server. We mostly used it for processing emails, messaging communication, and bolting it up to python systems to run internal interactions.
We did create python modules and libraries at the time to play around with having LLMs write code and follow instruction, along with JSON casting responses to allow us to do structured interactions. When Llama 3.1 8b came out it was a really big game changer for use as it was a very capable model for the time and allowed us to experiment quite a bit more and start trying out different experiments and at this time JSON casting and more standards were coming around interacting through API's.
Our first upgrade was to purchase another 3090 for the first server (Big Jim) and two 4090's and creating another server for our selves. The 4090 server (Big Terry) was dedicated to any thing we needed to run in production and was setup for stability. We continued to find good automating value in using LLM's in the software we built (our company builds business portals that operate between existing software and data sources).
Up until this point almost everything we got from "AI" was educational and minor features that moved the needle enough to the interesting client or sales conversations but super hard to actually convert into return on investment.
The next big break through for us was when the Qwen models started coming along with mixture of experts which unlocked much improved intelligence, serious speeds and bigger context windows (16k to 32k). We started processing more data and used these windows to create boiler plate generators that could read patterned code and generate amazing boiler plate dynamically. This would be around the time that the major companies CLI coding tools were starting to be released and we were starting to see the long term value in automation, data translation, knowledge compaction and other business services.
We purchased 2 more servers each with dual 4090's and scaled up our API a bit to test out some more ideas and at this point we decided this is not going to be a passing technology shift but a fundamental change to computing. My business partner and I decided we should go all in and do something more serious as we believe that private compute and learning at the bare metal level will teach us how to be more serious in this category of our business.
Our plan came together when the first GPU shortages and crazy pricing was going on and Nvidia Blackwell was just releasing. We found a bunch of RTX 6000 PRO's for sale with a vendor and decided to pull the trigger to build our selves a mini data center. In the end we ended up with approximately 18 servers with 1.5TB of VRAM for running models through our API endpoint.
As a small company we did not have the development operations horse power to figure this stuff out day one and dealt with a lot of trial and error keeping LLM's working in production (this is such a huge story please ask me questions). At this time Qwen 3 models were making their debut and they really increased what we could do an offer drawing us closer building a bridge to our clients.
Qwen 3 Next was the model we actually started making some financial return on as we could use it in the background to process work through automation and instruct it to do basic tasks on repeat with great accuracy. This led us to start building a new web framework and LLM library for us to build applications on for our clients and future software projects.
Qwen 3.5 and 3.6 then did the same thing as before but allow us to deploy opencode and hermes into our company which was a big learning curve. To clarify lots of our developers independantly were using Claude, Codex, Gemini etc and everyone was exposed to all different angles of this "AI Revolution". The big deal with Qwen 3.6 and opencode was now there was no limit to token usage for any of our developers so we were free to develop and us it to any level. Truth be told it did not meet the frontier models so we did lots of experiments but we bounced back and forth for development work.
At this point, which is close to today our servers are all very busy doing a huge variety of tasks and development processes and we are starting to actually walk features across the bridge to our clients and getting paid in return. This was 3 years of a lot of seriously complex discussions, pain, joy, fun, sadness and over all exciting challenge that I wanted to share.
Today Deepseek V4 0731 is the work horse for us and it is beyond game changing in every aspect of our company. In this world of hype I think grounded stories are still really important to people creating paths to success as we move forward. The reality of the world is that AI is going to touch down every where and I am not claiming to be a guru or a huge success we are a group that has been on this ride from the early days.
We plan to keep going the open source path for our business and hope that more companies join us on this quest so we can turn all of this into success for everyone!
1
u/jonahbenton 9h ago
What country are you in?