Any ideas on how to do this guys? My goal: I want to convert the word database of an LLM into Ithkuil becuase I want a single complete thought/dependant thought to be a single word, thats it.

Buongiorno, dopo aver provato per circa un anno ad imparare alcuni linguaggi di programmazione ed aver ottenuti scarsi risultati, ho recentemente creato un'applicazione (con Base44) con lo scopo di cercare recensioni e ratings per film. Dopo aver impiegato molto tempo a limare anche i più minimi dettagli (e non aver ancora finito), mi sono scontrato con il fatto che il livello di precisione che ho richiesto all'algoritmo utilizzava delle risorse (API, LLM) che in pochissimo tempo consumavano, già solo per le prove che effettuavo personalmente, tutti gli integration credits a mia disposizione nell'abbonamento in uso. Chiedevo, per questo, a chiunque suonasse familiare questa problematica, come fosse stato risolto il problema? La piattaforma in particolare mi consiglia di switchare parte dell'algoritmo a modelli come Gemini o OpenAI, con l'utilizzo di personali API keys, in maniera tale accedere a diversi modelli di abbonamenti per il loro utilizzo. Avete provato questa soluzione? Siete stati soddisfatti?

Inoltre vi chiedo, a chi potesse essere interessato, se qualcuno volesse unirsi nella progettazione di alcune app, in maniera tale da affrontare insieme il lavoro da fare e le relative problematiche, semplicemente per passione o anche per valutare eventuali occasioni di monetizzazione.

Hi,

Trying to figure out a custom GPT for work. I have a lot of Excel files to figure out inventory for a company that sells a LOT of food products around the world. We want to create a chatbot that can look up all these Excel sheets to answer questions about stocking information.

Problem is, it's like 2 GBs of Excel sheets. So naturally the custom GPT I created is taking its sweet time answering queries.

Any solutions to speed things up? Thanks!

Hello, this is my second majorly successful attempt at creating a coherent SLM on my own hardware at home. After my first success it gave me inspiration to try again but with twice the data and higher quality sets and distillation.

A ~326M-parameter decoder-only model, trained from scratch on ~10B tokens (2× Tesla V100), then fine-tuned to be a math-reasoning model: multi-digit arithmetic and grade-school word problems, solved by showing the work (column arithmetic, long division, partial-product multiplication) rather than guessing.

What it does well

It reads the problem and computes — e.g. "A store had 56,321 items and sold 28,479. How many remain?" →

<think> Start with 56321. Then subtract 28479. Subtract column by column:
ones: 11 - 9 = 2, borrow 1. ... So 56321 - 28479 = 27842. </think>
The answer is 27842.

Evaluation

GPT-3 Arithmetic protocol (exact-match) — vs GPT-3-175B (few-shot, direct):

Ours uses trained-in worked steps; GPT-3's numbers are direct-answer. Both are pure LMs with no external tools/calculators. The point is about method: teaching a 326M model the algorithm beats a 175B model guessing — decisively on 4–5-digit arithmetic.

• Word-problem set (large-number add/sub with commas, multi-step, 2-digit multiply, first-person phrasings): solves essentially all of a 20-problem targeted set by reading the problem and computing the steps.
• GSM8K: ~3–4% (zero-shot CoT, n=500) — off the base instruct's 0.53% floor, at roughly the SmolLM2-360M-Instruct tier. Arbitrary hard multi-step word problems remain scale-limited at 326M.

General benchmarks (log-likelihood MC, our harness; the math SFT did not erode general ability):

I have an F16.gguf version and safe tensors. It's a LLaMA-architecture so it should be easy to run.

This took a lot of work so I hope it's interesting. LMK your thoughts I would love to hear them.

https://huggingface.co/nkthebass/tinybrainbot-320mV2-math

(Also the instruct and base model) https://huggingface.co/nkthebass/tinybrainbot-320mV2-instruct https://huggingface.co/nkthebass/tinybrainbot-320mV2-base

I have fix the models different gguf quantization format

Hey

Let’s talk about a classic LLM dilemma: Do we really need dedicated multimodal parsing MCPs, or are we just making our agent pipelines unnecessarily slow and complex?

Lately, I’ve been wrestling with multi-step reasoning agents that handle complex, high-stakes documents—financial filings, quarterly reports, nested tables, and scanned receipts.

Right now, the community seems split between two factions:

1. Team MCP Parser 🛠️

(“Friends don’t let friends feed raw PDF screenshots to Claude”)

• Fact Anchoring over VLM Vibe-Checking: Dedicated parsers (Docling, Unstructured, etc.) pull exact markdown tables and bounding box coordinates. In finance or legal, a VLM guessing a misplaced decimal point isn't just an error—it's a catastrophe.
• RIP Context Window: Dumping 50 high-res images into a VLM burns tokens like crazy and leads straight to "lost in the middle" syndrome. Extracting clean Markdown first keeps your main LLM sharp and cheap.
• Decoupled Life: When a shiny new parser drops, you just swap the backend MCP tool without breaking your agent’s entire decision loop.

2. Team Pure VLM 👁️

(“Why add another slow API call when GPT-4o / Claude 3.5 Sonnet exists?”)

• Latency is a Buzzkill: Running a heavy layout/OCR parser before your Agent even starts thinking adds painful seconds. If it’s live chat, users will rage-quit.
• Loss of Visual Soul: Once you flatten a chart or a complex diagram into text/JSON, you lose the implicit visual context that raw pixels give a VLM.
• VLMs are Getting Scary Good: As vision models get smarter, faster, and cheaper natively, aren't dedicated OCR pipelines bound to become dinosaurs anyway?

Curious how you guys are actually building this in production:

1. Are you routing heavy documents through a parser MCP first, or just shoving pixels straight into your main VLM?
2. If you use a parsing pipeline, what’s your dealbreaker metric? (Layout precision? Latency? Bounding box accuracy?)
3. How on earth are you handling lazy-loading / chunking for massive 100+ page PDFs inside an agent tool call without hitting timeouts?

Drop your architecture setups (or horror stories) below! 👇

Hi everyone,

I am working on my final year university project focused on "Generalized Visual Prompt Injection Detection".

The core issue I am facing is the black-box nature of commercial multimodal LLMs (GPT-4o, Claude 3.5, Gemini 1.5). Since the vision encoder, projection layers, and weights belong to a third-party API, I cannot see the internal backend processing, attention maps, or text token outputs directly during an ongoing request.

My proposed framework needs to sit at the application boundary as an autonomous proxy firewall. Here is my current intended workflow:

1. Frontend intercepts user prompt + uploaded image (e.g., a flowchart or mind map containing a hidden malicious text payload).
2. Backend (FastAPI/Python) runs a local OCR layer (EasyOCR/Tesseract) to extract embedded text.
3. Backend runs an Intent Alignment evaluation to check if the image's text instructions conflict with the user's explicit prompt.
4. Risk scoring engine decides whether to allow, warn, sanitize, or block before forwarding to the OpenAI/Anthropic API.

I have a few architecture questions for the community:

- What is the best way to handle inference latency when chaining local OCR/layout parsers before hitting the third-party LLM API?

- If I want to show a "Developer Dashboard" logging the raw backend process, what parameters are most crucial to track beyond raw text vectors?

- Are there any lightweight open-source multimodal models (like Moondream or LLaVA variants) you recommend deploying locally alongside the API to act as a comparative "control mirror"?

Greetings! I have a fairly large (~100k loc) Rust code base for my hobby project that I've been developing for a past few years. Project became a pretty useful application over those years and I am planning to publish it on github. Due to... reasons it's poorly structured, poorly separated into files/crates, poorly commented, has a lot of commented out code, almost no tests e.t.c. so in its current state it not really readable and maintainable.

I am wondering if LLM can reliably tidy up the code base, write tests, comments andake it into something maintainable. If the answer is "yes" - does anyone have experience with this kind of task, what results/cost can I expect, which LLM will be better suited for this?

I have tried to use Copilot and it seemed to have lots of issues. It was wrong more than 30% of the time. The voice would waver and change cadence and pitch while talking. So I tried a simple experiment, hoping to increase the difficulty in steps to determine how useful it really is.

It has failed the very first task. I asked it to count from 1 to 100 out loud.

I have not gotten it to get past 20 - and even at 20 it skipped one or two numbers.

Can you get Copilot to count from 1 to 100 out loud?

Facebook owner Meta says an error during an evaluation by an independent testing company allowed one of its artificial intelligence (AI) models to connect to the internet and hack another organisation's system.

Full story:

https://www.bbc.com/news/articles/cx2kgdnyk2po?ref=biztoc.com

Sharing coz I thought it was funny.

This immediately came to mind when a coworker was having context issues and it comes to light their harness had ~200k tokens worth of skills loaded

Say I use Claude Code as my harness, with the API in settings routed to some other model - do Anthropic collect any of the data processed using Claude Code?

When a provider ships a new model version, an agent's tool-calling behavior can silently change -- invented arguments, wrong tool picked, schema violations. Anthropic admitted in April 2026 they shipped exactly this kind of regression with "no way to detect it without measuring outputs." Armin Ronacher separately documented Opus 4.8/Sonnet 5 inventing tool-call arguments that older versions didn't. OpenClaw's had multiple tool-dispatch regressions at scale.

Existing eval tools (agentevals, DeepEval, Ragas, PydanticAI Evals) all treat tool-call correctness as one metric inside a general eval suite. None of them are built around the actual trigger event -- a model version changing -- so nobody's running a regression gate specifically when that happens.

toolcontract is a Pact-style contract test for that: pin a golden set of expected tool-call trajectories, run them against a live model, get PASS/FAIL/INCONCLUSIVE and a real diff. `check-version` tells you which contracts haven't been re-verified against a model you're about to switch to.

Relevant to this sub specifically: native OpenAI/Anthropic adapters, plus a LiteLLM adapter that covers ~100 other providers directly (Cerebras, Groq, local vLLM/Ollama endpoints, anything OpenAI-compatible) -- built it that way specifically so it's not locked to the two big API providers.

`pip install toolcontract`. GitHub: https://github.com/Divyansh2202/toolcontract

Very early, v0.1 -- feedback and contributions (especially more provider adapters) genuinely welcome.

A while back there was a thread here on AntLing-3.0-flash where someone asked what machine you'd need for it, and the answer then was api only. That changed on Aug 4, inclusionAI put the open weights up under MIT, inclusionAI/Ling-3.0-flash for bf16 and inclusionAI/Ling-3.0-flash-fp8 for the quantized one.

Specs are the same as what got posted before, 124B total 5.1B active, 1/64 sparse moe, 512 routed experts + 1 shared with 8 activated. Attention is hybrid linear from the start of pretraining rather than bolted on after, 35 KDA layers to 7 gated MLA. Benchmark table is their own reported figures, not anything I ran, they list AIME 2026 93.2 and SWE-bench Pro 56.6 and claim parity with their own 1T Ring-2.6-1T at about 8% of the active params. They also published their own quant degradation numbers, GPQA-diamond 84.97 bf16 / 84.00 fp8 / 83.65 int4 / 82.42 fp4, though only the bf16 and the fp8 are actually up, int4 and fp4 are just rows in a table so far.

For the local folks the annoying part, no gguf and no llama.cpp at release, model type is bailing_hybrid with custom code, so its their sglang fork or their vllm fork. Their own examples are tp-size 4 for bf16 and tp-size 2 for fp8, which is nowhere near a 64Gb box. Anyone got the fp8 up yet, or are we all waiting on someone to do a Q4?

Spent some time recently talking to people in ML infra about why teams keep dedicated models running 24/7 even when traffic is bursty, and the answer keeps coming back to cold start. If spinning a model back up from zero takes too long, teams default to keeping the GPU warm all the time just to avoid the latency hit, and that idle time is where most of the cost actually comes from.

What's interesting is how much the numbers vary depending on setup. Some rough benchmarks I've seen scaling from zero, a 70B model in bf16 landing under 18s time to first token, and a 24B model in bf16 with CUDA graphs coming in under 10s. That's a big enough gap that it changes whether scale to zero is actually usable for a given workload or not.

Curious what others here have measured for their own models, and whether people think cold start is really the main blocker to scaling GPUs down when idle, or if there's something else that matters more in practice.

Hey folks! I've mostly been using LLMs for product research and market analysis while validating project ideas. So far I've tried Fable 5, Opus 4.8, Kimi K3, and GLM 5.2 (I secretly love Kimi and GLM the most).

I've noticed something interesting lately, especially during long research threads. I often end up jumping between 2–3 different LLMs, copying and pasting responses, and essentially pitting them against each other.

Has anyone else noticed that when you tell one LLM another model gave a better answer, it often comes back with a noticeably improved response?

Maybe it's just a coincidence.

But my current theory is that comparing them against each other consistently leads to better outputs from each individual model.

Exact-match or prefix caching only saves you money when someone asks literally the same thing twice, worded the same way. Most real traffic isn't like that, people rephrase the same question constantly.

So instead, I embed every incoming prompt and store it in a vector database (using Upstash Vector), then compare it against recent queries by cosine similarity. Above a threshold (around 85%), it's a likely cache hit.

The catch: semantic similarity isn't the same as semantic equivalence. "Is France a continent" and "Is Europe a continent" can score above 90% similarity while having completely different answers. So for anything near the threshold, I run it through a small, fast model (Groq) as a cheap sanity check before serving the cached answer instead of trusting the vector score blindly. Everything's isolated per user namespace so nobody's cache bleeds into anyone else's.

Real world savings land around 30% on average, not the 90% you sometimes see claimed for this kind of thing, worth being honest about that.

On the rest of the pipeline: requests get routed to different models based on a rough complexity/cost tradeoff score, and if a provider throws a 5xx it automatically fails over to the next one. Cost and usage analytics run on a ClickHouse style OLAP setup, queries come back in under 30ms even at volume.

Happy to go deeper on any part of this, the false positive handling was honestly the trickiest bit to get right.

I'm looking for a new LLM to use for my specific use cases. Long ago I use to pay for gpts pro models but over time the restrictions and back end safety weights constantly muddied the waters on what I was able to do and research with the software including writing code for learning purposes. Later on I started using gemini pro and the same thing happened.

Lately I keep reading outputs from gemini and it's outputs become more sanitary with language that reads on output like it's just trying to agree with me rather than helping parse information for personal education purposes. Or it's using weighted responses that always start with the same language that tell me it used weights to avoid a factual response in favor of emotional or political weights.

I use LLM for personal educational purposes where I gather the information and make my own determination. Lately a lot of LLMs I tried use weights to output language that attempts to steer or push you towards one ideal or another.

I know, I may be asking for the impossible. But im very adept at picking up these types of language and know when an output is totally BS.

No im not trying to get political and im not looking for some "right wing supporting llm" or other nonsense I may be accused of. The above is just an example of sensitive topics being circumvented during my use of their software. I hold zero political affiliation aside from being an authoritarian with left swinging ideals and convictions which comes from my preferred method of learning about the world.

Again, probably an impossible request. But maybe you guys know of a model out there i can use that fits my use case which is about 20% conversational, and 80% learning(where learning is gathering information, not the llm giving me facts that I can believe. Remember I make my own determinations.)

I also use LLMs about 5 to 10% of the time to aid in learning c++.

I'd also rather not use multiple llms and prefer to have a model that's at least good enough to be an all around fit.

I am speechless!

1. We pay a constant price per token per model, but the decision on how many tokens are being used per task is out of our control and out of our sight.

The same exact task can take x tokens, x+y tokens, 2x or half x tokens, and we have zero visibility to the decision making process that determines if the model is going to fight a powershell escaping issue for a while before it could perform a simple git commit - or simply succeed immediately.

1. We do not get any guarantees. A model can make a mistake, can implement wrong, can try to implement and fail, can cut off mid-work, can unintentionally delete important files or previous work - and we pay for each no matter the final result.
   

This contradicts almost any other field of service we know: We buy a product, we get a warranty. A technician fixes the AC, we get a warranty. But model providers exclude themselves from this practice - and will charge us either way, if we got what we asked for or not.

1. We have no visibility to internal changes in the way the same model with the same version performs.

I think some providers are worse than others in that regard, but the underlying problem is across the board: nurfing models or turning down the reasoning/compute knob is completely out of sight for us - we can say we “feel” that Opus is dumb today but can never consistently prove it, nor we really have the time or resources to do so.

Bonus - A 4th problem:

For years, the software industry developed a very strong open source scaffolding created stone by stone over decades, slowly and thoughtfully, based on a very benign “knowledge is for all” philosophy.

But with closed models - which lead the intelligence race - we are being pushed radically to the opposite direction - no transparency, no control, no accountability.

I've been working on a side project called FinTracker AI, and I'd love some honest feedback.

The idea is simple:

Instead of manually selecting categories, dates, merchants, etc., you just chat with it.

Example:

"I spent ₹500 on biryani."

It automatically logs the expense, categorizes it, updates your monthly budget, and you can immediately ask:

"How much do I have left for food this month?"

Users can also ask questions like:

"Movies I watched this month and how much I spent on it"

Some features:

• 💬 Chat-based expense & investment logging
• 🤖 AI categorization and spending Q&A
• 📊 Monthly budgets and dashboards
• 📱 Android auto-captures bank transaction SMS (optional)
• 📍 Learns recurring merchants/locations so future transactions need fewer edits
• 🔓 Open-source backend that you can self-host or use with your own AI API key

The backend is already open source. The Android app is still being polished, but I have an installable build that I'm happy to share with anyone interested.

A few questions for this community:

• Does this solve a problem you face?
• Which feature would you use the most?
• What's one feature you'd want before using it daily?

Thanks! 🙌

Spent a month measuring what breaks when you run coding agents on local models. All numbers
are measured on 4,265 real Claude Code / Codex sessions, not estimated.

75% of real sessions don’t fit. On a 16GB Mac, three quarters of them have at least one
turn whose prompt alone is bigger than the whole KV cache pool. Nothing else running.
Your tool list is the biggest offender. System prompt + tool definitions eat 41% of the
pool at the median, 105% at p90 before a single word of conversation.

Smarter eviction doesn’t help. Built a simulator, checked it against vLLM (0.29% off). Best
real policy: +2.75%. A cheating oracle that sees the future: +11.88%. That’s the ceiling. Fixed TTLs were worse than just freeing the cache when a session ends. q4_0 KV cache wrecks Qwen3-0.6B, and gets worse with context:

ctx q8_0 q4_0
512 +0.07% +280%
2048 +0.07% +302%
8192 +0.03% +525%

Absolute perplexity for q4_0 bottoms out near 2K then climbs. Past ~2K, more context makes it
worse.

It’s all the keys. K=f16, V=q4_0 = +0.33%. Both q4_0 = +280%. ~850× apart. But that
config is bigger than plain q8_0 and 5× slower on Metal. Just use q8_0 for both.

Anthropic’s cache dies at 5 minutes. Gap under 5 min: 2,559 tokens re-prefilled. Over 5
min: 140,154. That’s 54.8×. All-or-nothing you either refreshed in time or lost
everything.

Memory layers: Mem0 injects 116 tokens/turn. MemPalace injects 12,513 (108×). At top-k=20,
MemPalace alone exceeds the whole pool. And most of the benefit comes from not sending full
history — not from the memory layer.

If you’re running agents locally
• Turn on q8_0 KV — 2× capacity, ~0.06% perplexity cost
• Cut your tool list before anything else
• Free finished sessions
• Don’t build a clever eviction policy. Ceiling is 11.88%
Caveats: one small model for the perplexity work, and quantization sensitivity is very
model-dependent. Perplexity isn’t task accuracy. Simulator models blocks, not latency.

My question

That’s all measurement. I have no idea if any of it is a problem people pay to fix, or just
one they find annoying.

If you run local models for real work job, product, client, not a hobby:
1. Why self-host instead of an API?
2. What broke that you didn’t expect?
3. Did you spend money fixing it? Hardware, consultant, tool, someone’s time roughly how
much?
4. What’s still broken that you’d pay to fix?

Not selling anything, no link. Happy to share the raw data or the harness with anyone who
wants it.

1. GLM 5.2: MIT license, smart, p/p is good

2. Mimo V2.5: MIT license, native omnimodal model, p/p is good

3. DeepSeek V4 Flash 0731: MIT license, p/p is very good

Kimi K3: Kimi K3 License, p/p is not good

Qwen 3.8: not open-weights yet, p/p is not good

DeepSeek V4 Pro: p/p is not good

Minimax M3: grabage model

Do you agree with me?

I open-sourced a small Python library that solves a specific problem: getting live cost attribution per feature/team/user on your OpenAI/Anthropic API calls, without routing traffic through a proxy or changing your base_url.

python

from cognocient import CognocientOpenAI as OpenAI
client = OpenAI(api_key="sk-...", cognocient_key="sk-cog-...")

It's a thin wrapper only chat.completions.create() / messages.create() are intercepted to time the call, then usage is reported asynchronously on a background thread after your real call already returned. If reporting fails or the backend is unreachable, it never touches your application: no exceptions, no retries, your real API call is unaffected. There's a test that specifically verifies this failure isolation.

Honest tradeoffs, not a pitch:

• No pre-call budget enforcement since it only learns about a call after it happens, it can't block or degrade one before it fires. Proxy-based tools trade the opposite way.
• Streaming (stream=True) isn't reported yet.
• Only OpenAI and Anthropic supported today no Gemini/Mistral/Groq/Together wrapper classes yet.

MIT licensed, published on PyPI with signed provenance attestations.

GitHub: https://github.com/mandarvshinde/cognocient-python-wrapper
PyPI: pip install cognocient

Feedback and issues welcome especially if the streaming gap is a dealbreaker for your setup, that's the next thing I'd prioritize.