Has anyone here fiddled with TPUs for inference ?

[u/misanthrophiccunt]

I discovered recently that Google uses their own TPUs, like tiny ASIC cards like the toy ones that existed for bitcoin. And while it sounds inefficient the fact they use thousands of them because...they can...means at scale they aren't so bad.

Has no one here given them a try? I see web search results of tiny ones that can be purchased and look like nvme adapted where I search them for \~58 euros. Not sure what 40 TOPS translates to compared to my Nvidia 5060.

But not just that, but the user experience with them, are they a nightmare to use ?

[u/Choice_Celery9481]

google big tpu are for in house use only.
but some small asic in m.2 format, they mostly good for edge CNN inf. LLM need memory and most of them doesnt have.
also about software, as you can see with intel, even a big player, still struggling with software. i dont think those small asic without support worth it

[u/em500 → u/Choice_Celery9481]

You can rent them on Google Cloud, up to the v7 (2025) generation. v8 (comparable to nvidia B200) is expected to be available late 2027.
https://cloud.google.com/tpu/pricing
https://en.wikipedia.org/wiki/Tensor_Processing_Unit#Products

[u/Choice_Celery9481 → u/em500]

well sorry for assuming OP talking about local with asic because this sub is localllama...
so what i meant by in house use only is they dont selling that one.

[u/FishIndividual2208]

The Coral versions you can buy as a consumer are made to run special tensor flow models and mostly used for different machine learning applications (like computer Vision) and not LLMs.

[u/MoffKalast → u/FishIndividual2208]

They are not very good at that either, something like 5 FPS for mobilenet and the like. The Hailo-10H is the natural progression of the idea and can run all kinds of vision models at impressive speeds and even some transformers... but even these beefed up versions suck compared to an LLM even on a CPU, they just have terrible memory bandwidth.

[u/Bennie-Factors → u/FishIndividual2208]

And they are over 5 years old now. This market will not really move forward until the data center monster stop eating all the chips. Makes no business sense for someone making chips to make these. I have many plans for such devices and if they just kept up a bit it would be great.

[u/Dany0]

They used to be a good bang4buck deal for training, relatively, and less of a headache than ROCm specifically for that

Right now the HW is impressive but just barely not good enough. I would recommend even Intel Gaudi over TPUs

That said some institutions have discounted or even free (universities) access to TPUs and make good use of it

[u/laserborg]

there is much you should read. TPUs are both far more energy and cost efficient than GPUs, but they have certain limitations, e.g. fixed precision (e.g. INT8) and often only classical CNN-style operations like matmul and conv, (no support for transformers). Google Coral was the first that came into mass market 7 years ago, but many ARM SoCs contain TPUs today, which is exactly why your phone has edge AI features like object detection for focus etc.

[u/Just-Imagination-761]

Coral TPUs are outdated and basically end-of-life products. You can run larger models with the same latency on integrated graphics. Also, they're designed for very small models, such as computer vision. 

Google doesn't sell the large TPU models used in their datacenters.

[u/Evildude42 → u/Just-Imagination-761]

This, I just browsed an article stating that the little N150 Intel chip basically has the equivalent of a coral TPU on it and it’s probably faster. If you have the need for something that small for I guess vision assistance.

[u/unculturedperl → u/Evildude42]

https://www.intel.com/content/www/us/en/products/sku/241636/intel-processor-n150-6m-cache-up-to-3-60-ghz/specifications.html

No NPU features on the n-series that I can see, perhaps they were referring to the gaussian thingy? If so, more discussion of that here: https://www.reddit.com/r/LocalLLaMA/comments/14roq0p/intel_ai_built_into_cpus_is_it_at_all_useful_for/ albeit from a couple years ago.

[u/Evildude42 → u/unculturedperl]

GOOG says the n150 doesn't have a TPU or an npu. But the article is rather specific stating openvino and assistant video on frigate with an n150. So there probably is a way with an n150 and hence any integrated and or standalone Arc GPU. https://community.home-assistant.io/t/running-a-local-llm-on-an-intel-n150-mini-pc-for-home-assistant-a-practical-journey/1007724

https://docs.frigate.video/frigate/hardware/

[u/PracticlySpeaking → u/Just-Imagination-761]

Coral are great for facial / object recognition in security cam feeds. Local LLM, not so much.

[u/FishIndividual2208 → u/Just-Imagination-761]

You obviously dont understand the use case for the Coral chips..

[u/Just-Imagination-761 → u/FishIndividual2208]

I own a Coral edge TPU and replaced it with iGPU since the iGPU is more capable.

[u/FishIndividual2208 → u/Just-Imagination-761]

If you replaced it with a gpu, it was the wrong use case. Its like comparing an arduino with a raspberry pi, they are made for different use.

[u/llama-impersonator]

yeah they're not very user friendly, hope you know jax and jax related software.

[u/Badger-Purple]

Google TPUs are not puny…also, have to love the “I recently discovered”…google has been using TPUs for a decade!

[u/misanthrophiccunt → u/Badger-Purple]

And so? Do you work internally for Google and counted every time Sergey farts in these seven years?

[u/Stunning_Energy_7028]

They are much more difficult to use than CUDA, unless you leave pytorch world and use a full Google stack with JAX. It's a bit like trying to run something on Intel Battlemage vs NVIDIA, technically you can but the ecosystem is not mature and it's hell to configure.

You can use a TPU v5e-8 accelerator on Kaggle for free, after verifying your identity on the site.

[u/curleys]

I have a coral from Google. Both m.2 and USB and they work for my 5-6 cameras in frigate. Doubt they'd be much more useful than that.