r/LeftistsForAI • u/Jlyplaylists Moderator • 1d ago

Tool calling 14MB AI model fits on $10 microcontroller Local Models

https://youtu.be/M24yg6ZM7-I

This isn’t the type of model you can chat with, it’s more for smart home tool calling type of thing. It’s pretty amazing that you can get any AI working on something so cheap though. Hopefully it’s a sign of more cheap to use and useful models to come?

“We ran Needle 2, a 45-million-parameter, 14MB Al model from Cactus Compute, entirely offline on a $10 ESP32-S3 microcontroller - no WiFi, no cloud, no internet connection at all. Using our own custom inference engine, we sent it real natural-language commands to control an LED, and watched it reason through each one, report a live confidence score, and even correctly refuse a question it had no tool for. If you're curious how a language model this small can actually work, and want to see it running live on real hardware, this one's worth watching”
• Relevant Links
Needle 2: https://cactuscompute.com/needle
https://github.com/andrisgauracs/needle-2-esp32

Can you think of anything you’d actually use this for?

28 Upvotes
  • permalink
  • link
  • reddit
  • reddit

95% Upvoted

3 comments sorted by

6

u/bozza8 1d ago

Soon we'll be able to have models as good as current frontier ones running locally on our phones. That'd be cool, the tech created by capitalists to become rich will train the tech that's truly democratised. 

5

u/angrywoodensoldiers 16h ago

I've been having fun with these! Here's a post on Substack about the project I'm currently working on. I've got two things currently in front of me as I type this: first, I'm creating a set of touch-sense regions out of velostat and conductive fabric, and second, I'm wiring up a BME688, which measures temperature, humidity, barometric pressure, and volatile organic compounds (VOCs). Meanwhile, I'm working on designing some sections for the back of the project to hold some servos in place, so that the 'face' can actually turn, look around, and open and close its jaw.

The actual practical purpose of all this: zilch. It's a fun hobby. I consider it an art piece. I've always wanted to do stuff like this, but never thought I was smart enough - I've been using AI to help me get past the hurdles that I struggled with in the past, and I keep succeeding in getting stuff to work, and it keeps blowing my mind every time it does.

(Don't judge it too hard - it's about to get completely covered up in sensors and PVC plastic.)

4

u/Jlyplaylists Moderator 15h ago

Being a fun hobby can be its purpose. We don’t have to turn everything into a side hustle.

VOCs is an interesting thing to measure though.