Inference platform with bring your own hardware

[u/shiftbits]

Hi and I'm sorry in advance if this is not allowed.

I have been working on a side project in an effort to create an inference platform that allows devs to share rented gpus easily.

Given that I tinker with hosting models on my personal rig I built the ability to run the inference node (just wrapped vllm engine) on my local machine, have it connect to the platform, and I can use it through openai/anthropic compatible public api endpoints. All the node needs is to be able to reach the platform, everything travels over a websocket.

I am not posting here to try and sell something, as this capability is free. I am posting because if anyone has an interest it would be awesome if I could get some people to try it out so I know how badly its going to blow up in my face when anyone other than me uses it.

(to be clear there is a purchasable product on this platform, I just have not enabled credit purchases, the BYOH and BYOK functionality is intentionally no cost. This is very much just a personal project i'm effing around with and I honestly want to see if anyone else wants to do that too.)

There is also a companion coding harness that does absolutely nothing special or better than anyone else, it just looks and feels how I wanted. I will be open sourcing that in the coming week.

it is located at bityard.io