You’d have to start soldering the HBM and the GPU chips off the boards they’re on. Those aren’t PCI-E boards much of the time. Also, they don’t have display outputs.
I’d absolutely buy one of those servers to run open-weight models locally, but let’s not kid ourselves here: We’re talking about an ungodly amount of power for a single machine, think it was a few megawatts for a rack, or at least hundreds of kilowatts. A single server luckily should still be in two digit kilowatts. Could realistically only use it in the winter though.
I can. Compute is compute. Even if it’s extremely niche you give me enough cycles I’ll find something to run on it.
There is a geniuly cool idea floated a last year I think at xorg conf for a Linux driver that used GPGPU instructions to do the traditional GPU processing as well. So theoretically you could use some of the AI accell to instead compute things like rastors.
You used to be able to BIOS flash workstation GPUs to make them somewhat usable for gaming… not sure how far these AI GPUs have diverged from gaming cards these days and if a BIOS flash could make them playable.
AI GPUs have specialized very heavily for neural networks, including things like going down to 8- or 4-bit floats in favor of more parallelization. Most of that hardware is just useless for graphics tasks at this point.
Except the people I hear running “local models” report a lot of hardware, like a half dozen graphics cards. The common person is screwed, they have a smartphone.
Two or three years ago sure. Now…not even close. Tons of models run on basically shit now. Especially with MOEs. I run new lightweight models on an i9 and 64gb of ddr4 ram with a 4gb quadro card…in a laptop from 19’.
I think this is why a lot of these tech bros are saying silly shit about suggesting people need to rent from them. That’s their entire shtick. Otherwise they got nothing else.
If you’ve got at least 8 gigs of VRAM and at least 32 gigs of RAM you can run some pretty beefy Qwen models using Strata, far more than you normally could. Upgrade that to 12 or 16 and 64 gigs respectively and you can run a pretty decent quant of Qwen 3.8 flash next. It’s a LOT bigger than the usual 27 and 35 billion models people run locally that themselves require pretty beefy hardware. Strata is doing some magic by deciding which experts stay in VRAM vs which ones are in RAM vs which ones are so rarely used that they can just be read from the SSD as needed (though it’s best if you don’t have to result to the SSD as a backup option at all)
hardware that people cant USE, they are solely used for AI in those datacenters, its just going to be e-waste.
No it isn’t. 0h come on. Yeah thy couldn’t possibly take the same or similar hardware components and make them work in PCs.
They are already putting NPUs in PCs.
You’d have to start soldering the HBM and the GPU chips off the boards they’re on. Those aren’t PCI-E boards much of the time. Also, they don’t have display outputs.
I’d absolutely buy one of those servers to run open-weight models locally, but let’s not kid ourselves here: We’re talking about an ungodly amount of power for a single machine, think it was a few megawatts for a rack, or at least hundreds of kilowatts. A single server luckily should still be in two digit kilowatts. Could realistically only use it in the winter though.
I can. Compute is compute. Even if it’s extremely niche you give me enough cycles I’ll find something to run on it.
There is a geniuly cool idea floated a last year I think at xorg conf for a Linux driver that used GPGPU instructions to do the traditional GPU processing as well. So theoretically you could use some of the AI accell to instead compute things like rastors.
You used to be able to BIOS flash workstation GPUs to make them somewhat usable for gaming… not sure how far these AI GPUs have diverged from gaming cards these days and if a BIOS flash could make them playable.
They don’t even have headers for soldering a display output to them. They’re purely a compute card.
It’d work nicely as a pure accelerator for an actual GPU though, with the right firmware.
Almost back to the 3DFX pass-through days again… 😁
AI GPUs have specialized very heavily for neural networks, including things like going down to 8- or 4-bit floats in favor of more parallelization. Most of that hardware is just useless for graphics tasks at this point.
Government bales out AI companies and the bill they write to do it will make local LLMs illegal.
Piracy is illegal and they are of course doing a great job at stopping that…
Reverse engineering isn’t piracy
Unless you’re training your models…
I think you are right, but that seems VERY unenforceable.
Successful enforcement doesn’t look like catching every pivate LLM owner.
Except the people I hear running “local models” report a lot of hardware, like a half dozen graphics cards. The common person is screwed, they have a smartphone.
Two or three years ago sure. Now…not even close. Tons of models run on basically shit now. Especially with MOEs. I run new lightweight models on an i9 and 64gb of ddr4 ram with a 4gb quadro card…in a laptop from 19’.
I think this is why a lot of these tech bros are saying silly shit about suggesting people need to rent from them. That’s their entire shtick. Otherwise they got nothing else.
Got some references and keywords for research?
If you’ve got at least 8 gigs of VRAM and at least 32 gigs of RAM you can run some pretty beefy Qwen models using Strata, far more than you normally could. Upgrade that to 12 or 16 and 64 gigs respectively and you can run a pretty decent quant of Qwen 3.8 flash next. It’s a LOT bigger than the usual 27 and 35 billion models people run locally that themselves require pretty beefy hardware. Strata is doing some magic by deciding which experts stay in VRAM vs which ones are in RAM vs which ones are so rarely used that they can just be read from the SSD as needed (though it’s best if you don’t have to result to the SSD as a backup option at all)
I saw the video, you still need a fast NVMe, I’m still in data SSD…
https://youtu.be/8F_5pdcD3HY
This guy does a great brain dump on how little is needed to run LLMs on basically old junk.
Kudos
Local LLMs will never be illegal, because that genie is out of the bag
Bails
With what money? No one is buying bonds. The US would have to print so hard it’d cause hyperinflation, which still means the bubble pops
The fed balance sheet gets bigger like it did with AIG.