Except the people I hear running “local models” report a lot of hardware, like a half dozen graphics cards. The common person is screwed, they have a smartphone.
Two or three years ago sure. Now…not even close. Tons of models run on basically shit now. Especially with MOEs. I run new lightweight models on an i9 and 64gb of ddr4 ram with a 4gb quadro card…in a laptop from 19’.
I think this is why a lot of these tech bros are saying silly shit about suggesting people need to rent from them. That’s their entire shtick. Otherwise they got nothing else.
If you’ve got at least 8 gigs of VRAM and at least 32 gigs of RAM you can run some pretty beefy Qwen models using Strata, far more than you normally could. Upgrade that to 12 or 16 and 64 gigs respectively and you can run a pretty decent quant of Qwen 3.8 flash next. It’s a LOT bigger than the usual 27 and 35 billion models people run locally that themselves require pretty beefy hardware. Strata is doing some magic by deciding which experts stay in VRAM vs which ones are in RAM vs which ones are so rarely used that they can just be read from the SSD as needed (though it’s best if you don’t have to result to the SSD as a backup option at all)
Except the people I hear running “local models” report a lot of hardware, like a half dozen graphics cards. The common person is screwed, they have a smartphone.
Two or three years ago sure. Now…not even close. Tons of models run on basically shit now. Especially with MOEs. I run new lightweight models on an i9 and 64gb of ddr4 ram with a 4gb quadro card…in a laptop from 19’.
I think this is why a lot of these tech bros are saying silly shit about suggesting people need to rent from them. That’s their entire shtick. Otherwise they got nothing else.
Got some references and keywords for research?
If you’ve got at least 8 gigs of VRAM and at least 32 gigs of RAM you can run some pretty beefy Qwen models using Strata, far more than you normally could. Upgrade that to 12 or 16 and 64 gigs respectively and you can run a pretty decent quant of Qwen 3.8 flash next. It’s a LOT bigger than the usual 27 and 35 billion models people run locally that themselves require pretty beefy hardware. Strata is doing some magic by deciding which experts stay in VRAM vs which ones are in RAM vs which ones are so rarely used that they can just be read from the SSD as needed (though it’s best if you don’t have to result to the SSD as a backup option at all)
I saw the video, you still need a fast NVMe, I’m still in data SSD…
https://youtu.be/8F_5pdcD3HY
This guy does a great brain dump on how little is needed to run LLMs on basically old junk.
Kudos