That “deep Transformer trained by maximum likelihood” is the complex part.
Billions of parameters in a tensor field distributed over a dozen or more layers, each layer divided by hidden sizes, and multiple attention heads per hidden size. Every parameter’s weight is algorithmically adjusted during training. For every query a matrix multiplication is done on multiple vectors to approximate the relevancy between each token. Possibly tens of thousands of tokens being stored in cached memory at a time, each one being analyzed relative to each other.
And for standard architecture, each parameter requires four bytes of memory. Even 8-bit quantization requires one byte per parameter. That’s 12-24 GB RAM for a model considered small.
Deep transformers are not simple systems, if they were then it wouldn’t take such an enormous amount of resources to fully train them.
You’re really breaking the shitting on AI vibe when you make it sound like the height of human capacity and ingenuity. Can I just call it slop and go back to eating glue?
You can still shit on AI, just because it’s computationally complex doesn’t make it the greatest thing ever. It still has a lot of problems. In fact, one of the main problems is its consumption of resources (water, electricity, RAM, etc.) due to its computational complexity.
I’m not defending AI companies, I just think characterizing LLMs as “simple” is misleading.
Our whole economy is geared to consume resources, we have inflation targeting to prevent aggregate demand and prices from ever falling. If you want to lower consumption need hard currency, the cheap cash that the AI’s are riding on now is most likely still Covid stimulus and QE.
And speculation. Venture capitalists think they can create money by investing betting money that they predict they’ll have in the future. It’s how this circular ponzi scheme between Nvidia and OpenAI is holding itself up for now.
Those huge numbers that they count in their net worth don’t really exist. It’s money that’s been pledged by a different company based on money they pledged to that company in the first place. It’s speculation all the way down.
They’re hoping for a pay-off, but it’s a bubble of sunken costs kicking the can down the road for as long as they can before it bursts.
I do think QE and artificially low interest rates do lead to riskier stocks and commodity like Bitcoin doing better, where growth greatly outpaces value stocks.
Which I think this is a continuation of the Covid stimulus, and its up to economic gravity as to whether there is a debt bubble that will pop leading to a dramatic fall in money supply. I’m pretty sure we’re roughly saying the same thing.
Yeah, I wasn’t disagreeing with you. Just adding some detail.
But yeah, if you look at a graph of corporate profits and net worth, it literally exploded during covid and hardly ever slowed down since then. They got a taste for a greater flow of society’s lifeblood, and now they can’t live without it. Returning to pre-covid rates of “growth” would be labeled as “stagnation” and “failure.”
Meanwhile, consumers were/are facing runaway inflation and a ballooning cost of living, workers being laid off by cost-cutting companies, and yet these corporate executives still have the nerve to say “well we need to pinch pennies to stay afloat, we need tax-cuts and simultaneously we also need taxpayer-funded government stimulus/bailouts/subsidies. Inflation isn’t our fault, it’s just a normal part of a working economy, we’re totally not just artificially raising our prices because we can get away with it (don’t talk about deflation though, that’s a dirty word, that would mean we’re in a *gasp* recession). And no, we can’t lower our prices or pay our employees more; that would eat into our profit margins and we have a fiduciary responsibility to our shareholders.”
I wouldnt call this terribly normal. Though in the end the corporations raising prices are the definition of inflation, I’m sure the government would love to raise the money supply and not affect prices, but this isnt the USSR.
I wasn’t saying it is normal. I was saying corpos gaslight us by telling us it’s normal.
According to the classical economic theories they cite (which are honestly pseudoscience), prices are expected to rise and fall in cycles of inflation and deflation, keeping the system more or less stable over time. But they only cite the part that says “inflation is normal” and not the part where it says “and it alternates with periods of deflation.”
So they push their prices higher and higher because “infinite growth,” but any time there’s even a suggestion that prices might come back down, they panic and scream about a recession and they make the government intervene, because deflation would make their stock prices fall and they’d rather just keep screwing everybody over.
That “deep Transformer trained by maximum likelihood” is the complex part.
Billions of parameters in a tensor field distributed over a dozen or more layers, each layer divided by hidden sizes, and multiple attention heads per hidden size. Every parameter’s weight is algorithmically adjusted during training. For every query a matrix multiplication is done on multiple vectors to approximate the relevancy between each token. Possibly tens of thousands of tokens being stored in cached memory at a time, each one being analyzed relative to each other.
And for standard architecture, each parameter requires four bytes of memory. Even 8-bit quantization requires one byte per parameter. That’s 12-24 GB RAM for a model considered small.
Deep transformers are not simple systems, if they were then it wouldn’t take such an enormous amount of resources to fully train them.
The technical implementation, computational effort and sheer volume of training data is astounding.
But that doesn’t change the fact that the algorithm is pretty simple. Deepseek is about 1,400 lines of code across 5 .py files
You’re really breaking the shitting on AI vibe when you make it sound like the height of human capacity and ingenuity. Can I just call it slop and go back to eating glue?
You can still shit on AI, just because it’s computationally complex doesn’t make it the greatest thing ever. It still has a lot of problems. In fact, one of the main problems is its consumption of resources (water, electricity, RAM, etc.) due to its computational complexity.
I’m not defending AI companies, I just think characterizing LLMs as “simple” is misleading.
Our whole economy is geared to consume resources, we have inflation targeting to prevent aggregate demand and prices from ever falling. If you want to lower consumption need hard currency, the cheap cash that the AI’s are riding on now is most likely still Covid stimulus and QE.
And speculation. Venture capitalists think they can create money by
investingbetting money that they predict they’ll have in the future. It’s how this circular ponzi scheme between Nvidia and OpenAI is holding itself up for now.Those huge numbers that they count in their net worth don’t really exist. It’s money that’s been pledged by a different company based on money they pledged to that company in the first place. It’s speculation all the way down.
They’re hoping for a pay-off, but it’s a bubble of sunken costs kicking the can down the road for as long as they can before it bursts.
I do think QE and artificially low interest rates do lead to riskier stocks and commodity like Bitcoin doing better, where growth greatly outpaces value stocks.
Which I think this is a continuation of the Covid stimulus, and its up to economic gravity as to whether there is a debt bubble that will pop leading to a dramatic fall in money supply. I’m pretty sure we’re roughly saying the same thing.
Yeah, I wasn’t disagreeing with you. Just adding some detail.
But yeah, if you look at a graph of corporate profits and net worth, it literally exploded during covid and hardly ever slowed down since then. They got a taste for a greater flow of society’s lifeblood, and now they can’t live without it. Returning to pre-covid rates of “growth” would be labeled as “stagnation” and “failure.”
Meanwhile, consumers were/are facing runaway inflation and a ballooning cost of living, workers being laid off by cost-cutting companies, and yet these corporate executives still have the nerve to say “well we need to pinch pennies to stay afloat, we need tax-cuts and simultaneously we also need taxpayer-funded government stimulus/bailouts/subsidies. Inflation isn’t our fault, it’s just a normal part of a working economy, we’re totally not just artificially raising our prices because we can get away with it (don’t talk about deflation though, that’s a dirty word, that would mean we’re in a *gasp* recession). And no, we can’t lower our prices or pay our employees more; that would eat into our profit margins and we have a fiduciary responsibility to our shareholders.”
Their profit margins:
https://d1-invdn-com.akamaized.net/content/pic25ddb17638651a9c61a1f9f564472f46.png
I wouldnt call this terribly normal. Though in the end the corporations raising prices are the definition of inflation, I’m sure the government would love to raise the money supply and not affect prices, but this isnt the USSR.
I wasn’t saying it is normal. I was saying corpos gaslight us by telling us it’s normal.
According to the classical economic theories they cite (which are honestly pseudoscience), prices are expected to rise and fall in cycles of inflation and deflation, keeping the system more or less stable over time. But they only cite the part that says “inflation is normal” and not the part where it says “and it alternates with periods of deflation.”
So they push their prices higher and higher because “infinite growth,” but any time there’s even a suggestion that prices might come back down, they panic and scream about a recession and they make the government intervene, because deflation would make their stock prices fall and they’d rather just keep screwing everybody over.