Humans are historically pretty good at offloading mental capacity to some sort of tool in order to tackle larger and more complex problems. Consider solving a math problem mentally. Compare that to the kind of problem you would be able to some with a pen and paper. Then consider what you could do with a pen and paper and a calculator. An LLM purports to be all of that, and more, for any subject. It doesn’t matter that the results are often horrifically wrong, once they’ve offloaded the entirety of their mental capacity to the magic box and refocused their attention somewhere else.
This is both expected and mystifying at the same time.
How dead in the head do you have to be to RELY on such things?
Humans are historically pretty good at offloading mental capacity to some sort of tool in order to tackle larger and more complex problems. Consider solving a math problem mentally. Compare that to the kind of problem you would be able to some with a pen and paper. Then consider what you could do with a pen and paper and a calculator. An LLM purports to be all of that, and more, for any subject. It doesn’t matter that the results are often horrifically wrong, once they’ve offloaded the entirety of their mental capacity to the magic box and refocused their attention somewhere else.