The thing has no intelligence, it’s a statistical engine (at that’s the case not just LLMs but for ALL machine learning implementations)
So it doesn’t even has common sense and its not going to go: “Oh shit, this might have some really nasty side effects so better check with somebody” unlike even the most junior domain expert imaginable.
Worse, the probability distribution of its errors in the domain of the gravity of the consequences from said errors is uniform - in other words it’s no less likely to make a major life destroying error than it is to make a tiny error which isn’t a problem - which is why we get stories of for example LLMs advisings somebody who has voiced suicidal thoughts to “kill yourself”.
So you NEVER, EVER, EVER wire its output directly to something important - at the very least it needs to be cross-checked by somebody with the knowledge to judge it, the time to properly go over it and the power to stop it being actioned.
The thing has no intelligence, it’s a statistical engine (at that’s the case not just LLMs but for ALL machine learning implementations)
So it doesn’t even has common sense and its not going to go: “Oh shit, this might have some really nasty side effects so better check with somebody” unlike even the most junior domain expert imaginable.
Worse, the probability distribution of its errors in the domain of the gravity of the consequences from said errors is uniform - in other words it’s no less likely to make a major life destroying error than it is to make a tiny error which isn’t a problem - which is why we get stories of for example LLMs advisings somebody who has voiced suicidal thoughts to “kill yourself”.
So you NEVER, EVER, EVER wire its output directly to something important - at the very least it needs to be cross-checked by somebody with the knowledge to judge it, the time to properly go over it and the power to stop it being actioned.