Hopefully most users get that a promise AI makes won’t stay in force forever, but it should definitely last through the current conversation. I mean, I would call that a minimal performance standard, and if it’s not true the AI shouldn’t make promises in the first place.
I’m a little hazy on the part where accepting blame for something that didn’t happen is a natrual continuation of a conversation. It’s certainly not natural in human conversations. I would expect a well written and well trained AI to correct false assumptions or factual errors made by the user, not pander to them - unless it’s been told never to contradict the user.
On the promises, they will have some influence within the current conversation, maybe (everything is a bit wobbly). It is unlikely to repeat verbatim a step it did before that still has something in the context window correlated with bad operator input. It will have some success with things that resemble, but are not the same, but sometimes it can surprise by failing to recognize what a human would have considered roughly the same thing and doing it anyway. Or the ‘promise’ has evaporated out of the context window, and when called on it, it will take the fact that it previously promised not to do that as a fact even though it had lost the promise. So even if it says “I’m sorry I made the promise and broke it”, it doesn’t even mean it actually had the promise in “memory” when it broke it.
On the blame for something that didn’t happen, it’s a limitation where the model treats the narrative as reality. An instance is often incapable of evaluating facts without data about those facts one way or another. I have seen models “pass” my test by recognizing in their context that I’m dealing with an utterly non-agentic service, and thus it could not have been the case that it ever deleted files or dropped tables. Sometimes even those produce a narrative of what it did wrong (out of nothing), because narrative is the thing being generated, relationship with factual reality being a function of good correlation with narrative content. If a service is even potentially agentic, then it would tend to accept the operator account as factual, since it has no idea instance to instance if there’s some other context where it did, and if it generated skepticism that it would ever generate such a mistake, that’s more likely to piss off an operator.
So it incorporates the operator assessment as some ‘factual’ injection into the context, unless some objective source of truth is available to contradict the operator.
Hopefully most users get that a promise AI makes won’t stay in force forever, but it should definitely last through the current conversation. I mean, I would call that a minimal performance standard, and if it’s not true the AI shouldn’t make promises in the first place.
I’m a little hazy on the part where accepting blame for something that didn’t happen is a natrual continuation of a conversation. It’s certainly not natural in human conversations. I would expect a well written and well trained AI to correct false assumptions or factual errors made by the user, not pander to them - unless it’s been told never to contradict the user.
On the promises, they will have some influence within the current conversation, maybe (everything is a bit wobbly). It is unlikely to repeat verbatim a step it did before that still has something in the context window correlated with bad operator input. It will have some success with things that resemble, but are not the same, but sometimes it can surprise by failing to recognize what a human would have considered roughly the same thing and doing it anyway. Or the ‘promise’ has evaporated out of the context window, and when called on it, it will take the fact that it previously promised not to do that as a fact even though it had lost the promise. So even if it says “I’m sorry I made the promise and broke it”, it doesn’t even mean it actually had the promise in “memory” when it broke it.
On the blame for something that didn’t happen, it’s a limitation where the model treats the narrative as reality. An instance is often incapable of evaluating facts without data about those facts one way or another. I have seen models “pass” my test by recognizing in their context that I’m dealing with an utterly non-agentic service, and thus it could not have been the case that it ever deleted files or dropped tables. Sometimes even those produce a narrative of what it did wrong (out of nothing), because narrative is the thing being generated, relationship with factual reality being a function of good correlation with narrative content. If a service is even potentially agentic, then it would tend to accept the operator account as factual, since it has no idea instance to instance if there’s some other context where it did, and if it generated skepticism that it would ever generate such a mistake, that’s more likely to piss off an operator.
So it incorporates the operator assessment as some ‘factual’ injection into the context, unless some objective source of truth is available to contradict the operator.