My take on it is that software able agree it did something wrong
This would make sense if it made an independent subjective assessment and arrived at the same assessment of the user. But that did not occur, it simply incorporated the user’s subjective assessment as a new piece of the narrative and continued from there. If the user had expressed significant satisfaction that it took care of that email all on its own, it would have also agreed with that. In circumstances like this, the LLMs are generally set to agree with the operator in any subjective matters.
One could imagine the context having something like “before taking any action, if the user expresses dismay at the result and this would result in an agreement with user on poor assessment, then don’t do it”, but then if that were actually honored, then you pretty much disabled most all actions, as the llms are going to agree with any poor assessment.
This would make sense if it made an independent subjective assessment and arrived at the same assessment of the user. But that did not occur, it simply incorporated the user’s subjective assessment as a new piece of the narrative and continued from there. If the user had expressed significant satisfaction that it took care of that email all on its own, it would have also agreed with that. In circumstances like this, the LLMs are generally set to agree with the operator in any subjective matters.
One could imagine the context having something like “before taking any action, if the user expresses dismay at the result and this would result in an agreement with user on poor assessment, then don’t do it”, but then if that were actually honored, then you pretty much disabled most all actions, as the llms are going to agree with any poor assessment.