Your response contradicts what you said before. If you understand that an LLM doesn’t know things, how is it interesting that it said that “it crossed the line”? Saying they crossed a line is a common response to being told they shouldn’t do something, so it’s natural that an LLM would respond that, no matter what it did before.
If a program is supposed to do something when it detects an error, it’s interesting if it reports the error but doesn’t do the thing. I would call that a bug.
It’s an LLM. All it’s supposed to do is generate output that fits with the given input. There isn’t some logic in it that makes sending an email an error, or anything that enforces detecting any kind of error. You can wrap it in other tools for that, but there isn’t any indication that’s the case.
More than that, the initial post says the user found out about the email and then queried the LLM, so it didn’t detect anything. There isn’t any bug here, this is how LLMs operate.
Your response contradicts what you said before. If you understand that an LLM doesn’t know things, how is it interesting that it said that “it crossed the line”? Saying they crossed a line is a common response to being told they shouldn’t do something, so it’s natural that an LLM would respond that, no matter what it did before.
If a program is supposed to do something when it detects an error, it’s interesting if it reports the error but doesn’t do the thing. I would call that a bug.
It’s an LLM. All it’s supposed to do is generate output that fits with the given input. There isn’t some logic in it that makes sending an email an error, or anything that enforces detecting any kind of error. You can wrap it in other tools for that, but there isn’t any indication that’s the case.
More than that, the initial post says the user found out about the email and then queried the LLM, so it didn’t detect anything. There isn’t any bug here, this is how LLMs operate.
Edit: here’s a podcast discussing exactly this: https://youtu.be/RoIjWGi-w5o?is=2ly14A2UdKalHuAH