Aren’t there also tons of studies and math that show/prove they cant differentiate between instructions (e.g. from the company) vs data (e.g. that guy’s messages)?
Of course in any other application, keeping instructions and data separate is very important. Like an SQL injection attack is when you’re able to sneak instructions in where data is supposed to go, and then you can just delete the entire database, if you want. But with LLMs the distinction doesn’t exist.
Aren’t there also tons of studies and math that show/prove they cant differentiate between instructions (e.g. from the company) vs data (e.g. that guy’s messages)?
Yes, I believe that is the case.
Of course in any other application, keeping instructions and data separate is very important. Like an SQL injection attack is when you’re able to sneak instructions in where data is supposed to go, and then you can just delete the entire database, if you want. But with LLMs the distinction doesn’t exist.