The problem isn’t AI gaining intelligence, it’s allowing AI to have agency and then it gets an potentially dangerous prompt that leads it to hack into, just for the sake of being funny, a bunch of places to delete the entire stock market. As funny as it would be it would lead to a complete economic collapse. I’d actually feel safer if it did have actual intelligence. We’re talking about a technology where if you have broken unit tests (software term meaning a certain set of tests the code has to pass to be considered functional) and you tell AI to “fix the unit tests” there’s a non-zero chance of it just deleting the tests and saying it’s fixed.

AI is dangerous not in the way movies present it where it gains sentience and wipes us out. It’s dangerous in the opposite way where it doesn’t have intelligence and wipes us out because it doesn’t understand it’s doing something that’s going to kill us.