They’re not made with actual wood or conventional building materials, and fall apart so easily.

I’ve seen tons of prefabs being made now out of solid wood, just like a normal house. It’s just that they segment the usual house design into one that can be modularly attached/expanded, and position beams such that they can be directly attached to one another at the ends with little effort. If your prefab isn’t made of actual wood, then that’s not a good prefab, nor is it the kind of prefabs I’m talking about here. I agree those shouldn’t be the norm, of course.