Making people uncomfortable is not solely the problem of that person, no, but it would be entirely inappropriate for the glasses-wearer to respond by punching the person making them feel uncomfortable.

I don’t really understand what you’re trying to say here. “Being made to feel uncomfortable” is on a completely different level than being physically assaulted.