Do what you like. And I've ridden without the gear when I was young and invulnerable so I understand what it's like.
As far as "restrictive" and "uncomfortable"; neither applies, in fact as I've pointed out, the gear actually makes the ride more comfortable and thus more enjoyable.
So If you choose to not wear the gear, that's all you do, is choose not to wear the gear. No need to try to rationalize it.
And no, it doesn't make one less intelligent to make that choice, and I won't argue the converse either.
It's not choosing freedom or any other such noble pursuit, it's simply not wanting to wear the gear.
At work, I wear safety glasses, hard hat and safety toe boots. It's a condition of employment, but I would wear them even if it wasn't.
On the bike I wear the gear because I choose to (the helmet is required, the rest isn't).
The BIG problem I have with people who don't choose to is that they tend to try to make assertions that wearing safety gear somehow makes the rider less safe

Or they have to make excuses about comfort and freedom that are also misguided.
If you don't want to, just say so and leave it at that.
And if someone thinks you're a dumbass for making whichever choice you make...................well, I stopped giving a sh!t what other people think somewhere around the second grade.
I CHOOSE to make a risky behavior, hobby, choice, (addiction?

) a little less likely to seriously f*ck me up and leave my family without my adored presence.
Because I WANT to.