I would totally wear that outfit [the first one, not Mr Rogers's]. I actually had a top for work almost exactly like that, but black, and with short sleeves. Then I got pudgy and had to donate it.
I haaate sleeves on anything I can't easily remove if I want to. Even if I wear a longish-sleeved shirt I always turn the cuffs up past my elbows. Also sweater sleeves tend to be thick, bunchy, and constricting in annoying unpredictable ways. Also also, if I'm wearing one as a second layer over a short-sleeved shirt or sommat, I can't stand sweater material on my skin, it's always itchy or just weird and aggravating.
Edit: Estelore brings up a good point as well. For much the same reason as guys shouldn't tuck in their shirts, guys also shouldn't wear something to emphasize what they just don't have. But to be fair, even if you do have bigger boobs than I do, I don't really want you to emphasize them either. :/
And Brian, fair warning, I will fight you over how completely un-lame vests really are!
Also, I can kill you with my brain.