I usually take 'em off, but I'm not adverse to wearing (clean) shoes in the living areas of my house if need be - but never in bedrooms and the like.

When I'm just walking around my house, I wear (fluffy) slippers.

When you think about what your shoes are subjected to, it's kinda yuck to be dragging that through your house... but there is a point where you're worrying about it too much.