I happen to be one of those people, unfortunately.
Whenever I go out for a walk, i'm always listening to music. What I find is it enhances the experience even more, it adds a further sense of mystery and enchantment. It's not really a distraction thing.
What I listen to isn't some loud, brash rock-pop or anything. That has a time and a place. What I tend to listen to adds an atmoshpheric vibe to everything. Perhaps it's just the way I react to music (it's extremely powerful for getting at raw emotion) but that's how I feel.