It doesn't appear to be limited to any specific culture insofar as either region or religion, as much as a personality type as far as I've been able to tell. Understanding that we all have times and reasons we don't (believe me, in some of the countries I've been in, my cock was cleanest thing in the restroom) most who refuse to wash their hands after using the restroom appear arrogant. As if, for some reason, their immune to the practice. Surely in this day and age it couldn't be ignorance...could it?
I am personally bothered by it - not that I'll think less of you - not at all! I will, however, likely NOT shake your hand. Especially if you've just come from a stall in the men's room, and walk out without stopping at the sinks. Ghastly.
I'm also curious if those arrogant enough to walk out of the restroom without washing their hands are arrogant enough to discuss it under the scrutiny of the public eye:
Do you wash your hands prior to leaving the restroom?