Sex is a private matter and most understand that it is something that is not for unmarried people. Even though masturbation is not actually sex, it is sexual, so it is easy to assume that it must be wrong somehow.
We also know to keep our genitals private, so guys who masturbate, thus, do so in privacy, hiding what they are doing, which makes people again assume that something must be wrong since it is being hidden.
Then there is the unfortunate fact that many guys masturbate while watching pornography. Yet, at the same time people know that pornography is wrong: that it is a wrong to watch people being used for sexual gratification and violate privacy. In this case, it actually is wrong, but they are watching with their pants off and we tend to blame the physical (masturbating) over the spiritual (lust).
There is also the tendency to see things that are intensely pleasurable as being wrong. This is a confusion over the fact that most sins bring the sinner pleasure, so therefore it is assumed that anything causing pleasure must be wrong. Many religions take this to an extreme and declare that denying pleasure makes you more holy. "If you have died with Christ to the elementary principles of the world, why, as if you were living in the world, do you submit yourself to decrees, such as, "Do not handle, do not taste, do not touch!" (which all refer to things destined to perish with use) --in accordance with the commandments and teachings of men? These are matters which have, to be sure, the appearance of wisdom in self-made religion and self-abasement and severe treatment of the body, but are of no value against fleshly indulgence" (Colossians 2:20-23).
When you have an orgasm, there is a release of hormones that make you relax and at the same time the ejaculation relieves the pressure and lowers your sex drive suddenly. This sudden change in mood leaves some guys feeling guilty. I suspect that it is because these hormones are designed to make you feel close to your wife after sex, but in masturbation there isn't another person to focus your desire on and guys interpret the feelings as guilt.
The rumors that masturbating causes some health problems is more difficult to pin down. If we are convinced that something is wrong, then people look for justification. We think it can't be just a spiritual problem, so they look for physical reasons for telling guys not to masturbate. And guys fear that something they find compulsive and pleasurable must have a down-side somewhere; thus, they will tend to believe any rumor they hear.