Why are victimless crimes considered crimes?

I was having a conversation with a friend of mine a while back and one of the topics covered victimless crimes. At first, I was confused. If there's no victim, why are they crimes? To figure things out, I did a little bit of research. Victimless crimes seem to fall under two categories. The first group contains any illegal activities that only bring potential harm to the perpetrator; the second group is anything that is completely harmless. Looking at the former crimes, I could see how they were outlawed at some point. As for the latter crimes, I simply do not see why anything that has no victim should be illegal. While I could understand the thought process of the former being banned, I think both should be perfectly legal.

Victimless crimes that are potentially harmful to the person instigating them are a tricky grey area. Most crimes of this caliber constitute illegal drugs of some sort. While I think there are many drugs out there that are extremely dangerous and no sane person should use them, there are also many others which have no detrimental effects, or very minor ones at that. For example, marijuana may eventually harm the user, but it's overall a pretty tame drug. I don't use the stuff and never plan to, but I don't see why someone who does, and who understands the health risks, isn't allowed to. I mean, we allow people to smoke cigarettes, which will eventually give you lung cancer and emphysema. People understand the negatives and smoke them anyway. If those are legal, why aren't other, relatively comparable drugs illegal? It doesn't make any sense, especially if the user comprehends the risks and voluntarily accepts them. Again, there is no victim here (besides the informed user, who isn't really a victim at all). And if these victimless crimes were made legal and regulated, imagine how much money would be driven above ground and pumped into the economy. 

The other kind of victimless crimes are those with, literally, no victim. Common examples, such as homosexuality, sodomy, and prostitution, all have an underlying sexual theme. Now, it is my personal opinion that, in the privacy of one's own home, anyone should be able to commit any consensual, non-harmful sexual acts. If two adults of the same gender want to sex it up (as gross as it sounds to me, no offense) they should be able to, as there is no victim. If a couple wants to experiment with different techniques outside of the mainstream, they should be allowed to get into sexual shenanigans, as once again there is no victim. If someone wants to pay someone else for a professional sexual experience in the privacy of one's own home, that should be legal, as yet again there is no victim. Assuming all of these situations are consensual, what's wrong with them? Yet, across the world, attitudes vary significantly for all three; some countries ban each one, while others allow sexual deviancy but not prostitution for some reason. Why is this the case? It's kind of weird, if you think about it.

If no one's getting damaged physically or psychologically, it shouldn't be illegal. And if a person understands the risks of self-harm surrounding an activity, it shouldn't be illegal either. People should be allowed to deviate and harm themselves all they want, so long as it doesn't hurt anyone else.