All good points so far. A few more big factors IMO...
1. What the deer are feeding on (a deer that has been eating corn for months is going to be much better than one that has been eating evergreens).
2. Shot placement, and how the deer is field dressed. Being careful to not puncture the digestive tract, bladder, and glands is key. This is why I always butcher and process my own. When you bring a deer in to get processed, you are most likely not getting that deer back. You are getting a mix of several, especially if you are getting sausage and/or burger. I have no interest in Joe Blow's gut shot deer that he couldn't find until the next morning.
3. Removing ALL the fat makes a huge difference.
All that said, venison is not supposed to taste exactly like beef. I absolutely love it, but you can do everything right, and some people still aren't going to like it.