I wonder if there's something genetic to what you taste, sort of like there's a gene some people have that they can taste something in spinach or broccoli that is bitter to them, but not for other people.
Or, it could be that growing up around it, you associated the smell with food before you associated it with smelly armpits (the description of rotten onions is how armpit odor smells to me). I can eat stinky blue cheese, even if it smells like stinky feet because I was lucky not to smell stinky feet before learning to like the cheese. But, I can't stand even the mildest goat cheese because it tastes like male goats smell to me, and that smell is associated with some bad habits they have.
I'm just wondering if your neighbors will hate you if you grow durian and they don't like the smell. Then again, if you want them to keep their distance, it might be the perfect thing to grow.