I love growing food-producing plants in general, especially exotic and/or heirloom types that aren't found in grocery stores, that's why I have a lot of non-chile plants, fig trees, paw paw trees, miracle fruit, currants, kumquats, pink blueberries, Meyer Lemons, heirloom tomatoes and fruits, herbs, etc. With chiles, I love the flavor and heat, nearly all my favorite foods involve some type of hot pepper, and grocery store peppers are usually pretty terrible. The jalapeños are never sold ripe, have barely any discernible heat, and taste like bell peppers. The serranos are big and fat and don't look like or taste like serranos. The habaneros are orange and usually half rotten. The finger hots taste like grass and have no heat. The bell peppers are way too expensive. And the unripe Hungarian hot wax suck too. The only decent peppers sold fresh here are ancient sweets, mislabeled as "pasilla", and they're sweet peppers. So I kind of have to grow my own if I want anything decent and uncommon. Plus it's fun starting something from a tiny seed and raising it to a large, food-producing plant. It's rewarding and gives me something to do.
I don't find growing peppers all that hard compared to a lot of things, so I can't help you there.
Then again, the plants that are supposed to be difficult usually do the best for me, the ones that are supposed to be easy do the worst for some reason...