My notes on some of the most common species:
Capsicum Annuum - Fairly easy to grow, but of course it differs a lot from variety to variety.
Varieties: Bell peppers, Cayenne, Chiltepin. I love the black and purple varieties such as Peruvian Purple and Checkoslovakian Black.
Capsicum Baccatum - I find baccatum easiest to grow and this species include my favourite chili varieties. Fruity flavours.
Varieties: Hot Lemon, Aji Norteño and Omnicolor are some varieties. First two with great taste, Omnicolor is beautiful with white, orange, purple and red pods.
Capsicum Chinense - Hard to grow for me. Probably due to the Swedish climate. Chinense need a lot of light and heat.
Varieties: Martinique is the only variety I've manages to get ripe pods from myself, it made for a great salsa! Well known varieties are Bhut Jolokia, Scotch bonnet and Red Savina.
Capsicum Pubescens - Slow growing, sensitive. Also probably due to the climate. These can stand cold due to the "hair" but I think they are picky with lighting.
Varieties: Rocoto, Canario. I'm growing Alberto's Locoto this year, hope it turns out.
These are only my observations and I think you have to find the varieties that work well for you.
[Edit:] Talking of varieties, would be great if you wanted to answer the question on my blog today:
http://www.chiliez.com/