yeah I was wondering about this too with mine a little while ago, I trimmed them off my Bulgarian carrot, the first one to get flowers and it grew/branched out really quickly after that.. then most of my other plants starting getting buds everywhere, and the BC was plenty big for now in it's 1 gallon pot (about.. 20" maybe not really sure, I clipped it when it was from 10-15".. it grew the extra 5+ inches really fast) so I just let them all go, too many tiny buds to try to pick off.. and now it is cool, I have about... I dono 20+ peppers forming now about a month later...(and actually that pepper in my avatar is the first one... I just checked tonight and it is about.. I think an 1"-1.25" long now, pretty cool stuff (Bulgarian Carrot)
with that said, I have 60+ plants in party cups i need to transplant once Agway gets Perlite in, hopefully this thursday, and the Cherry Peppers and a few other fast growing pepper plants have buds and flowers on them.. since I will be repoting them, and I would personally like them to get a little bigger (the cherry plants are about 10" tall, but they are cherry plants, so they have a lot of growing to do), and I have 2 months or so before they go outside.. I will clip all the flowers off that I see when I transplant them, but after that I will probably just let them go.. too much hassle to keep after them, and they can do what they want, but I found that the initial picking of the buds promotes a lot of fast new growth, at least in C. annuums, not really sure about all the other kinds, I will probably do the same if they do it when they are small, just because it is easy when there are only a few, and I would like to get those slower growing plants boosted up a little.. and I don't need pods right now lol
again I go into more detail than I planned, but bottom line, it's personal preference really and like everyone else said, what you plan to do with them