I wouldn't worry. I have a young bell pepper plant that had maybe 50 buds on it. When 3 of them set pods first and grew, all the others fell off. Just the plants way of saying this is how much I can handle right now.
Just to add also, you can see how healthy and green your plant is and the only...