A wise man once said, its not the size of the pepper that counts... oh wait, that applies to something else...
A plant that is not fully grown, will produce smaller pods. They will also produce smaller pods late in the season, and early in the season as well.
Alot of people here recommend pulling the first round of pods to make the plant put its energy into growing. I tend to let my plant make up its own mind. If it wants to fruit, let it. If it wants to drop its buds, thats fine too. Especially my chinenses.. they seem to be more particular.