you have to look closely at your plants each day...if you see some shiny spots on the tops of the leaves, look under the leaf on top of that leaf and you will find aphids...
the shiny spots on the leaves are droplets of honeydew the aphids produce....
if you ever let them get out of control thats where the trouble starts...check out post 175 of this thread....
http://www.thehotpepper.com/topic/14738-mels-growing-adventures/page__st__160
I started with one aphid on day one and at day 30, there will be over 8000....IMO if you don't catch them before day 21, you are screwed...