most of the damage caused by aphid feeding in peppers is due to virus transmission but can cause damage to your crop by their feeding also. the honeydew they secrete serves as a medium for the growth of sooty mold fungi which reduces photosynthesis. other diseases spread by aphids on pepper plants includes alfalfa mosaic virus, broad bean wilt virus, chilli veinal mottle virus, cucumber mosaic virus, egglant severe mottle virus, henbane mosaic virus, indian pepper v (sick of typing virus so v will have to do from here on in), marigold mottle v, peanut stunt v, pepper mild mosaic v, pepper mottle v, pepper severe mosaic v, pepper vein banding v, pepper veinal mottle v, peru tomato mosaic v, potato aucuba mosaic v, potato v's A and Y, beet western yellows v, carrot mottle v, and potato leaf roll v.
I believe the only impact on the actual fruit (apart from the sticky honeydew that Potawie pointed out) is one of aesthetics however the above diseases will impact on your plants ability to produce foliage, bud, blossom, fruit and then ripen fruit adequately. if it's your end of season and you aren't going to overwinter your plants just get the bugs off and don't worry too much about it.
my plants pretty much got smashed by aphids last year and most still fruited adequately, though I naturally still wanted more!!!