i wouldn't. keep fertilizing on a regular schedules. peppers need green growth to set fruit just because of their branching structure.
also, yeah, leaf size is partly genetics, but at the same time, I have a couple of 7 pots growing in my backyard from various sources and their leaf size...