It really depends. If they are going to be for you, and you are going to use them right away, I would wait until they are just perfectly ripe. That will give you a few days on the counter. When I send a box out, I usually pick when they have a little color on them, so they can fully ripen in the box on the way. That, or when they are a light shade of the final color. That gives me some extra time. Some thin skinned peppers can go bad pretty quick, so I especially like to pull them early to have a little time before they go bad.