They don't even need to be fully ripe. They will continue to ripen after your pick them. It really depends. If you want a fresh pepper to last a little longer, then it might be ideal to pic it while it's mostly ripe. After my plants get full of peppers, they stop flowering. So it's best to harvest right away at that point if they are ripe.