My understanding and experience is something like this:
Once a pepper has reached its "final" color, it is ready. Some, like the one you have (some kind of cayenne), go straight from green to their ripe color. Many others go from green, to a second color (often orange) to their final ripe color. And yet others, especially ornamental varieties, go through several colors before finally reaching their ripe color. There are some exceptions, generally certain bells and ornamental varieties that may start out purple, yellow or whatever first and may go through or ripen to a cream or some other color, but to avoid confusion forget about those for now; just know that they exist.
Peppers picked at different stages will often have quite different flavors (especially at the two ends of the spectrum (very immature vs. very ripe). They will also have different levels of sweetness, heat, firmness, etc. The closer to "ripe" they are without actually being ripe, the longer they will last off of the plant. If they are ripe or past ripe when you cut them off the plant, it might be a good idea to just use or preserve them (freezing and drying/powdering are my favorite methods) soon if you take them off the plant.
When to pick is up to you. IMO, most are best fully ripe; they tend to get a sweeter, fuller flavor. Tabasco is the main exception I've found, which I find to get worse and worse as it goes through its colors (it's sharp and just keeps getting sharper...), so I pick it when it's larger but still light green or possibly as it's changing to its second color. Some of the varieties seem to not only get hotter as they ripen, but it seems to me like the *type* of heat actually changes (ie. a red-ripe tabasco pepper has a quick, sharp burn that goes away quick, while a green one has a slower onset but longer duration and feels hotter to me).
Even those peppers that change from green to their one and only ripe color will turn a noticeably darker hue within a week or so after you might have thought they have fully changed colors. This darkening of the color is what I use to decide when to pick my C. chinense species peppers,... I want them as ripe and sweet as possible, even if that means a slight sacrifice in heat (just add more!), so they get cut off the plant about a week or more after changing their final or ripe color. In my experience, jalapenos start going soft within a week or so of turning fully red, which is not exactly bad, but I prefer to pick them within three days after turning fully red for this reason (if I let them get red). That way they're sweeter, but still crisp.
One more thing... when you cut a pepper off of the plant, the plant will take that as a signal to produce another pepper. For this reason, you may want to consider cutting them off earlier, and ripening them off the plant in a paper bag with an apple or banana around room temperature. This will cause the plant to produce even more peppers if you viciously cut them off just as they're turning their ripe color. My preference is still to have them fully ripen right on the plant, but I can see the bag-ripening method as being a good alternative.