Until they look like this. It`s warm and dry here (San DIego), so I let them just start to wrinkle a bit. It concentrates the flavor, makes them sweeter and they become a lot hotter.
Leave them on the plant until you just can't stand it any longer and have to eat one, or until the flesh barely starts to soften at the end of the red ripening stage, whichever comes first.