We have a very short growing season here, and I grow my chiles in pots because of that. When the temps start getting too cold, I bring them inside the house. They get a fan, extra light, and continued feeding until they stop producing. I doubt I'd have some ripe pods, such as the rocotos, if I didn't do this. So no, it's never too late, you just have to plan accordingly.