Personally I find that there are tough breeds of both species. I feel it also depends how long or how many generations the strain has been gown in the local area so has gotten used to the conditions.
This last year I have had both Annum's and Chinense, most died over winter. But some of each have also proved to be much tougher than others.
e.g. most of my 7 pots died, my bhuts died, all my sweet annums died.
The interesting thing is out of all my plants, the ones that survived the best by far (over winter) are the ones that I had bought from a local nursery (a simple habanero, and a cayenne). As far as im concerned they are simply tougher because they have been bread locally for a period of time and have become more acclimatized to the conditions. Both of those plants continued to pod over winter, NONE of my others have done that.
They are the only two to keep their leaves, and continue to fruit over winter.
A strain suited to the local conditions I think is the most robust.
EDIT; in sheer physical robustness I think chinense is more robust..stockier, thicker, stiffer stems. Annums can be a bit lanky and not handle the wind as good etc. Physically a bit more fragile.
The bugs also prefer my annums over my chinense's