Ya, most are OK for adding some heat but they are lacking in flavor. I wouldn't call them earthy, more bitter and low on flavor. I personally describe pequins/tepins etc. to be earthy but in a good way
There are always some ornamentals that surprisingly do have a good flavor, but generally they are bred for looks, not taste.