"Julius theorizes that after triggering an initial burning sensation, capsaicin can have the longer-term effect of desensitizing the TRPV1 channel, as well as the nerve fiber at large, silencing these pain sensing nerves."
So one can build up tolerance, at least in theory.