I pull them off, especially when they're pinning the cotyledons together. If you don't and if it doesn't free itself anyway, the sprout will still grow out from between the trapped cotyledons. They just usually grow better without the stuck seed coat so they can open up to the light and you can usually remove it without any damage. I just sort of pin the cotyledons from below the seed coat so if it tears it doesn't tear off any more than what's stuck in the seed coat. Usually they come of without tearing, but when they do tear it doesn't make any significant difference to the sprout so I'm not too concerned about it. You can also do things like wetting the seed coat with a drop of water to make it come off easier, but I don't really think it's worth even the fairly little time it takes.
Oh, one thing you can do is squeeze the seed coat from the edges, not the flat side, when you remove it. Squeezing from the edges will open it while squeezing from the flat sides will pin the cotyledons and make it more likely to tear the ends off.
What kind of peppers are they?