Those temps won't kill them by any means, that will only happen when you get closer to freezing. The real problem with low temps is that it has a tendency to slow or stunt plant growth especially with younger plants. a couple of days of low temps [40's] won't cause this, but extended periods of low temps will. I had a habanero last year that I planted out early, it spent a couple of weeks with lows in the 40-50 range. The main stem pretty much stayed the same height, and didn't grow much taller, it only branched out from there. The other ones I planted later reached full height. The biggest thing with peppers is they don't like "cold feet", and when they experience it can cause stunted growth.
One option you have is if you bring them in at night and just put them back out in the morning you should be fine.