I ended up with a decent crop for the year and really didn't know a thing about growing peppers. It had to because of luck. My ground consists of decent (not hard clay) dirt. However, I had a swimming pool in the area, so there was a layer about two inches thick of sand. I got a huge load of topsoil, and it was about three inches thick. (This is not a freeze-thaw thickness, though). It still was not high enough, so I hauled dirt from a local cemetary. Some of it - but not most - was decent but it had quite a bit of clay. I spread about three inches of this. While I did try to rotary till it, it still ended up - from the top down - as clay, top soil, sand, good dirt.
Early in the year, we had downpour after downpour. But even after two inches of rain, in two days I could walk in the dirt - the water soaked through. Then we had a drought. I watered the garden somewhat, but after a while it seemed futile. Yet, when I dug up my potatoes and got down about five inches, the ground was extremely moist. The only fertilizer was those blue Miracle Grow that is mixed with water and applied with a garden hose.
My guess it is not your soil or fertilizer, but lack of sunlight. I had a plot of ground down the side of the yard I decided to sow various and sundry flowers and plants in. The upper half was in direct sunlight most of the day, the lower portion anywhere from 3/4 to 1/2, because it was shielded by trees. The plants in the shade were half the size of the ones in the sun. If you are going to go to containers, I would find a way to raise them up at least three feet, a little more for the row cloest to the fence.