Thanks for all the information everyone!
So I've been watering my chilli plants way less, and I have a layer of diatomaceous earth on top, which really seems to help.
I still see them, but not many.
I'm also using stickies constantly, which seem to keep their numbers down.
I think I...