You are right, since the pepper taste is going to be so strong you won't be able to tell what oil you used anyway!
I simply use the cheapest oil, or what I have in the cupboard.
I treat this like cooking with wine, I use Olive Oil for things I would use Olive Oil with naturally and other oils depending on what their use is going to be...I would not want peanut oil in my salad, but I may want to make a vinaigrette using hot pepper oil made with Olive Oil. It is all about how you plan to use it or your personal taste preference.
You are right, since the pepper taste is going to be so strong you won't be able to tell what oil you used anyway!
I simply use the cheapest oil, or what I have in the cupboard.