Up to you, putting starter in the fridge is optional and only recommended if you're an infrequent baker (though, there is some debate on this too, some people always use fridge starter)
But anyway, to answer your question, you leave it out until atleast that 6th day because you're trying to capture and establish your wild sourdough culture, it's vigor would be dulled in the fridge.