Plus one on dragonsfire's and solid7's posts.
No heat source at all might be tough, but you wouldn't likely need extra heat frequently. Something you could turn on temporarily when the temperature drops could get you through the cold spells.
Harbor Freight sells propane tank top heaters that get pretty decent reviews and aren't too expensive.
https://www.harborfreight.com/15000-btu-tank-top-propane-heater-63073.html
That seems a little large for a 4' x 4' space, though. Maybe a backpacking canister stove on simmer would work.
A (very) small wood stove, something like a backpacking wood stove, might work, though you'd have to babysit it through the night.
Placing jugs of water around the interior of the space can buffer the inside temperature against outside temperature drops. And just an old-fashioned incandescent light bulb can take the chill off. Especially if you can cover the structure with a blanket or something.
I have an 18' x 30' frame I plan to set up and cover with plastic before fall and I plan to use the HF propane heater and a wood stove. I live in zone 6b/7a, I think. It has changed over the years I've lived here. I remember one winter it didn't get out of single digits for a solid week. But that was decades ago and I'm counting on climate change to do me a solid.