Ants are good for gardens. They aerate soil, break down organic matter and reduce compaction. Since they are eating your aphids (I get the impression that they are since you mentioned on three occasions that they are eating them.), it's an added bonus.
Ants only become a problem when they start farming aphids on your plants (as mentioned above). If you notice they do start farming aphids on your plants, just hose them off or spray with insecticidal soap. It's probably not worth it to go balls out trying to eradicate them this late in the season.
If you're worried about aphids becoming a problem next season, just plant some flowers like marigolds or lantana or salvia near your plants next spring. They will attract predator insects (ladybugs and lacewings etc.) that will keep your plants clean all year long. As long as you have your aphid problem in check, ants are nothing but another beneficial insect for your garden.
Flowers keep ladybugs fed and happy all season long, regardless of fluctuations in aphid population. Plus they're much easier to apply than pesticides. Just stick a few in the ground and let them do their thing!