Definitely not harmless. I took care of them with a Neem solution, but they are so tiny and hide so well that it takes quite a few applications to get them under control (especially since we've had a very rainy summer). Make sure to spray well inside the flowers, on any new and forming growth, and on the underside of leaves; that's where most will be. There are other pesticides you can use, but I don't have any experience with them, so maybe other users can give you some pointers there.
Depending on where you live and where you're growing, it's also possible that the population of thrips is being kept under control by its natural predators (thrips predator, lacewing, ladybugs, etc.)
If available to you, and not present already in your grow in adequate numbers, you can also purchase these as a pesticide alternative. Can't really recommend this, though, since I've never tried it myself.