Col, whether or not it is legally regarded as an invasion of privacy, that is the way people here are treating it. The problem is that such drones represent a hazard in flying over a populated area. If some of these drones are as large as have been described, a falling drone would be heavy enough to kill a person or at the least, cripple their unfortunate target. That is part of the reason why we are not allowed to shoot them down... to avoid collateral damage.
It probably doesn't help that recently, some commercial delivery services have started trial runs of drones as a package delivery device. There is no way to know if a night-time drone is delivering something or spying on something. In either case, I predict that there will eventually be some kind of accident and the clamor will rise for drone restrictions.