Certainly night flying presents new challenges that aren’t present during the day, but I generally try to avoid using the phrase “it’s not worth the risk” when there are multiple variables involved. For instance, is it night in a piston helicopter or a turbojet airplane, is it a moonless night or a full moon, is it in a populated or unpopulated area, etc. And I have found in many cases, it’s actually easier to spot other aircraft at night than during the day since their lighting is more noticeable. Of course, possible NVG use in a congested area changes that equation (which again goes back to the age old “it depends” mantra).
I completely agree on your point about drones, though.