I recently woke up to this email from Digital Ocean:
Hi there,
On December 4, 2019, we identified a bug in our systems that affected how Cloud Firewalls were applied to some Droplets. From November 20, 2019 until December 5, 2019, Cloud Firewall rules were not enforced on some of your Droplets.
We identified the root cause of the issue and reapplied all Cloud Firewalls. They are now operating normally…..
In other words – the Digital Ocean firewalls that restricted access to my cloud servers were not actually enforcing those restrictions, even though the web control panel showed me that they were.
I have had situations like this happen in the past, and have started using differential nmap scans to check for issues like this. Using a service like HackerTarget, you can schedule nightly nmap scans against your public IP space and have it alert you when there is a change from the previous scan; this is predicated on the idea that you know your public IP space, which may not always be the case.
Another way to look for misconfigurations like this would be to schedule an osquery query that looks for open connections to a local service from an external/routable IP. For example, the following query looks for open connections from a public IP to the local system, on port 3389 (RDP), 5985 & 5986 (WinRM Remoting).
SELECT process_open_sockets.local_address,
process_open_sockets.local_port,
process_open_sockets.remote_address,
processes.path, processes.cmdline
FROM process_open_sockets JOIN processes using (pid)
WHERE process_open_sockets.local_port in (3389, 5985, 5986)
AND process_open_sockets.remote_address NOT LIKE '10.%'
AND process_open_sockets.remote_address NOT LIKE '172.16%'
AND process_open_sockets.remote_address NOT LIKE '192.168%'
AND process_open_sockets.remote_address NOT LIKE '127.0.0.1'
AND process_open_sockets.remote_address NOT LIKE '0.0.0.0'
AND process_open_sockets.remote_address NOT LIKE '::'
AND process_open_sockets.remote_address NOT LIKE '0';
The underlying assumption of this query is that since they are well-known services/ports, there will be almost constant connections from automated scanners & toolkits. Any results from this query should be followed up on, as it means that these ports are publicly accessible. From personal experience – with this query, I have found (previously unknown) production Windows servers with RDP publicly accessible. (!!!)
Replace the RDP/WinRM ports with any other relevant ports for your infrastructure – 9200 (ElasticSearch) / 6379 (Redis) / etc
h/t to @gepeto42 for the original query idea – https://twitter.com/gepeto42/status/1128383095428743172
Defensive Depth 