Advisory: put IOT devices on a separate LAN/vLAN for better security

[NeverDie]

A little more than one year ago the FBI recommended that everyone put their IOT devices onto a separate LAN because.... that cheap LED lightbulb with built-in wi-fi can be a penetration point for hackers to gain access to your home network.
https://www.zdnet.com/article/fbi-recommends-that-you-keep-your-iot-devices-on-a-separate-network/

I wasn't aware of the FBI advisory until now, so I'm a little behind in setting up a solution. A brute force solution would be to setup a completely separate LAN for IOT, but a lot of savvy people seem to utilize vlans for network isolation instead.

My current plan: run pfsense on a VM to act as a firewall/router and replace my current dumb ethernet switches with a couple of inexpensive netgear GS108Ev3 managed ethernet switches to configure the vlan. Apparently I can also run a security program (perhaps Security Onion 2. Any better suggestions?) that will scan incoming packets and do protection and/or detection of suspicious packets , but I'm not sure what kind of latency that might introduce or even whether there's enough added protected to be worth the effort. I've read some people use the Ubiquiti Dream Machine as an all-in-one to do these things, but I checked the Dream Machine specs and found that it's running on just a not very powerful ARM CPU. So, I'm guessing the same kind of security packet inspection software inside a vm powered by a decent Intel/AMD CPU should be no worse in terms of performance. That way I'd be free to choose whatever the best security software might be rather than be locked into whatever software the Dream Machine developers happened to pick.

Well, that's my plan. What hardware/software are you all using to isolate your IOT devices and provide good overall security?

[hek]

For the paranoid. Just make sure to mitigate the VLAN hopping exploit risks.
https://cybersecurity.att.com/blogs/security-essentials/vlan-hopping-and-mitigation

[monte]

@NeverDie don't run pfsense or other firewall in proxmox or other virtualization. I tried it and it's a mess. You can buy cheap $100 Dell or HP compact office desktop and add network card and run firewall on that, if you're on a budget. But in any case firewall as the gateway to your network should always be standalone hardware.
I am not saying that your setup in proxmox won't work, just think for when you'll want to restart your proxmox, or do any maintenance on it.

[evb]

@NeverDie my setup consists of the mandatory modem/router from my internet provider with a Zywall 110 behind it and then a managed NETGEAR switch. The provider's modem/router acts only as a conduit.

All my wifi enabled IOT devices like the heating, the energy Smappee, the sonoff modified ESPHome switches, etc are connected via a wifi access point to the specific IOT VLAN.
The MQTT VM server for the MySensors environment is also connected to the same VLAN.

Specific rules in the Zywall are permitting internet access for specific IP's on the VLAN. For example the heating, Smappee, the rituals dispenser, ... have internet access because they are cloud based vendor applications.

The controller HomeAssistant is running in the normal LAN and has a specific firewall rule to allow access to the VLAN.
Another rule is allowing only my desktop computer to access the VLAN for admin tasks like updating devices, etc

The question is all about how far you want to go in protecting your network.
There are still weak points in my setup. You can say that each firewall rule is a potential weak point...
For example the smartTV and the Bose Touch speakers are in the normal LAN because I'm playing video and music from my NAS. But they are also IOT devices...
Because I have touch screens with the home assistant (HA) GUI, the home assistant sits in the normal LAN.
I want also be able to consult HA on my smartphone, etc.

It's like securing your home, make it more secure than your neighbor and they will give up and visit him

[NeverDie]

As only just recently reported, even the Dream Machine got hacked:
https://it.slashdot.org/story/21/03/30/2057237/ubiquiti-massively-downplayed-a-catastrophic-security-breach-to-minimize-impact-on-stock-price-alleges-whistleblower

and that's a device meticulously designed to not be hacked!

If even your security appliance may be insecure.... it's persuasive to have a decent air gap between whatever is vital and whatever is not.

It may (?) also be an argument in favor of ditching IOT wi-fi in favor of roll-your-own radios that no one in the world is likely to understand but you, since IOT wi-fi devices can skirt around whatever defenses you may have put around the main gateway.

[mfalkvidd]

@NeverDie reminds me of this quote

Security software is not necessarily secure software

[skywatch]

@NeverDie How can a system of radio connected systems be 'air gapped'?

I only use wifi for my phone and even then it is 802.1x PEAP with radius server as authenticator. Any phone can be used to hack wifi or bluetooth so those are key areas to consider before using them. I guess that encryption and signing will help keeping things safe in this domain for a while yet.

Even with military budgets things still get hacked and leak data. At home we can only do our best!

[Puneit Thukral]

@NeverDie creating VLANs and restricting access of the IOT devices to must have access is the method I follow.

[NeverDie]

@skywatch said in Advisory: put IOT devices on a separate LAN/vLAN for better security:

@NeverDie How can a system of radio connected systems be 'air gapped'?

I suppose you could run anything critical using only wired ethernet on a network that's isolated from anything that's wireless. By "critical" I mean anything fungible, like identity and access to financial accounts. With those out of reach, I can't think of anything that would keep an attacker interested other than possibly malice or very limited voyeurism or perhaps casing for a physical burglary, all of which strike me as highly unlikely.

[Thomas Weeks]

FBI recommendation or not.. Never trust closed source, non-human (IoT/embedded) systems on your human network. Take most any Chinese IP webcam or similar device and set it up on a sniffer (wireshark or tcpdump) and you'll see many of these devices "calling home" to China (C&C?) and even exfiltrating your WiFi passwords. Not cool.

I've been running a near-edge IoT DMZ for years now. All Bluray players, VoIP terminals, and even network printers (if you set up a reverse established related rule) should be pushed outside your human network.

T.Weeks

[OldSurferDude]

@NeverDie You going to need an NTP Server. You can make one with an arduino nano, gps module and LAN module for under $20.

[petter]

as noted before you should always move IoT stuff on a own VLAN, for security purposes you should always segregate traffic as much as possible and create different security zones.
This will allow to control traffic flow with firewall rules and other systems such as IDS/IPS.
The Firewall rules controls what traffic you allow between the zones, and an IDS will control and verify that traffic. They can scan the content of a data packet and look at the content of the message or recognize applications (such as SSH over port 80 as an example)

back to your question, yes PfSense is a great "enterprise grade" firewall which gives you the toolset you need such as:

• Firewall rules between zones/subnets
• 2 different IDPS systems (Snort & Suricata)
• DNS filtering & interception
• RADIUS server for mac filtering, 802.1x EAP-TLS etc.
• IGMP proxy and mDNS services for stuff like Sonos speakers etc.

so the firewall gives you the ability to control and verify the traffic, however it offers no correlation, intelligence and management. for that you use something called a SIEM (Security Information Event Management) A SIEM will capture all the logs from your firewall, switches, endpoints, mirrored traffic etc. and do correlation and analysis.

let say you have an outgoing HTTPS connection which you allowed in your firewall, (your IDS wont be able to analyze the content unless you decrypt the traffic which may break stuff). the only thing your IDS see is $&$#$&%^%*%$%^ a.k.a garbage. with a SIEM you can get open source threat intelligence etc. which will generate an alert IF a connection is made to a compromised IP/domain. A great example on a open source SIEM is Alienvault OSSIM. its an all-in-one and easy to install. Im running it as VM's in my system, together with PfSense (also VM) and some other firewalls.

If you do choose to use VM's you should use ESXi as a hypervisor. this is free, very reliable and it is what most businesses are using in their server room or datacenters. on my PfSense VM I get about 800-900 Mb/s throughput with 1 IDS enabled, so if you set it up correctly you will get the performance you need.

[evb]

@petter, When I read your reply, I notice that you are at home in this security matter.
I think many readers, even on this technical forum, are already dropping out when reading so many jargon terms

As I said, how far do you want to go for your home security?
Already, with my 'simple' setup (see above), I am the only person at home who can fix it if there are 'internet' problems for the other family members.
Once everything is set up, there is the regular maintenance in the form of updates. If updates have breaking changes, then there is extra work involved to reconfigure everything, etc.

In these covid times I had to face the facts, what if I drop out, what will my family members do?
I have several colleagues who also have security measures set up at home and are also SPOF or Single Point Of Failure....
I have also thought about it for my MySensors network.
Actually, that is quite a complicated setup for an outsider: sensors connect to a gateway, then, in my case, an MQTT server and then a home controller (Home Assistant), all running in VMs on the NAS.
This is also where I am the SPOF....
And the family members are already starting to rely on Home Assistant. The kids have named it Lexa

So my advice, provide basic security, but don't overcomplicate it....

[NeverDie]

@evb said in Advisory: put IOT devices on a separate LAN/vLAN for better security:

@petter, When I read your reply, I notice that you are at home in this security matter.
I think many readers, even on this technical forum, are already dropping out when reading so many jargon terms

As I said, how far do you want to go for your home security?
Already, with my 'simple' setup (see above), I am the only person at home who can fix it if there are 'internet' problems for the other family members.
Once everything is set up, there is the regular maintenance in the form of updates. If updates have breaking changes, then there is extra work involved to reconfigure everything, etc.

In these covid times I had to face the facts, what if I drop out, what will my family members do?
I have several colleagues who also have security measures set up at home and are also SPOF or Single Point Of Failure....
I have also thought about it for my MySensors network.
Actually, that is quite a complicated setup for an outsider: sensors connect to a gateway, then, in my case, an MQTT server and then a home controller (Home Assistant), all running in VMs on the NAS.
This is also where I am the SPOF....
And the family members are already starting to rely on Home Assistant. The kids have named it Lexa

So my advice, provide basic security, but don't overcomplicate it....

I get what you mean, but it's nonetheless helpful to at least have a target to aim for. Ideally there would be something entirely turn-key that one could either just buy like an appliance (as Dream Machine tried to be) or else download and run, but AFAIK neither is available yet. The problem is that quality matters, and the consumer marketplace tends to attract junk, so I'm not sure it will ever get sorted out like it should be.

[NeverDie]

@OldSurferDude said in Advisory: put IOT devices on a separate LAN/vLAN for better security:

@NeverDie You going to need an NTP Server. You can make one with an arduino nano, gps module and LAN module for under $20.

Is an NTP truly essential?

[NeverDie]

@petter said in Advisory: put IOT devices on a separate LAN/vLAN for better security:

If you do choose to use VM's you should use ESXi as a hypervisor.

Is ESXi essential? Basedon @monte 's recommendation above, I just recently made the leap from ESXi to Proxmox, and so far so good, including the ability to migrate VM's from one physical host to another. A bit of a learning curve, but so far it seems rock solid. The only downside I've noticed compared to ESXi is that booting from a USB isn't recommended, but that's minor.

[Ikes 72000]

I have 2 hp servers at home running esxi from an SD card, and it has been working fine for 2 years now.
I also tried proxmox for a few months, but encountered some stability issues and bugs.
Both solutions have pros and cons, but now I clearly prefer esxi.
Datastores and network are easier to manage and it is very easy to passtrought hardware to virtual machine. All the hard disks of my servers are managed by VM (Open Media Vault, Raid 5 software), without any problem.
I also use a Pfsense VM to manage all my networks, with different VLANs. It is very easy to manage VLAN under esxi with vSwitch.
In the next few weeks, I will be switching to OPNsense.
I chose this solution to mutualize the material resources, knowing that this solution can pose problems in certain cases.

[NeverDie]

@Ikes-72000 said in Advisory: put IOT devices on a separate LAN/vLAN for better security:

Datastores ... are easier to manage

Yes, I can believe that: https://forum.mysensors.org/topic/11607/best-pc-platform-for-running-esxi-docker-at-home/55?_=1617658580974

[monte]

@NeverDie said in Advisory: put IOT devices on a separate LAN/vLAN for better security:

Is an NTP truly essential?

I use my pfsense box as NTP relay and pass it's port in firewall rules for unsecure subnet. I don't think it can be as much a problem in home networks.

[NeverDie]

Originally I thought: why not just read the BIOS clock on the motherboard? Well, I suppose that would work if there were only one physical PC. For coordinating backups and file date stamps among multiple PC's, I can see how an NTP server might be helpful. I'm still doubtful absolute accuracy on a home network matters for that, as long as all the PCs sync to the same time. Then again extreme accuracy would cover all possible edge cases, including ones I can't even imagine. I could just turn the GPS clock on and forget it...forever, with no NTP port on the gateway firewall. So, after thinking it through while writing this, I can finally see how for $20 it might be worthwhile after all. For a home network you'd only need one, and it would last a lifetime.

[monte]

@NeverDie I might be missing something. What is the problem exactly to get time from the internet? Sure you may not need precise sync, but why bother yourself with GPS clock, or setting clock by hand, if every OS can pull time from public NTP severs?

[NeverDie]

@monte Good question! I'm completely new to security hardening, and so I just assumed that the guy who posted the GPS NPT suggestion was aware of some threat. But if you aren't aware of one, then maybe there's no reason for it.

[Thomas Weeks]

@NeverDie I wouldn't.. check out this dedicated/embedded hardware.. used in many pieces of high end network appliances:
https://pcengines.ch/apu4d4.htm

T.Weeks

[NeverDie]

@Thomas-Weeks said in Advisory: put IOT devices on a separate LAN/vLAN for better security:

@NeverDie I wouldn't.. check out this dedicated/embedded hardware.. used in many pieces of high end network appliances:
https://pcengines.ch/apu4d4.htm

T.Weeks

You wouldn't... what? The pcengine you linked appears to be a 4 port router. What should I be learning/realizing from or doing with that piece of information? Is it meant for creating vlans or is it meant for getting the time from an atomic clock somewhere on the internet? Or something else?

[monte]

@NeverDie that's not just a router it's a an x86 motherboard that you can install pfsense on. I think he meant, he wouldn't install pfsense in vm
It is also fairly priced, I have one of their products before, but didn't check them recently, that's a great choice for self-built firewall.

[mfalkvidd]

I have an apu4d4. Started with Esxi and Opnsense in a vm, but esxi crashed when I was running a lot of traffic in opnsens. Instead I installed Debian on it and run just a basic nftables firewall. I run Proxmox on it as well. Here are performance benchmarks:

[monte]

@mfalkvidd you installed proxmox on apu4d4? I wonder why opnsense is so much slower, can it be some misconfiguration? Did you try pfsense, I'd like to know if it is any faster.

[mfalkvidd]

@monte yes I did.

opnsense is so much slower because of BSD. pfsense has the same problem. I don't know the details, but it boils down to some important part of the BSD networking stack being single-threaded. It seems to be a well-known problem in the router world. I applied multiple tweaks to the BSD kernel, but they did not make any significant difference.

[monte]

@mfalkvidd one more question. Did you measure throughput WAN-LAN or LAN-LAN?

[mfalkvidd]

@monte LAN-LAN

[NeverDie]

Two things convinced me to have a box dedicated to pfsense (or whatever I end up using):

1. @monte 's earlier advice on the subject, and
2. The need for it to keep working during a lightning storm, during which I typically unplug any expensive or delicate machines but also during which I still want to maintain wifi internet access. Meaning: it could still get nuked by an electrical surge from a nearby lightning strike, but at least the replacement cost would be low. Hmmm... I suppose better still would be switching to some kind of completely wireless internet access during such storms, by maybe converting my cell phone into a hotspot or using a Verizon jetpack.... In that case, having the router be low power would be very nice indeed, because then it could run on batteries during the storm and thereby have no lightning risk at all.

[NeverDie]

My plan is to put pfSense onto this Supermicro motherboard, which has 6 software programmable gigabit ethernet ports plus a seventh for IPMI access:
https://www.supermicro.com/products/motherboard/atom/x10/a1srm-ln7f-2758.cfm

I'm not sure what "software programmable" means in this context, but it connotes that maybe not everything needs to be routed through the CPU.

It consumes at most 20 watts when under load, but, IIRC, roughly 6 or 8 watts when idling. The Intel atom processor is built-in to the motherboard. The atom doesn't have much single thread oomph, but it does have 8 cores and all the features required to run a hypervisor, which might be worthwhile, especially if it turns out that the software programmable ethernet ports can be configured to manage all the routing on their own.

If I didn't already own this board, I'd probably pick up a similar Supermicro board that has six 10-gigabit ports on it. Right now there are a ton of used ones on ebay for around $80 per board, some including an E3-1200 processor in the $80 price. i.e. they cost even less than the pcengine board. Probably more than 20 watts TDP, but considering its a SuperMicro motherboard and each of the six ports supports RJ45 10gbe.... Heck, at that price for a 10Gbe node, I should buy 3 and build an HA ProxMox cluster.

[skywatch]

@NeverDie Quite a lot of this thread is very interesting even though it all goes way over my head. Do you think one of these would do?

https://www.ebay.co.uk/itm/HPE-Synergy-12000-10x-480-Gen10-Platinum-D3940-38TB-SSD-2x-40GB-16GB-FC-Frame/283693062050?hash=item420d6c97a2:g:XK0AAOSwbqdd4Rl0

:}

[NeverDie]

@skywatch said in Advisory: put IOT devices on a separate LAN/vLAN for better security:

@NeverDie Quite a lot of this thread is very interesting even though it all goes way over my head. Do you think one of these would do?

https://www.ebay.co.uk/itm/HPE-Synergy-12000-10x-480-Gen10-Platinum-D3940-38TB-SSD-2x-40GB-16GB-FC-Frame/283693062050?hash=item420d6c97a2:g:XK0AAOSwbqdd4Rl0

:}

No.

[monte]

@NeverDie I would like to know where did you find those for 80$ and even with cpu included?! I want one!
Anyway just to throw in this one if you haven't seen it yet, there are pfsense boxes from the netgate themselves https://shop.netgate.com/products/2100-base-pfsense. I know there is ongoing controversy with netgate as an entity, but you may consider this as an option even though I prefer to build things myself.

[NeverDie]

@monte https://www.ebay.com/itm/X10SLH-N6-ST031-Supermicro-E3-1200-v3-LGA1150-Motherboard-3x-X540-T2-6x-10GbE/184546263249?hash=item2af7d080d1:g:CCUAAOSwoP1gSWhO

[monte]

@NeverDie oh yeah, of course I was looking for the wrong one
I personally have Intel DQ77KB that I use as my home server, but planning to make it a pfsense or opnsense box. It's also LGA1150 and has two ethernets onboard, and I don't think I really need to use a firewall as a switch.

[NeverDie]

@monte said in Advisory: put IOT devices on a separate LAN/vLAN for better security:

Intel DQ77KB

It doesn't support ECC memory, but maybe it makes no difference, since each packet would contain its own error correction anyway.

What kind of switches are you using to create and manage your vlans? In an earlier post I mentioned I was planning to use relatively cheap 1gbe managed netgear switches, but if I could get 10gbe transfer rates using the ebay supermicro boards for just a little more money plus some memory and a powersupply, I'm inclined to do it. I could certainly live with 1gbe, but to speed along backups or vme migration or restoration from a backup server, 10gbe would be a nice luxury because it's a good match for the read/write rates of pci-e 4.0 nVME drives. Or, some of these older ebay Supermicro boards can be had with gobs of ram (128GB or even 192GB of RAM), where you could easily run entire virtual machines inside of just RAM. Then there's no nVME drive to wear out, and you could pretty much transfer files or VM's as fast as the ethernet will carry it. Or a file server with such outlandish amounts of RAM could have a positively enormous RAM cache to facilitate ultra fast file transfer rates. Then maybe you don't need much nVME on the local machine and just boot from the network instead. It might be a good way to amortize the cost of the RAM expenditure. I don't know for sure how well it would work in real life, as I haven't tried it, but that's the theory. Anyway, for doing those kinds of things, 1gbe just wouldn't be fast enough compared to local nvme, but 10gbe just might be, even after deducting for the ~20% ethernet overhead. This back of the envelope calculation assumes no meaningful network contention, but I'm comfortable with that assumption, because on a home network there wouldn't be.

[monte]

@NeverDie my setups are quiet simpler. On one location I don't use vlans at all, as I don't use third-party iot devices that I need to actively separate from my network. On the other location I use Mikrotik RB260GS as a managed switch plus I have 3 LAN ports on my pfsense server, where one of them is used for dedicated subnet for outdoor cameras and wifi's.
I would like to have hardware that could take advantage of 10gbe network, but for now I just keep things simple and slow

[NeverDie]

@NeverDie said in Advisory: put IOT devices on a separate LAN/vLAN for better security:

@monte https://www.ebay.com/itm/X10SLH-N6-ST031-Supermicro-E3-1200-v3-LGA1150-Motherboard-3x-X540-T2-6x-10GbE/184546263249?hash=item2af7d080d1:g:CCUAAOSwoP1gSWhO

Lest I mislead anyone, I subsequently contacted to the seller and, despite the wording, it doesn't include an E3-1200 with the board. He just meant that as shorthand to refer to the processor family that's compatible with the board. That said, there are a ton of inexpensive used LGA1150 CPU's on ebay that could serve the purpose.