as noted before you should always move IoT stuff on a own VLAN, for security purposes you should always segregate traffic as much as possible and create different security zones.
This will allow to control traffic flow with firewall rules and other systems such as IDS/IPS.
The Firewall rules controls what traffic you allow between the zones, and an IDS will control and verify that traffic. They can scan the content of a data packet and look at the content of the message or recognize applications (such as SSH over port 80 as an example)
back to your question, yes PfSense is a great "enterprise grade" firewall which gives you the toolset you need such as:
- Firewall rules between zones/subnets
- 2 different IDPS systems (Snort & Suricata)
- DNS filtering & interception
- RADIUS server for mac filtering, 802.1x EAP-TLS etc.
- IGMP proxy and mDNS services for stuff like Sonos speakers etc.
so the firewall gives you the ability to control and verify the traffic, however it offers no correlation, intelligence and management. for that you use something called a SIEM (Security Information Event Management) A SIEM will capture all the logs from your firewall, switches, endpoints, mirrored traffic etc. and do correlation and analysis.
let say you have an outgoing HTTPS connection which you allowed in your firewall, (your IDS wont be able to analyze the content unless you decrypt the traffic which may break stuff). the only thing your IDS see is $&$#$&%^%*%$%^ a.k.a garbage. with a SIEM you can get open source threat intelligence etc. which will generate an alert IF a connection is made to a compromised IP/domain. A great example on a open source SIEM is Alienvault OSSIM. its an all-in-one and easy to install. Im running it as VM's in my system, together with PfSense (also VM) and some other firewalls.
If you do choose to use VM's you should use ESXi as a hypervisor. this is free, very reliable and it is what most businesses are using in their server room or datacenters. on my PfSense VM I get about 800-900 Mb/s throughput with 1 IDS enabled, so if you set it up correctly you will get the performance you need.
