Our previous posts have focused on cybersecurity in building automation systems. We discussed the large attack surface of these systems which have thousands of smart devices connected on open protocol networks. Because smart buildings are run by complex computerized systems there is almost always a third-party company maintaining them. The pandemic, IoT technology and other factors have encouraged service providers to connect to buildings remotely using VPNs over the internet. Building automation systems are often installed on the same physical network infrastructure as business systems, and from this vantage point, they can provide a convenient and obscure position for bad actors and automated bots to launch cyberattacks on the enterprise at large. Unclear ownership of network resources in multi-tenant buildings furthers the risk of exposure.
We also discussed best practices for securing building automation systems including deployment of a cybersecurity management framework, engaging with vendors to ensure that systems are deployed properly, creating zones of control to isolate key assets, and maintaining a defense in depth through human diligence and use of cybersecurity automation software.
Ideally, we would report that many in the industry were following best practices to establish defense in depth, but in reality, organizations are moving too slowly while threats are increasing. Having led the development of a major building automation system for almost a decade, I understand firsthand the complexity faced by the manufacturers and customers alike. Cybersecurity attacks have become increasingly sophisticated and militarized as governments and state-sponsored hacker groups have entered the field with vast resources at their disposal. This paper by the US Department of Energy is a very good resource if you are still uncertain about the realities of the situation. From my own experience, these researchers make the case very clearly, and I agree fully with their views.
We can easily imagine a scenario where a technician from a local systems integrator is servicing his customers. Under pressure and behind schedule, he moves from building to building across a large city to keep systems and equipment operating. Though he is aware of cybersecurity and takes steps to prevent obvious risks, he sometimes takes shortcuts for the sake of efficiency. On this day his laptop is having problems, so he loads software patches developed in his office onto a thumb drive and brings them to the customer site.
The first customer he visits is a multi-tenant office tower. Maintenance of the facility has been outsourced to a third party while the tenants in the building rely on their own remote IT staff for technical support. They all share a common backbone network, and though there are separate VLANs, the passwords are sometimes shared across vendors and tenants. A few of the tenants have hijacked Wi-Fi from their neighbors and network hop to avoid their own company’s Internet restrictions. Our technician enters the equipment room and inserts his thumb drive into an HVAC automation server’s USB port. The automation server is running Linux, and he borrows a local terminal to instruct the system to load the software patch. He completes the upgrade and returns the thumb drive to his pocket.
Following a quick lunch in his service van, he drives to the second customer’s building; this one is a large hospital. The facility team at the hospital are long term employees who operate from a well-organized control center in the basement of the main building. Our technician checks in and heads up to the roof to troubleshoot a ventilation unit that has been acting up. His laptop is still not working, so he plugs his thumb drive into the controller to download the operating history of the unit. He leaves the roof with the data he needs and makes his way back downstairs to sign out of the facility.
By the time he arrives in the control center he is greeted by the manager who has a concerned look on his face. Apparently, a suspicious network event from the controller’s IP address was detected and an alert was texted to members of the hospital’s facility team. After a brief conversation, the technician admits that he had taken a procedural shortcut to use a thumb drive instead of his secure laptop when he connected to the controller.
The team immediately follows their established procedure to remove the controller from the network to prevent further spread. They also quarantine it for forensic examination. Within thirty minutes the ventilation system is back in service and the problem contained. The facility manager calls the cybercrime unit of the local police department because he knows members of the team personally, and they send experts over to collect the unit later that afternoon.
During the next week, the technician and his manager are reminded by hospital management about the restriction on thumb drives, and they are asked to sign a letter which puts them on notice that further disregard of cybersecurity policies could result in contract cancellation and legal action.
The hospital staff reviews the incident in their monthly cybersecurity team meeting and decides to take steps to physically prevent access to USB ports where possible. They also send letters to other vendors to explain what happened and to remind them of the thumb drive policy. They then review the findings from the forensic analysis and determine that the malware was designed as a man in the middle attack which could be used to exfiltrate patient data, including medical and credit card information. Because they were well prepared, it was detected quickly and isolated according to procedure before damage could occur.
Unfortunately, the facility team and tenants from the high-rise office building were not as well prepared. By the time the police conducted their investigation, the malware had spread throughout the building automation system and into several of the tenant’s business systems. Further tracing revealed that the source was actually one of the tenant’s business networks. The building was closed temporarily until the automation systems could be wiped clean and reprogrammed. Steps were also taken to establish zones of control and to further separate the tenant’s systems from the building automation systems. Passwords were also re-issued and more tightly restricted. Several tenants lost credit card information and confidential documents, this took several months and many thousands of dollars to repair.
Situations like this are not uncommon and are growing. Establishing a cybersecurity framework, zones of control and defense in depth with automated cybersecurity monitoring requires teamwork and focus, yet this is inexpensive compared to the alternative. As always, we welcome your thoughts and own experiences, and you can leave comments or questions for us at this link.
