In our previous post, we reviewed the role that vendors play in securing building automation systems. Because these systems are made up of hundreds, or even thousands, of intelligent devices distributed on open protocols throughout large buildings, and because they are often connected to the internet, it is essential to engage systems integrators and manufacturers to achieve installations which are cyber secure by design.
In this article we will review best practices for maintaining a cyber secure building infrastructure after it has been commissioned. Building automation systems include the full scope of operational systems in large buildings. This encompasses HVAC, lighting, and fire control, as well as physical security including badge access systems and video surveillance. Mechanical systems such as elevators, power systems and emergency generators are also included.
A standardized cybersecurity management framework and proper system design including zones of control are necessary for defense in depth of critical building infrastructure. Once the foundation is established, the ongoing security of these systems will require staying current with cyber threats, monitoring for cyber intrusion and taking continuous defensive measures (including software upgrades).
Cybersecurity automation is essential because the systems are large and complex, and because new threats appear continuously. A combination of real-time alerts and regular reporting is the most effective approach to maintaining a strong defense. State of the art cybersecurity automation software will detect malicious events in large systems and notify the appropriate technical staff in real time. By routing alerts to technical experts, organizations achieve the fastest possible response to attacks when they occur.
If a system has been designed properly, with zones of control and defense in depth, it may be possible to automatically detect an intrusion and lock down a specific device or network segment. This can ensure that malicious agents are isolated quickly and prevented from spreading. Automatic isolation is a tempting solution for buildings, and is increasingly used in business environments by IT departments. However, building automation systems perform life safety functions, and facilities experts must be involved to ensure that automatic isolation is feasible given safety mandates. In some critical facilities, such as hospitals, special care would absolutely be required when considering automatic isolation of control elements or networks. Regardless of isolation, real-time alert management is an essential feature of automation that will provide many benefits, including more efficient after-hours coverage and guaranteed escalation.
While real-time alerts are the first line of defense, equally important is using detailed cybersecurity health reports to drive continuous improvement. This best practice involves systematically and regularly reviewing existing and emerging cybersecurity threats vs. system configuration, network infrastructure, revision levels and event logs with the cybersecurity team. This practice can be simplified by installing a cybersecurity automation system with built-in reporting that complies with the format of the management framework that has been selected by the organization (for information about industrial cybersecurity management frameworks, please see this previous blog post). Regular reviews set the stage for prevention and are the best way to anticipate and defend against high likelihood threats. The threat landscape is ever-changing, as are the systems themselves, meaning that vigilance and continuous improvement are necessary to maintain defense in depth.
Out of date revision levels are a systemic problem for building automation systems. Equipment is often installed in difficult to reach locations, and because of the sheer number of devices and the slow throughput of open networks (such as BACnet over RS485), software updates can be tedious and labor intensive. Updates often cause control outages requiring manual operation to keep equipment (including air conditioning units) operating. This situation is not at all unusual and means that upgrades usually require extensive planning and coordination. Once the systems have been upgraded, a cybersecurity automation tool can monitor the landscape of threats and the revision levels making future patch planning faster and more efficient.
Regular back-ups of programmable devices and servers is essential to ensure rapid recovery from cyber incidents. Ransomware is an increasing threat, and fast recovery may depend on smooth installation of a trusted secure replacement. When a building automation server, fire monitoring system, elevator control or security system has been compromised, the building may need to be evacuated. If a ready back-up is not available, then it could take hours or days to fully restore operation. Back-ups should always be stored off site and should be scanned and secured to minimize the likelihood of cross-contamination.
People sometimes forget that cyberattacks are a crime. To preserve evidence it is important that compromised equipment be removed from the building network and secured for forensic analysis. Spare devices and computers should always be kept at the proper revision level to accept recent back-up software and to replace compromised equipment.
Securing large building systems requires automation and management discipline, and the best teams will plan for likely scenarios. They develop procedures, train and stage simulated attacks, establish escalation practices, and know exactly what to do and who to contact when problems occur. By being prepared they place themselves in a position to react quickly and decisively to contain any impact. Small problems can and will escalate when people are unprepared and working under stress. Because cybersecurity attacks on building systems are increasing, preparation today is essential.In our next post, we will compare two scenarios: one where the organization is prepared, and one where they are not. We will compare the cost of preparation vs. the cost from problems are left unchecked. We appreciate your input and ideas, so please let us know if you have comments or questions at this link.
