[audio mp3="https://www.industrialdefender.com//www/wp-content/uploads/2020/03/Phil-Dunbar-MST.mp3"][/audio]
In this edition of the Waterfall Industrial Security podcast, CTO Phil Dunbar talks about the birth and rebirth of Industrial Defender with Andrew Ginter, Waterfall's VP Industrial Security.
Phil and Andrew discuss the origins of Industrial Defender as a SCADA technology company, its rise as one of the original industrial cybersecurity solutions companies, its past few years as part of Lockheed Martin, Leidos and CapGemini, and also what Industrial Defender offers today, now that it is back as a standalone company.
For more information on this podcast, visit the Waterfall Industrial Security Podcast page here.
Nate Nelson
Hey listeners, welcome back to the industrial security podcast. My name is Nate Nelson. I'm sitting with Andrew Ginter, the VP of Industrial Security at Waterfall Security Solutions who will introduce the subject and the guest of today's show. Andrew, how's it going?
Andrew Ginter
I'm very well, thank you Nate. Our guest today is Phil Dunbar. He is the Chief Technology Officer at Industrial Defender. Our topic is "Industrial Defender Returns". What does that mean? Well Industrial Defender was one of the pioneers in industrial security. I mean I worked there before I joined Waterfall. I WAS the Chief Technology Officer at Industrial Defender for a number of years. I hired Phil and he was my VP Product Development at the time. Industrial Defender was a going concern, they were acquired, and over time I saw the brand name disappear. It's clear that the technology and the people were still there, but as of January, they're back. So this is the story Phil is here to tell us. Let's listen in.
AG
Currently, you are the Chief Technology Officer with Industrial Defender. Industrial Defender has been around for a very long time. Can you talk about sort of how Industrial Defender fits into this history of Industrial Security.
Phil Dunbar
Well Industrial Defender actually had its beginnings as a SCADA company. It all started with RTAP, an ICS SCADA product that was being developed at HP. Then during the big Hewlett Packard split, the SCADA business was spun off to Agilent, and then it was later sold to a company called Verano which was one of the many supply chain integration startups just before the dot com bust. Verano became the business that was bought from Agilent and subsequently the supply chain technology was mothballed. I believe that was around 1999. At that point, Verano started building the world's first ICS SIEM and called it Industrial Defender. The SIEM was developed I believe around 2002 and came to the market in 2004. After that, the company decided to rebrand the company name from Verano to Industrial Defender, and that was basically the birth of Industrial Defender.
Industrial Defender grew organically, introducing new host intrusion detection agents and some network intrusion detection systems. Through acquisition, they bought a 24/7 ICS SOC (Security Operations Center). They acquire an ICS pen testing company and a security assessment business. They were doing this to build managed services and professional services offerings on top of their product offerings. Then around 2010, Industrial Defender acquired TelTone, primarily for its secure dial-up gateways which are used extensively for power substations in North America.
Post 9/11, the company started to focus on utilities customers with the emerging NERC-CIP standards, which were driving cybersecurity compliance. As a result, we got quite a bit of interest from a number of suitors, large companies looking at investing in OT security. We were eventually sold off to Lockheed Martin in 2014 and continued supporting and enhancing the product line.
Lockheed Martin merged this entire IS and GS business, Integrated Services and Government Systems business, with Leidos, and at that point, Industrial Defender became part of Leidos Cyber. This all occurred between 2014 and 2018. In the process, much of our focus on ICS security products waned. We no longer had a dedicated sales staff and marketing support for ICS security products. The company started focusing more on IT security and we kind of got lost in the noise.
When CapGemini purchased Leidos Cyber last year, CapGemini was primarily interested in the professional services part of the business and not so much in the ICS cyber products. At that time, it seemed as though a spin out or a carve out was the only thing that would make sense.
It's important to understand that during this period of time, even though we lost focus on marketing and sales, our parent companies continued to invest generously in our product development. We did have a loyal install base and got additional sales but primarily through word of mouth.
AG
So let me add to what Phil said there. I was working at Industrial Defender, we parted ways in about 2010. In the early days, there were IT vendors who said "Industrial security? We do that too," and they basically tried to sell their IT stuff into the industrial market. Even then, there was a role for that, there was a lot of IT technology in the industrial market. But there's a lot of industrial stuff in the industrial market that the IT stuff really is blind to. So the significance of Industrial Defender back in the day was that we were one of the very few companies that were focused on industrial applications.
I remember working on the Industrial Defender SIEM, the security information and event manager. SIEMs were a very big deal back in the early 2000s. The big enterprise SEIMs had been built and sold for a couple of years. SIEMs were a very big deal. So Industrial Defender tried to produce an industrial SIEM. Back in the day as I recall, there was a bunch of sales and a bunch of install base, but it seemed like what industrial enterprises wanted to buy was enterprise SIEMS that understood industrial, also. But they wanted sort of one pane of glass. Phil has talked about the SIEM historically, but to my knowledge, the SIEM is not part of the Industrial Defender offering anymore. It morphed into other technology. But back in the day, I was very proud of it. It was an industrial SIEM. I got feedback on the SIEM from basically the Chief Technology Officer of one of the big enterprise SIEM vendors. He looked at our stuff, I looked at his stuff and we were talking partnership at the time, and his feedback was, that's a very simple product. And he meant it in a good way. This is something that people will actually understand, whereas the enterprise SIEM was powerful, big, complicated. It's what people wound up buying but one of the things that industrial customers demand is a degree of simplicity so you can actually understand what's going on.
Anyhow, so that's me rambling for a bit. This is why Industrial Defender was important in the day -- it's because they came out of an industrial background, as Phil explained. They brought a deep knowledge of industrial systems to the question of securing industrial systems.
AG
That's a long history, but there's been some big changes in the last 6 months. Can you talk about those changes?
PD
Yes, we put together a business model to carve out the original Industrial Defender part of the business. The ICS/OT security products part of the business and the associated services with that. We were able to find several private equity firms that were interested in investing. We picked one, TELEO Capital, primarily because of their expertise with doing this kind of a carve out from a larger company. We were able to recruit Jim Crowley, as CEO. The reason Jim is important here is he's one of the top people in our industry in marketing and sales. He's very successful and experienced and also he was with Industrial Defender back in 2014 at the time of the Lockheed Martin acquisition.
So now we've finished the carve out, we've reincorporated as of January 1st and we're a private, independent company. We're in the process of putting investment into marketing & sales focus, and to continue on our product development roadmap.
AG
So you're back. That's great in my books. The world needs more industrial security. You mentioned what had been carved out and made independent. Can you talk a little bit more about that? What is Industrial Defender today?
PD
Industrial Defender has a suite of products for ICS security and compliance. This includes endpoint management, network IDS (intrusion detection sensors & monitoring), compliance and audit reporting. Some of the areas in endpoint management we deal with are asset discovery and inventory, both hardware and software. We have host intrusion detection agents. We generate compliance events. We manage baselines, your current operation, and we manage policies if you want to implement a NIST policy, for example, or an NEI policy, or a NERC-CIP policy, we have templates for doing that. When we find deviations from those policies, we generate exceptions.
We have a really broad -- and this goes back to our experience with industrial automation and our long history -- we have a very broad native support for industrial devices. We support hundreds of different industrial devices from a multitude of vendors.
More recently, we've been investing in vulnerability management where we can collect the installed software on your equipment and this can include everything from the firmware revision of a switch or an industrial device to the accumulated installed software in a Windows or Linux machine. We have back-end server that's constantly scanning vulnerability databases and vendor advisories and we can match up your software or firmware revision level with potential CVEs, vulnerability advisory reports and tell you what in your system is potentially vulnerable and should be patched.
AG
Can you give me some examples here of how people use all of this?
PD
Well on the endpoint management side, I'll give an example where we collect a lot of information on what's going on on your host equipment. We had a European customer that had a SCADA system that was behaving strangely and the customer's SOC thought that their SCADA server had been compromised by an external actor. It turned out that they found our tools the most useful because what they were able to do was scan the access logs to their various servers and found out that on this particular SCADA server, someone had logged in, someone was authorized to log in but shouldn't have been on that authorization list. They had misconfigured the server. We never found out whether that was malicious or just maybe an oversight or incompetence, but the SOC was able to pin it down by searching through our access logs.
AG
So I left Industrial Defender just as the transition to the policy manager product was starting. A lot of the SIEM technology was relevant to the policy manager space and was being used in that new project. Everything in the policy manager space is new for me, and so that's what I asked about next.
AG
The other thing that caught my ear in your description of the endpoint protection you have here is that you were talking about policies and compliance. What is that? Can you give me an example of that as well?
PD
Let me give you examples of both. Many of our customers are utilities and they're subject to NERC-CIP compliance. What they like about Industrial Defender is once we're deployed, we're collecting data pertinent to NERC-CIP. Ports and services. Password complexity rules. All the different aspects you'd expect to see in a NERC-CIP compliance audit. So we collect the rules, but we also have an extensive set of NERC-CIP report templates, and we automatically populate the templates with real-time data and keep your reports up to date. This is something that once you automate and get it going, you have the continuous ability to pass compliance audits. In fact, we've never had a customer fail an audit since they've been using those tools. It goes beyond NERC-CIP, we have hundreds of different compliance templates. We have templates for NIST, NIS, and NEI for the nuclear industry as well. We also have the ability for you to create your own custom reporting so if you have your own internal policies, you can create reporting for that and generate and schedule those reports on the fly.
As far as policy is concerned, what you can also do is create a policy. Say you have a standard company policy or you derive a policy from NIST, once the system is running, it will alert you to deviations from the policy. Then you can go through a workflow to correct those deficiencies.
NN
Andrew I believe that's the first time on our show that someone has mentioned policy management. Can you give us a little background on that concept?
AG
I'd love to. Policy managers are comparatively new in my understanding in the industrial security space. What do they do? Imagine that you've written down, in 30 or 40 or 50 pages of manual, a security policy. All users with this class of account shall log in and change their passwords at least every 90 days. It'll be this complex of a password. Users with that kind of account, they only have to do it once a year. You've put a policy together saying what you need the organization to do in order to remain acceptably secure. The policy might not just include what users do, but, this class of equipment shall be patched right up to the second that class of equipment can supply security updates within 30 days of them being released. You've got all of these policies. How do you know if you know you're doing what you say you ought to be doing? This is what a policy manager does. It gathers a lot of information, compares it to a machine version of the policy, and comes back and says, "Yes. You're complying with these 73 directives. But these 4 directives have exceptions. 3 users didn't change their passwords. 8 machines don't have the right version of software installed or the latest security updates. And so on." Now you can tell if what you decided you need to do is actually being done.
NN
Phil also mentioned NERC-CIP. How would apply this all to NERC-CIP.
AG
NERC-CIP is a long set of very specific rules. You could imagine that the policy manager applies there. And it does. With NERC-CIP this is even more important because when you get audited for NERC, for CIP, the auditor shows up at your site, or an audit team more likely shows up at your site. They're not going to go into your server room and start tracing cables. That's not what they do. They're not going to log into your machines and say "Do you have the latest security updates?". They're going to look at your record keeping. So its not enough at a NERC-CIP compliance site to do what the policy says. You have to PROVE what the policy says. If you're doing it manually, how can you [keep up]? Let's say every time someone changes a password, you get a log entry somewhere for certain kinds of machines. But other kinds of machines, PLCs might not produce log entries. You might be doing the policy, but how do you know you've DONE the policy?
Well you might manually grab up the log entries and put them into a report once a month. You might ask people when they change their passwords on the PLCs to send you email. Then you'll have to go through the email every month, or every 3 months and ask the question, "has everyone who has to change the password, sent me an email saying they've changed their passwords?" If 4 people didn't, you've got to send an email back, you've got to produce a report that says "I scanned my email and these 4 people were not in compliance and so they fixed it 3 days later." It's a lot of labor. If you can press a button and get these reports automatically and say to the auditors, "Here you go. Here's the history of compliance. Anything you want to ask about, press the button and here's the report." A.) The auditors are delighted because it makes their jobs much easier. B.) The business is delighted because you didn't have to do all of that paperwork manually.
AG
So when you're talking about this endpoint management function, it sounds like you have what's called "active technology". It's stuff that's installed on the Windows or Linux host. It's stuff that connects to the PLCs and asks them questions to figure out what firmware version they have. Have I got that right? Do you get pushback from end users or from the vendors about this sort of extra stuff that's happening inside the control system.
PD
Yes, absolutely. Some plant operators are very wary about you contacting their industrial equipment, in particular PLCs. We will do that. We'll query a PLC. We can use CIP commands. A lot of PLCs that are in other controllers have embedded web pages that yield quite a bit of information so we can go into it on HTTP or HTTPS and scrape the results. I know this is viewed as potentially disruptive, but since I come from a PLC design background, I think the risk is minimal if it exists at all because these machines are designed to read inputs, process logic and then perform the outputs. This is done in a cyclical way. There's a time slice in there for communications. The reading inputs, processing logic and generating outputs has priority so its only going to do the additional communications time slice as part of a deterministic cycle. Having said that, since customers are concerned about that, we do both. We do passive monitoring and active monitoring. We do something in between where we could send out a kind of broadcast. Something like RSLINKS to Rockwell PCs that says basically "Identify yourself on the network." We may send out a ping broadcast like that maybe once a day and get the information.
The advantage of active monitoring is you can get a lot more information. It's limited what you can get over the wire, and also with passive monitoring, if you don't prompt the device, there's a lot of information that it has no information to put on the wire. You could go for months and months and months and not pick up what the firmware revision is on that device. I think the only answer is good active monitoring installed agents work very well on servers and ours are throttled to use up very little device system resources.
For remote access, we might SSH into a switch to collect information from that. So we have remote active monitoring, installed agent active monitoring, passive monitoring and passive monitoring with an occasional broadcast.
In practice, we prefer to do primarily active monitoring and we do get pushback from customers that don't want you contacting some of their industrial devices. This is what prompted us to get into network sniffing, attaching to a mirror port on a switch and just looking passively at the network traffic.
Then we've talked to customers who thought passive monitoring was the be all and end all but then realized that its very limited. So now they're warming up a little bit to [asking] "what's the least intrusive way we can get additional information, other than just from passive monitoring?" And I think that's what prompted some vendors to go to assisted passive monitoring, and then where possible to active monitoring and even installing agents.
NN
Andrew, maybe I missed it in his answer, but by passive monitoring, does Phil mean automated?
AG
No, passive is where you're watching every message on the network, but you're never putting a message on the network yourself. Generally you're looking at messages on the network either through a tap, which is looking at the messages on a single wire, or you're looking at a span port or a mirror port, where a switch is sending you some or all of everybody's exchanges on the network. Basically you can see but you're not actually doing anything on the network.
Active scanning is what he's talking about as being superior in terms of the data that it acquires. Active scanning is not just putting a message or two on the network, it is sending messages to machines saying, "can you tell me what version of firmware you have installed?" Which is a pretty benign thing to do to a Windows box nowadays. The CPUs have got so much spare capacity. They've got entire cores sitting idle to for questions like this to try and do something useful. But if you send that message to a PLC, a programmable logic controller, that every ten milliseconds has to hit its loop and read all of its inputs and calculate all of its outputs and push all of the outputs, or the machine malfunctions, now the question that Phil is addressing is, can you do that without interfering with the tight time constraints these PLCs operate under? His answer was yes, it was in terms of time slices. These PLCs do not use sort of the laissez-faire, the lackadaisical scheduling that conventional Windows and Linux and even Android phones use. They [PLCs] use hard, real-time scheduling. The functionality that needs 10 milliseconds -- more like they need 9 milliseconds every 10 milliseconds -- that function is scheduled every 10 milliseconds without fail. It runs for 9 milliseconds and you've got a 1 millisecond time slice left over. What are you going to do with it? You're going to fire up the communications software and say "Try and answer a message if you've been sent one." And a millisecond later, stop that software and say "Hold on, I need to go do something important." And do that 9 millisecond scan again. That's a crude example, but it means the PLCs are getting their job done, and any leftover process -- answering questions about firmware revisions -- are happening as a very low priority background task that is very much designed to not interfere with the primary function.
AG
You also talked about network monitoring. What have you got there?
PD
There's a couple of things we do with network monitoring. We have a vast library of signatures we're looking for that get updated on a regular basis. This is on our network intrusion detection system, which is an appliance that we sell as part of the product hierarchy. That's one of the things we do.
We do deep packet inspection, and this is centered around industrial protocols. We support deep packet inspection for the major protocols so that if we see something that's suspect, we can alert you on that. I can give you an example of that. I was at a power generation plant recently that had installed our
NIDS, our Network Intrusion Detection System. Right off the bat we were seeing the status of their PLCs on the wire. These are Rockwell PLCs thats [information is] published on the wire using the CIP protocol, and we noticed a couple of things. One is that they had a couple of PLCs, maybe 20% of their PLCs, had a physical switch in the REM position. This is the remote position. If you leave a Rockwell PLC in the remote switch position, that allows somebody to make changes to the PLC remotely. Normally, a systems integrator comes in and put the PLCs in remote mode, they hook up a laptop, they do firmware upgrades, they do logic changes, configuration upgrades. Once they're happy with that and they've proven that out, then they put it back in the RUN position and they go away. The plant operator didn't realize that the SEI was doing this and he says that gives us some really good ammunition and go back to the SI and say, "We're watching what you're doing. You better put those in RUN mode after you're done with maintenance."
The other thing we picked up is the firmware revision on their PLCs and we found that a number of PLCs have firmware revisions with vulnerabilities reported against them. So we could also make a recommendation that a plant operator have his SI upgrade the firmware in those devices during the next maintenance cycle.
In general what we look for on the wire and packet inspection is maintenance operations like starts, stops, logic changes, logic downloads, firmware downloads. These things can be potentially nefarious and used in malicious ways. At a minimum you could issue a stop command as a denial of service, but you could also spoof logic and do even more damage than that. In these plants, there's usually a short maintenance window, maybe one week out of the year where these operations are taking place. During normal operations you shouldn't be seeing PLC maintenance operations like starts and stops and logic changes. That's the kind of thing we want to collect and alert on.
NN
Admittedly I kind of lost my footing on that last answer there from Phil. He started using a lot acronyms. Can you help me explain what he was just talking about?
AG
The heart of the thing that Phil is talking about here is a Network Intrusion Detection System. NIDS he called it. He's talking about a signature-based function in the NIDS. A signature is like a rule. It's like an anti-virus signature. What's an anti-virus signature? It's a rule that says, "When you see one of these, one of those and one of those, that's bad." The NIDS appliance is looking at a mirror port. It's seeing all the packets and it's matching them against rules. And if you see one of these packets and one of those packets, that's an attack signature, and you raise an alert.
The point that Phil was making sort of towards the end is that when you're putting these signatures together, it's not just known attack patterns that you want in these signatures. What they've got in their signatures for these industrial devices is anything that's unusual. He talked about a remote programming switch. He talked about start messages, stop messages, logic download messages. None of those functions should be happening in a normally running system. You want alert on those things. Here's an example. Let's say you're in the security operations center and you're getting an alert that says somebody just sent a stop message and a restart message to a PLC. Then downloaded a bunch of new firmware. What's that SOC analyst going to do? Well they're going to bring up their system and say, do I have any work orders open for that part of the industrial process? The answer is either going to be yes. "Ok then I expect to see this message here. Dismiss it. On to my next task." Or the answer is no, in which case you pick up the phone and you call the site and you say, "It looks like someone is reprogramming your X, Y or Zed, but they don't have a work order open for it." Then they go and tap people on the shoulder and say, what's going on? Or someone says mea culpa and they open a work order and they fix it. Or, nobody's doing it. And then you scratch your head and go "That's not good." And you open a security incident response process because someone is in there reprogramming your PLCs and it's none of your people.
This is the point here, that intrusion detection technology looks at the packets and applies rules. Some of the rules are not just known attack patterns but anything that's unusual that really has no business happening on a running system.
NN
Andrew, that makes sense, but when we talk about intrusion detection like this, my thought is that, as important and significant as these alerts are as you just said, I have an antivirus program on my computer, and it sends me a lot of notifications. Whenever I'm on a new app, that my microphone is accessed, or my camera is accessed. It has this glitch where Microsoft Edge just knocks it all out of wack. Because I get so many notifications from antivirus program, my thought is generally that if I get a new one that actually matters, I might just click the little X button because that's what I've gotten in the habit of doing. So my question to you is, if we're doing alerts for starting and stopping processes and everything that Phil just mentioned, is there a risk that the system may be overloaded to the point where the human operators might overlook a real problem and not be as dutifully responsive as you just described there.
AG
This is a real problem. I think we talked about it in a previous episode. It's a questions of sensitivity. It's balancing false alarms versus missed attacks or, in the parlance of the industry, false positives versus false negatives. For example, you might have a PLC that is a little bit sick and is slowing down in terms of its responses and the operator is seeing timeouts. Or it might be that you have a device that you've lost a wide area network (WAN) connection to and again the communications are timing out. The operator is then trying to figure out how to get information from the device back. The operator might press a button that sends a stop and a start command to restart it and hopefully clear whatever problem was there. The operator might send 2 or 3 of these things. To your question, how do you know which alerts are real and which are not? In a sense, this is not the problem of the intrusion detection system, this is the problem of the SIEM. The SIEM is getting thousands, or even hundreds of thousands of log entries, you can call them alerts if you want, from all over the enterprise, and is integrating them and trying to make sense out of them, and trying to figure out what's real, what do I percolate to the top of the priority list so that the SOC analyst looks into it, and what is less important. This process of distinguishing and prioritizing alerts is one of the ways that SIEM vendors compete with each other. Each of them boasts of having the best of these ways. It is a real problem. It's a problem that a lot of money and effort is being invested into. I don't know that there's a known solution that everyone is using. All of these SIEM vendors are out there inventing their own algorithms and inventing their own machine learning and their own artificial intelligence and competing with each other in this arena.
AG
So you've talked a lot about different bits and pieces here. Do you pull this all together? Do you have sort of a high level interface to all of this.
PD
Yes. At the top level of the system is an appliance we call the Automation Systems Manager. This is where we consolidate all of the normalized data we collect from the network infrastructures. In general, Industrial Defender is a multi-tier hierarchy of devices collecting information from hosts directly or passively through an intrusion detection sensor. It all gets fed up into the top level Automation Systems Manager, and this hosts a database, file stores, has an interface, a pane of glass where you can start off with a dashboard and then drill down into reporting or event searching or vulnerability monitoring, or a variety of other different modules within that application.
AG
You've been talking about products. I know that in the old days at least, Industrial Defender was heavy on services as well. Do you still do services?
PD
Yes we do. We have two service departments. One is the deployment services. Generally speaking, customers have us deploy our systems on-site. This is a very experienced group. They've got a tremendous of field experience at customer sites. They're well-trusted. They also have international experience. We're doing quite a bit of work in the Middle East and Europe in addition to North America. For deployment, they'll go on site. It takes about a week to deploy our products and then they can come back periodically to do sustaining maintenance. They can come back on-site to do training and that sort of thing.
The other group is the customer support group. They're available to give support remotely, either through email or on the phone.
So those are the two services departments we have right now.
NN
Now I believe that you mentioned earlier that while you guys were doing these deployment services, you were also selling control systems technology. Were there any cases in which your teams were supplying security solutions to your own control systems machines?
AG
Definitely. A lot of our early security customers were customers who already trusted us on the controls system side. Why do you ask?
NN
That's interesting, because I'm wondering, could it be construed as negative if the same vendor is supplying control systems as well as providing the security of those control systems. The reason I ask is because I could see that being very convenient on some level. On the other hand, if you have multiple vendors who each specialize in their own particular lane, so if you have the folks who supply the control machines, and the folks who do security on those control machines, specialization is a very potent force. The security people may be more motivated to be a check on the control systems provided by the other vendor, where the vendor themselves, put in that position, may not be, they may know their systems better, but will they be as motivated to act as a check on their own machines?
AG
This is a question that I think -- there's no consensus in the industry. Let me back up away from security for just a moment and then I'll come back to it, but more generally, if you're building a brand-new natural gas fired power plant, gas turbine power plant, you might well go to GE or Siemens or whoever and say "Please build me this power plant. Here's the land. Go." They will clear the land. They will build the buildings. They will build the gas turbine. They will transport the turbine to site. They will install the turbine. They will instrument the whole thing. They will configure all of the instrumentation. They will put security in place. They will connect the security to their cloud security center. They will start monitoring the system. They'll do everything but operate it remotely. They might even do that nowadays. One vendor, top to bottom.
As a plant ages, though, you might start being more cost conscious because you don't have a capital budget anymore. You might see a new product come out that's brand new that Siemens doesn't have, or GE or your vendor doesn't have. You start doing add-ons. It gets even worse. If you're in a big company and you have like 70 power plants, you probably have 15 different vendors involved. A bunch of your plants, the older ones especially, are mixes of vendors. You might try and standardize, but you just can't. So there's a constant tension between all one vendor, all the eggs in one basket, thoroughly integrated because the vendor knows how to make all of their stuff work with their own stuff really, really well -- versus best-of-breed. Now you're struggling with a bit of integration, because maybe these best of breed things don't quite talk to each other. But best of breed means specialization, on the security side as well. It's not a question that has an answer. The bigger the organization, the less likely it is that all of your sites are going to be the same. It's almost certain, no matter how hard you try to standardize, that your sites are going to be all over the place. This is the nature of the beast on the industrial side.
AG
So we've been talking about industrial security for a while here. This makes sense, but your history was producing and selling some very powerful control system product. You've got a widely used dial-up product historically. Is that still part of Industrial Defender going forward?
PD
Yes absolutely. The RTAP SCADA product has a very devoted customer base. It's a very stable, high performance product and it has the advantage that it can be deployed on Windows systems, Linux systems, and some of the legacy Unix systems are still supported, like HPUX, AIX and Solaris. We had our origins in industrial control and so we still have a very knowledgeable staff that comes from industrial control orientations. That gives us a depth of knowledge that I think some of our competitors don't necessarily have.
The dial-up gateways are so widely used, most substations are dial-up, they're not internet connected. The secure dial-up gateway is substation hardened, and it dovetailed very nicely when we started getting into NERC-CIP compliance and focusing on North American utilities. In addition to our Industrial Defender products, we have the TelTone gateway.
AG
So Nate, on a personal note, it's nice to hear that some of the code that I wrote is still running out there. There was a time when I managed the RTAP development team. There was a time when I wrote code for the RTAP product. Pretty much everything else that I wrote earlier in my career has been retired. Years of my life have been invested in code that no longer exists. It's nice to know that something I wrote is still out there and still running in important applications. It makes me feel a little bit better about the years I spent writing code in that space.
NN
Alright Andrew, I don't want to rain on your parade here, that was a very cute sentiment, but could it be construed as a negative thing that the same code you had written over a decade ago is still in use? I don't really use Windows XP anymore.
AG
That's a good question. I had never asked that question. Let me think for a second. There's two sides to the coin. On the one hand you're right. The world evolves and the latest and greatest control system products look very little like the stuff that I worked on. The latest stuff is all Java and its all dynamically loadable code, and it's objects and modules, so I think that's your point. The other side of the coin is that if you have a very important physical process, a very important system of any sort that you really need to keep running and that you really need to keep predictable, change is the enemy of reliability. If the lights go out, or the oil stops flowing the pipeline for any length of time, that's a very bad thing. That's a fancy way of saying, "I don't know."
NN
Fair enough.
AG
So we like to leave our guests with the last word. Have you got a thought you'd like to leave with our listeners?
PD
Yes. It's something I've been thinking about recently, over the last couple of months. We, and by we I mean the ICS security community in general, not just where I work, we keep adding to our spectrum of collected information. We collect more and more information on the endpoint, the network infrastructure, what's going on in the network itself, configuration data, event data, vulnerability data, threat data, anomalies, baseline deviations, I could go on and on. We're going to continue to add to do this collection capability. We're looking quite a bit at link layer protocols now, starting with profibus. Discovery and configuration protocol, which is a link layer protocol. There's quite a bit of damage you can do if you're putting packets out at that layer in the sense that you can reconfigure a PLC and cause a DOS. More recently it was noted that CISCO's discovery protocol, which is a link layer protocol, has some vulnerabilities that can cause you to penetrate between different network segments, kind of reducing the effectiveness of segmented networks in OT. We'd like to use at user behavior indicators. Anomalies on user behavior. This may not just include OT data but maybe physical access to various areas and that sort of thing. Collecting information on users and how things are deviating.
Kind of a side interest of mine are side channels. I've been talking to people and reading up on RF, radio frequency side channels, EMF side channels, even acoustic side channels. There was some work done at IBM where they discussed picking up acoustic signals. Apparently on printed circuit boards, the ceramic capacitors vibrate and send out an acoustic signal. You can use these to deduce what's going inside the CPU and even use that to leak information.
So now I think we need to put more focus on how the end user is going to assimilate and digest all that stuff. What can we do as a vendor to distill, digest and simplify this well of data to the point that it's understandable and actionable without a team of 24/7 experts. What analytical tools, what visual tools, what can we do about risk scoring -- that's a big effort that we're putting time into right now. Anything we can do to provide clarity at a glance. There is after all a severe workforce shortage professionals -- especially in the domain of ICS cybersecurity. Basically the question is how can we do more and provide more with less.
AG
So I have to agree with the last comment. Every industry is automating. The industrial security industry is no exception. The more data we produce, the more we need automation, the security that we can automate, the more we can take our short supply of experts and focus them on higher value tasks like dealing with the more sophisticated attacks.
NN
Is it more easy to beat a bot than a human?
AG
It can be easier to beat a bot than a human, unless the human has to look at 100,000 alerts, in which case, you kind of need a bot.
NN
On that note, let's wrap things up here. Thanks to Phil Dunbar for speaking with you Andrew and as always thank you Andrew for speaking with me.
AG
My pleasure, always a pleasure Nate.
