NERC CIP-007 R2, or it’s common name “The One about Patching”, has been a pain to me and many others. For those uninitiated in the ways of CIP, it may seem like this is much ado about nothing. I mean its patches. It’s something we’ve been working on for 20 years, so what really is so hard about it? I’d even argue you’d be right to question this. Why does it take so many FTEs to manage something the enterprise does with half the people? However, to make such rash assumptions would be a disservice to those individuals who relentlessly pursue this requirement day in and day out. Because it’s not as easy as you would think on the surface. Let’s take a look at a hypothetical DCS system, and its NERC CIP compliance responsible, Dario the Diligent.
It’s Monday, September 1, and Dario wakes up bright-eyed and ready to tackle yet another patch cycle under the NERC CIP regulations for Medium assets. He’s got a 5-day grace period to check his patch sources for new patches. Sounds easy right?
Like any great admin, he pulls out his handy dandy spreadsheet of assets and software. But wait, we just came out of outage yesterday and several vendors were here making changes. So, Dario starts making calls to all the lead engineers around the plant to see what has changed. Most are pretty good and promise to send him updated inventories by Wednesday. Sounds simple enough, but Calvin over in cooling is out on vacation, so Dario checks his alternate, only to realize that person left 3 months ago. Well, time to go hunt someone down. By 4 PM Dario finally finds Calvin’s alternate, Jake, who promises to get the spreadsheet done by Wednesday. Dario, not having much success, heads home to rest up, refuel and tackle this again tomorrow.
Tuesday, September 2nd isn’t much better. To be safe, he starts to check the main sites he knows he needs to. Good thing too, because five of the 80 vendor websites moved their patch pages on him, and even more aggravating, two others changed their patch downloads from .csv to another format, so he spends all day fixing his macros. He was so busy that he forgot to follow up and make sure he’s getting the official inventory tomorrow.
Dario wakes up a little slower this morning, but he’s hoping to have all the updated inventories today, so he can make whatever adjustments to get through his checks by that NERC-mandated 35-day window. Time is running out. He gets most back. Bob in environmental says he forgot, but he will get the scrubber system updates to Dario by Thursday. Jake from cooling is MIA. Dario does what he can, including putting the final touches on last month’s mitigation plans for patches that didn’t make the outage and updating old patch mitigation plans for what he does know was addressed during the outage, but he can’t close out some of them until he hears from Jake in cooling.
Now it’s Friday, and Dario is out of time and patience. He looks like a new dad on day 3 home from the hospital who proudly just took the night shift for feedings. Haggard and hazy and maybe a little irritable, but no worries, because it’s TGIF, and he’s got big plans after work. He decides to run down to cooling and hunt Jake down. Turns out Jake is in the office and struggling to fill out the spreadsheet with the changes. He didn’t want to bug Dario, but has been working since Monday night to try and fill this out. Dario feels bad, like he should have checked in sooner, and so now plops down in a 1970s vinyl office chair that’s been here since the original commissioning and helps Jake out. They wrap up around 4 PM, and Dario has until midnight to get his review done. He rushes back to his office and fortunately only a few titles have changed, but the vendor did decide to update the .pdf reader for documentation from Adobe to something he’s never heard of, so it takes a while to acclimate himself to this vendor and where to find the patches and release notes, etc. He wraps around 11 PM and decides to head home.
I am now literally out of space given to me by the editor and only through the first requirement for Dario. As you can see, there’s a lot that goes into just one of these requirements (NERC CIP-007 R 2.2 if you are curious). We haven’t even begun to explore the whole mitigation plan world, but that is the one that really drives your NERC CIP people nuts, because of how the requirement is audited. Essentially, if you don’t apply a patch within 35 days of identifying it, you have to write an INDIVIDUAL mitigation plan detailed to the actual vulnerability. This is a lot of work, and copy-paste will only get you so far.
So, you do all of this, and you are rewarded with doing it all over again.
There is some relief, though. An industrial-focused asset management tool can collect information, including firmware from switches, routers, PLCs, IEDs, etc., Windows and Linux data like OS versions and patches, and also all the software titles (what is “software” is another thousand-word essay all by itself). Having a tool like this to ease the routine of repetitive data collection saves a ton of time, increases data quality and reduces the chance of an administrative error leading to a CIP fine. If Dario was using an OT asset management tool, he would not have had to chase Jake for a week.
Another area you can streamline or outsource is the source of security patches. This is again mind-numbing work that doesn’t add a ton of value, and it is really easy to make a mistake that places you squarely on the wrong end of an audit. Minimizing sources (NERC conveniently allows you to name a source) is one of the key ways to help manage the CIP-007 R2 requirements. Combining these two suggestions goes a long way towards streamlining your patch management process, and allows your people to focus on their real value, which is addressing the unique risks to your organization.
To learn more about Industrial Defender and how we are helping NERC-regulated utilities simplify NERC CIP-007 R2, check out our webinar “Squashing Spreadsheets: How to Orchestrate OEM Patch & Vulnerability Management”.
