Les Moreland: When you have a problem, don't hide the information. Let people know you had an attack and what happened and what the vulnerabilities were. Because if everybody doesn't say what's going on, and how it's affecting me and what you need to do to correct it, then somebody else is going to get the same attack.
Teri Viswanath: That was Les Moreland, the chief executive officer for Wiregrass Electric Co-op, a member-owned cooperative. For today’s podcast, we are going to explore two very different types of security threats that, unfortunately, electric utilities are increasingly exposed to — that is cybersecurity and man-made physical system threats.
Hello, I’m Teri Viswanath, the energy economist at CoBank and your co-host of Power Plays. As always, I’m joined for this discussion, with my colleague and co-host, Tamra Reynolds, a managing director at CoBank. Hello, Tamra.
Tamra Reynolds: Hey, Teri. We ended 2022 with a series of physical attacks on the grid. Given the nature of our industry, there have always been bad actors but the nature of these attacks (both physical and cyber) are growing. And, there is a real need to make sure that the preparation and response is keeping ahead of these developments.
We had an opportunity to sit down with two co-op leaders, Wiregrass Electric CEO Les Moreland and the COO at North Carolina EMC, Amadou Fall.
Viswanath: This is our discussion with Les on the cybersecurity attack that occurred at his co-op back in 2021.
Viswanath: Let's just start at the basics. So Les, I'd like to hear an overview of Wiregrass Electric, your location, the size of the co-op, some demographics, and just to get an understanding of background.
Moreland: Wiregrass Electric, we started in 1939. We're in the southeast corner of Alabama. It's called the tri-state area. Our territory borders Georgia on the east and Florida on the south. We're kind of a ribbon across the southeast corner of the state, 90 plus miles east and west, and 25 miles north and south. That's about 2,100 square miles of footprint so we have a lot of windshield time. We have 26,500 or so meters, about 19,000 members, about 3,200 miles of line.
We're very lean. We have 62 employees and we have to be lean to control our costs because we have a really bad load factor, which makes our power cost high because we are over 80% residential load and around 20% commercial/industrial.
We're part of Power South, which is headquartered in Andalusia. There's 16 cooperatives in four municipalities. The south half of Alabama is a Power South area.
Reynolds: Les, let's talk a little bit about the co-op's 2021 ransomware attack. What happened exactly, and how did the co-op become aware of the situation?
Moreland: Well, you know when traumatic things happened, you remember where you were.
Well, July the 3rd, right at 12:10, a call center CRC in Dunlap, Tennessee, was unable to take a payment from one of my members who was using a credit card to pay their power bill. When things don't work, they have an on-call person for IT issues. They called my IT manager, Danny, and he tried to make a payment and found out that it didn't work for him either.
He escalated to network support and by 12:13, three minutes later, he had a technician from INTECH log into our system and see what was going on. Before he could get completed, two of our members had attempted to make payments at our payment kiosk in our Dothan office and in our Ashford office, and they were unable to make payment through the kiosk that are in our lobbies. Our standby supervisor was notified about that. We had a common thing going on.
These are all part of a separate network, which is segmented, where we take payments through credit cards. By 12:22, our remote technician had found some ransomware software on our payment server. Identified it as Decrypt8070 ransomware. He was smart enough to start shutting all our systems down. He immediately shut all our servers down to keep the ransomware from spreading. That was at 12:22. At 12:24, while I was walking down the sidewalk on my Saturday walk in front of Evergreen Presbyterian Church, my cell phone rang and my IT guy notified me that Wiregrass Electric was under a ransomware attack.
Now, fortunately, we have a cyber incident plan which has been done in advance and everybody's got a responsibility at Wiregrass Electric when we have a cyber attack.
The first decision that we were faced with was whether my IT person and my network support team, which is a contractor, small company in Dothan that has 14 employees, provide support to my network. We had to decide could we handle this or were we going to hire somebody to help us? We have a long-term relationship with a company called Secureworks, and they provide products that help us monitor the network and also layers of protection to keep things like malware and viruses from spreading. We called their after-hours support and reported that we were under attack. We found out that unless we wanted to do engagement with them and promised to spend about $25,000, that they could not help us through the attack.
The first decision I had to make was, yes, I want to buy 50 hours worth of support at $500 an hour in order to figure out what to do about this attack. Ultimately, that was paid by our insurance.
Over the next 48 to 72 hours, they ran scans on about 90 systems, servers and workstations. Actually, it was 90 workstations and 35 servers as I think about it, and did confirm after a lot of expense and a lot of scanning that the only thing that was hit was the one payment server.
We realized immediately that some of the things that we'd done over the last decade as a consequence of cybersecurity audits that had been done two or three different times over the last decade, where they said, "It would be better if your payments that are on the internet were in a separate network," and that's called segmentation. We had systematically hardened our system and created different layers of protection, and we learned after the trauma of it all that we had done a lot of things right.
Viswanath: That's interesting. All of a sudden you had to deploy a lot of people. You had mentioned upfront you've got 62 employees and you have one IT person and sort of outsourced. All of a sudden, you needed to beef up into this area pretty quickly and figure out what to do to
Moreland: That's right. I've had conversations with others that were under attack. Another one of my Power South members came under attack later. We could have restored from a backup and gone right back up, had our systems back online. We made the decision, which I believe was the right one, to not do that until all the systems have been scanned.
Now, I know of another case, and I've heard of others where people are anxious to get back up. They go back up without doing the right forensic analysis of what happened, and they get re-attacked immediately, or the virus or the malware is already spread and they didn't know it. It's worse after they come back up because then it spreads further. There's an inflection point where you have to make a hard decision. Do you err on the side of caution and really check everything and get to the bottom of it, or do you try to come up as quickly as possible?
Reynolds: If you could rewind the clock, what are some other things that maybe you learned that you've taken on since then that maybe have given you a position to really future-proof what comes next?
Moreland: The first question of what did we do right or what protected us, over a decade ago, we first got the idea we needed cyber insurance. When you buy cyber insurance, you can't get it without answering a very long questionnaire about what you're doing as far as protecting your network. Just in filling out that application, we realized that we needed to have a cyber security audit by a third party come in and really look at our system. We immediately hired a company to just come in and just tell us what we're doing right and what we're doing wrong. We did that and they gave us a list of things to fix. Since that first one, we've done it two other times. Over that 10-year period, about three different times, we've gotten somebody to look at everything we're doing and tell us what to do, and made corrections along the way.
In a cyber attack, the backups are the most important thing you have because once your system is infected with something, you're going to have to go back to a backup or you're going to have to pay the ransomware folks to release your data. We've spent a lot in the last decade or so making sure we have multiple layers of backups.
The other thing we had with Secureworks was 24/7 monitoring of the 34 servers and 95 workstations. We started that a long time ago. We were spending about $50,000 or $60,000 a year on monitoring services. As a result of this, we've doubled that.
Early detection with 24/7 monitoring, and people that are empowered to do something immediately and not let it go a day or two because then you end up having to lose several days worth of data if you have to restore. The other thing that we learned out of this is that some of the decisions we had made not to do software updates immediately as things came out create problems. We have tools that tell us when software updates need to be run, and we have decided to go ahead and do updates sooner than we used to.
The reason we don't like to do software updates immediately is everybody knows when you update your software, it breaks something. We used to let everybody else have a few days to do it, and then let them report the problems and fix them so that, when we did our updates, all the bugs have been worked out. Well, we've reset our thinking a little bit. You really can't afford to do that because a lot of software updates these days are to protect against vulnerabilities. We've become more aggressive in our screening of emails, using more additional products to filter emails. Our attack, from unusual standpoint, did not come through an email attachment.
Another lesson is network segmentation because what saved us was the fact that these payment services, the kiosk, and the online payments, were in a separate network. And it’s like having an air gap between that server and our other servers.
We have more and more network segmentation going on. Things like our SCADA system are on separate networks.
Viswanath: You mentioned that you had to double the size of the investment, which I would imagine now with your board members having gone through this, but walk me through how you got them comfortable with the investment that's needed especially in today's environment with the critical nature of your systems.
Moreland: Well, that's easier than you would think. When our board goes to statewide meetings and our G&T has trustee meetings, they've done an excellent job in Alabama of bringing cybersecurity speakers in, and they always tell horror stories.
We spend time in the boardroom every year. We have a cybersecurity policy, which is a board policy. It's about six pages long, and we review that with our board. There's a lot of education going on during that. In our local area, Houston County School systems were shut down for extended period of time, and they weren't prepared. Just recently, Tallahassee Memorial Hospital, which is about 90 miles from here, had a ransomware attack and they were down for 11 days. On paper. They could not use any of their systems at a big hospital for 11 days. We talk about these things so the board understands that we have to make these investments.
Viswanath: You can't prevent bad actors, but you can think about how you're going to respond.
Moreland: Exactly.
Reynolds: Understanding ‘how to respond’ also needs to evolve, staying ahead of the emerging threats to the industry.
The first documented ransomware attack actually occurred back in 1989, but cybercrime remained mostly uncommon until the mid-2000s, when bad actors began utilizing more sophisticated methods. Until that time, most co-ops were dealing with physical threats to their systems, amongst these were infrequent events when someone malicious damaged equipment. However, even here, we have noticed more sophistication and frequency in the attacks.
Viswanath: Our next guest, Amadou Fall, the COO at NCEMC, describes the physical attack on one of their member systems.
Reynolds: Vandalism and physical attacks on the electric grid have been on the rise for the last decade with well over a hundred reported incidents in 2022 alone. The one that's probably the most memorable is the one that happened in December in North Carolina. Can you discuss that incident with us?
Amadou Fall: Back in December 3rd, there was a shooting incident, an attack on Duke substation in Moore County. That substation also was a source for one of our member cooperatives, Randolph EMC. As a result of the attack, the Duke downstream, meaning going down that line that supplied one of the delivery points for Randolph, was offline. That impacted about 2,700 members of Randolph. In total, counting Duke's members that were out, there was almost 47,000 of them, and they were out.
They effectively shot at transformers. Transformers have oil that's running through the interior, and there are these veins when that oil heats up, that helps cool. The veins were the ones that were attacked, and that resulted in leakage for the oil. That oil actually, when it gets to a certain point, there are systems that can detect the level, and then it'll shut down the transformer to protect the equipment. Effectively, that's what happened.
Randolph was the ones and their staff, their frontline folks, who were looking at what could be done to try and restore service immediately. We were helping in terms of coordinating with the transmission service provider, which is Duke, but also, from the, let's say, the federal, as well as the state entities in terms of engaging.
For instance, there's the federal program called E-ISAC. It has Department of Homeland Security, it has folks from DOE, and then it also has federal folks that are on it. We were able to perform a situational awareness, see what's going on. Then, we also had the role of just being able to coordinate in terms of from the operation side, what we could share also with the state. The state has a version of that E-ISAC called Fusion Center. There are rules that if you get to a certain level, which we didn't, of outages, that you have to report to, for instance, the utility commission and even some of the federal reliability entities.
The real focus is back on the efforts that Randolph did. That happened on a Saturday, and they started doing some, what I call, moving some load. They were able to get a certain number of their 2,700 back online within 24 hours. They took some innovative steps. One was, they looked at a line that was there, but it wasn't active, and they looked at essentially upgrading that line. By early morning, Tuesday, that line was up and they were able to back-feed from the other lines, their other supply to feed that. Then, we have, and this is something that the North Carolina electric cooperatives do, and they have what I call mobile substations. They were able to take one of the mobile substations at one of the facilities and use that as a temporary, which on Monday, brought some of those folks back before those lines came up.
Reynolds: What other details around that are useful for co-ops when they're thinking about setting up a plan or strategy when something like this happens?
Fall: To the extent that that cooperative is working with their association or to the extent that the cooperative is doing it all in-house, they need to have essentially, an incident plan. It's not a departure from incident plans that they would have or from, let's just say, man-made causes or whether it's mother nature. Having that from a communication standpoint, and then leveraging all their different communication channels, social media, print media, pretty much on, I'd say, television, et cetera.
It's not necessarily that they can provide all the details because some of it might be for security purposes or other reasons where they can, but at least, the awareness, a situational awareness type that needs to happen, and then it needs to continue. It needs to be continuous throughout.
Teri Viswanath: In terms of planning and thinking about physical security, this seems to be a new era that we're entering into, so I want to understand how that changes the mindset around planning to beef up security with regard to these types of attacks.
Fall: There's always been that physical security, in terms of having fencing.
Even before this, you'll recall, there was a period where there was a fair amount of, I'd call it van-- Well, not vandalism, because it was really theft where people were entering and trying to steal copper when copper was getting to certain prices, et cetera. There was that sensitivity, and because of that, most distribution cooperatives would've had these physical security and would've had even additional steps, such as monitoring, as well as sensing. They would give them a sense of what's going on, and they would've also been coordinating and have coordinated with local authorities around that issue.
What's different here is just now, it's a different threat. We've always had situations where the hunter who goes out at night, or let's just say, an immature young person, and they're bored and they shoot at anything that moves.
This new phase also couples with some of the issues we're considering about folks getting access and plugging in, for instance, from a cybersecurity standpoint to some of the local access there. There is a national entity responsible for reliability on the North America Electric Reliability Corporation, which is NERC. They do have standards that they talk about, and I won't get into which standard. They're at the forefront of that in trying to come up with what should be something from a standards, which would be a mandate. Mandatory, as well as good practices. Beyond that, our members have been working on that aspect of security. I can't speak to specifics because sometimes, you can provide or reveal some of that information where then it could be leverage to circumvent those activities.
Viswanath: We're making sure we're hardening against not necessarily the nature, but the human-made problems that we're having with outages.
Fall: I mentioned one of the tools that was utilized was this mobile substation, and the NC Electric Cooperatives have actually formed what they call a mobile substation pool, where we have different mobile substations of different voltage levels that are actually diversely and geographically spread. That provides for just the ability to address all of the different member service area.
Back-feed is another mechanism. It is something that is spreading among the membership where they're having, we call them, ring buses. All that means is that you circle your territory, and then you have connections that can allow you, for instance, if somethings aren't happening on the north, you can switch and use something from the south to address that need. That is something that is part of our overall efforts in planning.
Viswanath: When the lights go out, most people assume it’s due to a storm. But that’s not necessarily the case.
With about 55,000 substations deployed nationally, most in remote, rural areas, they’re vulnerable to attack. As Amadou highlights, given the sprawling area that we are talking about, it is extremely challenging to monitor and protect this infrastructure.
Reynolds: There is also the challenge of the operating space that is under the jurisdiction of our co-ops to control. The average sniper rifle has a range of about 600 meters. A 50-caliber rifle has a range of 1,500 meters. So now you have to decide if we're going to put up a perimeter, how far away do we put it? Now you're putting a perimeter in a space that you don't even control or own.
Viswanath: My take-away from the discussion with both Amadou and Les is that understanding what we control and how we’re going to react to these bad actors is critical. Also, making sure that the co-op board and really the entire co-op team is on the same page on what those actions will be is important.
Reynolds: I do hope you’ve enjoyed this program and will tune in next month as we will be joined by Neil Chatterjee, the former FERC chair.
Viswanath: That’s going to be a great discussion. Bye for now.
