Webinar Recording: Building & Implementing Effective Incident Response & Recovery Plans

Posted by Living Security Team
March 29, 2022

Share Article

On March 24th, 2022, our CSO Drew Rose was joined by featured guests Vinitha Varadarajan, Senior Director, Information Security at Rubrik, and Wolfgang Goerlich, Advisory CISO of Duo Security to dive into incident recovery & response plans.

Throughout the conversation, the panel covered a wide variety of topics associated with securing your "Human Firewall", including:
  • How is an Incident Response plan different from a Business Continuity and Disaster Recovery plan?
  • How do you build plans tailored to your organization’s needs?
  • How do you implement your well-crafted plan in real-life scenarios?

 

Here is the full transcript of the webinar:

Drew Rose: Welcome. Welcome all of our early, early risers, just on time attendees. It's so great to see you all here. My name's Drew Rose. I will be the host of today's webinar. Going to wait for everybody to jump in, get situated. Feel free to jump into the chat. We got some fun questions we're going to be asking. Initially, while we're waiting to get started, I’d love to learn where everybody's coming in from today. So, jump in the chat. We'll come back when we're ready to get started.

Oh, as you can see, I think we got our first poll running, as well. Living Security's community asking if you're a member of it yet. So, it looks like we have some strong members, and we have some participants that would like to become members. That's awesome. Community's a great place to have conversations around all things human risk management.

Ah, Wendy. Good to see you. Was hoping to see some familiar names in the attendee list. Yes. Wendy is a strong ambassador of the awareness community, Living Security community. If you do get to join, I'm sure you'll see lots of her engagement there. And I thank you for that.

Awesome. We have a strong attendance. I don't want to delay this too much. We're definitely going to get here. My name is Drew Rose. I'm one of the founders of Living Security. At Living Security, we do a lot of things. We typically play in the human risk management arena. We have a couple different products out there that we'd love to chat with you about post this conversation, but we are going to keep on topic, where the topic today is building and implementing an effective response and recovery plan. Coming out of the COVID arena or where we still lie in the space, a lot has changed in the last couple years. And so, I'm definitely excited for this conversation. Just a couple of housekeeping items before we get started. To keep things interactive, we will have more polls throughout the presentation. You're more than welcome to complete them as they pop up.

We'll also be aiming to save time at the end for some questions and answers. So, feel free to send those questions in. We'll arrange them and try to go through them towards the end of the end of the webinar. I'll be keeping track of the chat, as well, during the conversation, so feel free to stay engaged there.

Two more quick items. We will be rattling off two swag packs. Those who do interact with us, answering a poll, asking a question, hanging out in the chat, and all the breaking security webinar recordings can be found in the resources section of our website. So, if you find this super helpful, and there's some clear takeaways that you want to share with your community, we will inform you when that is posted online, as well as a transcript, and you can share that out to your community.

So I'd like to introduce you to our amazing panel today. Vinitha Varadarajan is the co-lead of information security at Rubric. She is a cybersecurity and risk expert with over 16 years of experience. The bulk of her career was in management consulting through Deloitte's cyber risk services, where she advised blue chip technology, healthcare, and financial service clients. She led multimillion dollar onshore and offshore engagements. She's also worked with Veritas and PWC before finding her way to Rubric in 2019, where if you have been tracking, a technology company is scaling like crazy right now. And she's been charged with building a security risk management customer trust and compliance program, as well as a security culture that is inclusive and relevant.

On the other side, we have Wolfgang Goerlich. He is an advisor CISO at Duo Security at Cisco. He is a cyber security strategist and an active part of the Michigan security community. He co-founded the OWASP Detroit chapter and organizes the annual Converge and BSides Detroit conferences, Wolfgang has held roles such as VP of consulting security officer and VP of technology services. He regularly advises clients on topics ranging from risk management, incident response, business continuity, secure development life cycle, and more. So, big welcome to my two panelists, Vinitha and Wolfgang. Thank you so much for joining us today.

I am quite excited about this topic. Prior to starting Living Security, I was charged with running information security for several different organizations, but it's been about five years. And obviously, in the digital age, so much has changed in the last five years, that when I was running the program, the most pressing manner was worrying about cold storage of hard drives and finding little briefcases that, if they fell, wouldn't damage a hard drive, and that was the most fun I was having in the business continuity era. I thought if that practice was still alive today, which I would assume not in most cloud first environments, but could you imagine how big of a vault AWS might need if they were practicing cold storage of hard drives for tape backups? Thoughts, Wolfgang? I mean, what do you think? Are we talking like Fort Knox size or maybe smaller?

Wolfgang Goerlich: Easily. Easily. And just the carts and everything, but AWS has all the robots. So, imagine like a fleet of Roombas all driving the hard drives ahead of them.

Drew Rose: That's right. It'd be a hundred percent automated. Vinitha, any statements on the yesteryear of business continuity there?

Vinitha Varadarajan: Yeah. It's crazy, right? It's exactly the way you described it. And Wolf, I liked your description of a fleet of Roombas. So, yeah. One can't even imagine. I think we're so far away from that, and it's just unfathomable how far we've come in this space, in just the information storage space, let alone how connected we are now.

Drew Rose: Yeah. Just like I said, the last couple years, we've seen so much. So, let's dive just into some of the questions. I think the first thing that we want to kind of lay down on the line is really to understand how incident response plan and execution differs from business continuity and disaster recovery. So, Vinitha, do you want to talk to us about the very broad basics of an incident response plan?

Vinitha: Sure. At a very high level, when you think of an incident response plan, I would think of it as a component of an umbrella business continuity plan. And for the purpose of this discussion, I'll focus an incident response plan on a security incident response. Because you can have different types of incidents. So, I think a lot of it is in the term response. So, you have a lot of the cybersecurity forensics technical components in an incident response plan that talks about not just the methods, but then the different types of incidents that can cause a technical team to then go figure out, "Okay, what was the security incident?" Do some analysis on where it originated from, what kind of techniques they used, and most importantly, what within your organization they were looking to target.

So that, to me, is the technical component of a security incident response plan. And as you can imagine, it has a lot of components around like people, process, and technology, but as important in my mind, in an incident response plan is the coordination, the PR, the non-security related aspects, because depending on what the target was, a company can have compliance obligations, privacy obligations, regulatory things they need to deal with. So, as we've all seen in recent incidents, there's a transparency PR communication component that's as important. All of which, in, let's call it the modern incident response plan, has a lot of importance.

Drew Rose: Yeah. That's really helpful. I think making sure we, as a community, focus on the people, technology, and process as a equal amount of focus where we can a hundred percent be aligned and ready to respond from a technology perspective, all the backups, whether they're cold storage or hot storage, or we're trusting AWS is in place and testing and ready to go. If we haven't addressed the people's side, we may never even get to the point of responding to the incident, whatever it may be. Wolfgang, what are your thoughts on the business continuity disaster recovery?

Wolfgang: Sure, but I want to pull on something you just said, "the people process and technology." We oftentimes debate that. Do we need better people? Do we need a better process? One of the things I thought was really interesting about the security outcome study that Cisco was they actually pulled on that and found that it was the people that were the main driver. And so, oftentimes we think, "Well, we need automation," but why do you need automation? Is it because you don't have people? The best use of automation, the best use of process, the best use of technology, is to make great people better. So, when we think about planning these things out, we think about what makes a good BCDR plan, I think it has to start with that in mind.

How do we make great people better? How do we get them the great information? How do we equip them to know what to do? Obviously a business continuity plan is a little more high level. Strategically, how do we keep an organization running in the face of adversaries, hurricanes, fire, flood, feast, and famine. And how do we make sure that the operation side is still going? The disaster recovery side tends to be a little more tactical, tends to be a little focused in on the technology itself. Given this particular scenario... You mentioned AWS. If AWS goes away tomorrow for four hours, what do you do? And who does what? And who owns that decision? And how do you make sure that it's communicated outright?

So, a good BCDR plan should top down, cover the business all the way down to the technology that enables the business, and a good DR Plan, bottom up, should look at those technology dependencies and make sure we know how to respond, and combined, that with the IR plan that we were just talking about, should make good people great.

Drew Rose: Yeah, I appreciate that. Make great people better. That really stuck out to me. I have a script. I love to go off script. Sometimes, my team doesn't appreciate that. Before, the majority of everyone was relying on web services, AWS, Google Cloud, Rackspace, you name it. When there would be outages or incidents that caused an outage, it was self-imposed. It's because something they did wrong, right, or miss was part of that outage. And now, my company being a SaaS company, we've been impacted by many outages over the years. And you know, it's definitely a different risk decision that we're making to trust wholly on a third-party vendor, that they are going to protect us from business continuity. It seems like a step backwards when trying to achieve internally five nines or six nines of uptime. Now we're going out and we're trusting that a third party's going to do that. What are your thoughts on the maturity of the BCDR planning when we have so much reliance on third-party vendors?

Wolfgang: So, I actually did a long project prior to my current role as doing consulting, and we evaluated hundreds of vendors, because that exact question was asked. "If our recovery time objective is here, where are our vendors?" Hopefully, it's a little bit above, so we're okay. If it's below, then we're in trouble. And it was also, in doing that, very disturbing to me because so many vendors were like, "Oh yeah, we're fine. We're covered by our cloud hosting provider." I'm like, "No, you're not. Yes, they will get your stuff back up, but what happens when everything is rebooted? Who is going through and making sure the services are still talking and everything's put back together?" So, I would like to think we're getting better, but I believe that with the shift to Cloud, with the shift to remote work, and everything else has happened over the past couple years, I think a lot of us have work to do to bring that BCDR plan back into date and back reflective of the technology capabilities that we're using and that we have available to us.

Drew Rose: Yeah. On that same subject, Vinitha, when it comes to risk acceptance, there's a major breach in the news right now that we are learning more about. And I think we'll have a couple opportunities to talk about it, but when it comes to authentication, and Wolfgang, you're uniquely familiar, for organizations my size and a lot bigger, it's almost impossible to try to replicate the authentication services that yours provides. And so, the risk acceptance that we're approving is like, I trust that Okta or Duo or whoever is going to be up. And if it's not, it's going to cost too many dollars for me to have a backup anyway, so I'm just not going to do it. Vinitha, what are your thoughts on that?

Vinitha: Yeah. We live in a really complex environment, right? And to your point about risk acceptance, I think one of the most important things, which some of us as security professionals do everyday, requires us to live in the trenches and really live in the detail. And I think one of the most important things around... Because we live in a world, to your point, where we don't control everything. We don't always control our own destiny. Right? So, there's a certain acknowledgement that there is a shared responsibility of risk. But I think part of that is really important to identify. There's a term called crown jewels that's used a lot, but really identify what's most important to you. And I think Wolf, he didn't particularly say this, but he alluded to it, because ultimately it's the data. That's what's most important to you.

Knowing where that data lives, what's most important to you, and then communicating to your executive team, to your board, on, "Here's the risk we're accepting. Here are some business decisions we've made against what's most critical to generating revenue for us. And here's why we've made those decisions. And here's the residue of risk." To your point. We've made this decision to trust in this vendor to be able to run this critical part of our business. And if we are down, that's an issue. And I want to add one piece there, Drew, because I think we're all sitting here and sort of drawing up the problem statement.

To me, a lot of it, in a nutshell, is I think it's the data that's the most important. And given that we are talking about BCDR and backups and having the right plans, I think it's also important to note that in recent times, one of the shifts we've seen, which I don't think was the case when I got into this industry close to 20 years ago, which is, hackers are much more sophisticated as we know, and they go after backups. So, it's not just important to back up your data. It's also important to make sure that that backup is secured, that that backup is, the fancy term, immutable, which is it can't be changed. It's [inaudible]. So, logically, it's different. It's not on your online network. So, in the past, we used to say, "Our company data is backed up. We're good." That's not the case anymore.

So, I just think there are multiple layers, but when we know whether it's authentication, whether it's pyramid security, whatever it is, let's just assume all else fails. Protect your data layer and protect your backups. That's where my head goes whenever we have these kinds of conversations.

Drew Rose: Yeah. I think that's the beginning and the end. You just stated the goal of what we're trying to achieve, but the beginning is to understand. Step one isn't to say, "I'm going to come out and create this amazing DR plan, and I'm going to have every process ready to go." It's to understand what we are protecting, and then craft your plan around the most critical resources. You may not need to protect the printers from working if the printer service goes offline for 12 hours or 12 days. That might not be important to your environment any longer since we've learned to adapt without printers over the last couple of years. Wolfgang, what's your thoughts on both, I think, having a backup authentication service? Where do you see clients investing in such a thing if something like an authentication service goes down. And then also, what are we protecting and why? I know we're kind of two threads here going at the same time.

Wolfgang: Yeah. So I'll answer the last one first and the first one last, if that's alright with you.

Drew Rose: That's good.

Wolfgang: One of the things that's really intriguing is we talk a lot about crown jewels, but as security professionals, we're not always good at crown jewels. I was a security officer at a money management firm. And I was like, "I've got all our crown jewels and all our key applications done." And we would go into DR, and one of my traders would be like, "Hey, what about that new app I installed the other day that I didn't tell you about that's on my desktop? How do I run it?" So, the problem with business continuity, disaster recovery, and instant response is trying to figure out scope.

And back to that security outcome study that I mentioned with [inaudible], when they went and asked, "Hey, how much of your environment are you protecting from a continuity and recovery perspective, and also how effective are you at recovering?" It wasn't 20%. It wasn't 40%. It wasn't 60%. As a matter of fact, if you look at effectiveness, all across that line was almost flat until you got to 80% of the environment that you're recovering. So, one of the things that I used to say all the time was, "Make sure you're spending a lot of time trying to understand your crown jewels. Make sure you're spending a lot of time trying to document your apps" I've almost shifted that on top of its head and said, "Make sure you're just doing basic coverage for everything first, and then, deep dive in your crown jewels, because that difference seems to drive much more of recovery outcomes than doing it the other way around. And it's also quicker. I'm going to try and do base level coverage.

Now, within that base level coverage, oh my goodness, what happens if your authenticator provider goes down? What happens if you're doing social logins, and now you find out this social login that you provide has been compromised. What happens in any number of those things? And the reality is that oftentimes building a separate redundant one isn't cost effective and also could introduce other vulnerabilities, which we could discuss in a bit. So, oftentimes, what that looks like is good coms, good procedures internally to disable and degrade key functionality, so the site doesn't just break, so our SAS apps don't just break. They still have some functionality at [inaudible] state and really good resumption. So, when something bad happens, and we know it will, for all the companies we're building on top of them, they're all going to fall at some point in time.

I love the phrase, by the way, hug ops. If you guys haven't heard this, whenever everyone goes down, all the ops teams start tweeting out like, "I'm sorry. Let me give you a hug. This could be us." When that happens, as a company, I think the key is to get ahead of it, communicate it, let everyone know that they get a snow day, and the minute those key components come back online, be ready to resume as fast as possible. I think the front end and the back end is really where we can optimize around an outage of any one of our key components.

Drew Rose: Yeah. And it's interesting. You bring up from an awareness communication perspective, there's an internal team-based awareness when you are relying on another service that may go down and be transparent to them. Then, obviously, there's the external transparency when an incident, an outage or something, occurs. And I know that there is no right answer in this situation, but in what we're going through today with the Okta breach, what are your thoughts on the transparency that has or has not been provided?

Vinitha: Do you want me to go first, Drew? I can provide my point of view, and I'm sure. Look, I'll say a lot of us in the security community are very opinionated people for the right reasons. So look, the first thing I'm going to say is this is a hard situation to be in. Anytime you get breached, you don't have all the details. You may have top-class, world-class security capabilities, whatever you want to call it. I just want to acknowledge that being in a position and a place where you have to respond to a breach is hard. That said, I think we're in a day and age where we've seen enough breaches. We've seen the impact that it can cause, the negative impact, so we can all acknowledge that transparency is number one.

And by not receiving it, there may be very good intent for not putting out that level of transparent messaging. It may be positive intent, but for users of that service, even if you're not a user of that service, it really tarnishes your brand and reputation. Even though, there may not be anything you're hiding. So, I think it's a moment. It's a learning moment for a lot of us, as companies to say, "Look, that is number one." And now, I will say the breach that we are seeing in the news now, the first response, in my opinion, was not great. It was a very canned response. We can all sit here and speculate. It could very well be that their hands were tied for legal reasons, whatever that might be. But I think even that is a learning moment.

Since then, I've seen a lot more detail come through. I've seen information that's been helpful. And I think that's really how it should be. Because when I go back in time, and I think this was in the 2014-2015 time period, and the landscape has changed a lot since. I don't know if people recall there was a large healthcare provider that was breached, and their response was phenomenal. It was a health insurance provider. And that CISO got a lot of kudos simply for transparency. Now, the nature of the breach, the lack of security capabilities they had, really didn't matter at that moment. What mattered was the level of communication and transparency that was put forth that shows that you care for your customers and the user community you're serving. And I think regardless, trust is number one. Contrast that with a large entertainment and media provider that got breached shortly after in 2015. Completely different. So, I think that's super important, and it's a learning moment with what we are witnessing right now.

Wolfgang: You know, the CSO of the organization you're talking to, I had the honor and privilege of meeting him a couple years after the event, and we were talking about it, and he goes, "It was in the news. It was in the news. We were having to testify. Everything was going on. And my mom called me, and you always think your mom has your back. Your mom always is like, 'It's okay, everything's fine. You did a great job.'" He goes, "My mom called me, and the first question she asked was, "Why haven't they fired you?" It's like, ouch because there's so much heat. But I think you're spot on, the authenticity, the trust, the clarity in which he communicated, guided that organization through. And, of course, kept his position until he was ready to move on.

Whenever any of these things happen, this applies really well to business continuity, disaster recovery, and instant response. Whenever any one of these things happen, I always think it's a great opportunity to document how it played out, use some of that story as a learning moment, and do a tabletop. "How would our organization handle this? Are we prepared? What would happen? What would that look like? Who would need to be involved? How are the communication paths established?" So, I think this is a really good opportunity for empathy. And it's a really good opportunity for us to learn and bring back to organizations and ask, "What if?"

Drew Rose: Yeah. I mean, empathy's key word. Saw several LinkedIn posts this morning around having empathy for this organization and what they're going through. And there's no right answer. There's no perfect answer. And you're not going to make everybody happy in this situation. There's a couple quotes that come to mind. The famous Mike Tyson quote, "Everyone has a plan, and then they get punched in the mouth." It's like you don't know. It's a state of shock. You're reacting. And this quote probably has a lot of references to business continuities and response, but this is a very clear example of like, "Okay, am I falling to the ground? Am I hurt? Am I concussed, or can I get up?" And am I ready to keep fighting?"

I think that the other interesting thing about the Okta service outage. This quote also came to mind. I don't think it's real, but I'll say it anyway. "So, one day Albert Einstein wrote on the blackboard, 9x1=9. Then, he wrote on the next line, 9x2=18, 9x3=27. He went all the way up. He got to 9x10, and he wrote 92, and everybody in the class mocked him and they just pointed at him and shouted, 'What are you doing? That's not even close to getting right.' And Albert Einstein responded and said, 'Hey, where were all the kudos and the congratulations when I got all of these simple math problems correct? Why is it this one mistake, and all of a sudden everything's wrong?'"

And I think that's what it comes down to. I think when we rely on services, there is definitely a certain reputation that we're targeting, that we can trust them with our data and the access and the service and whatnot. But I'm not sending out flowers every year. Servers X, Y, or Z don't get breached or when they're doing a good job, but I'm quick... Me, not personally. But people are quick to Twitter, quick to LinkedIn, quick to play armchair quarterback when one thing goes wrong that either affects them or affects maybe somebody else. And so, again, the empathy thing is huge, and the hug ops and the reliance. We're building services on top of each other to run our businesses from small startups like myself to large enterprises like the one you work for. And so, I think there's lots to learn, but there's also... I don't know of a time where there's been that perfect, textbook study response of this scale breach.

So, let's dive more into the planning phase of it. How do you ensure that the key players are aware of the role they play during an incident? Because, like I said, every plan goes out the door to get knocked in the mouth. How do you maintain that consistency, so when there is an incident or an event or a potential for a disaster, how are you ready and prepared for that? Vinitha, do you want to start in this one?

Vinitha: Yeah. I think the way I describe it, there are two components to that. One is when you create the plan itself, don't create it in a silo. Make sure that all the teams, the individuals, the roles they play, are involved and have inputted the creation of the plan. It may be the security incident response team and the CISO who owns the crafting of it, but at the end of the day, we need to make sure that all the folks who are involved and are related and are part of creating that plan. Now, once it's created, we all know practice makes perfect if you're lucky. So, it's really, really hard to say, "Look, I have a plan, and I know what I'm going to do in the event of an incident, a breach, a crisis, if it escalates to a crisis. It means nothing if you don't practice it.

And I actually think it's more nuanced than just having a technical team review your plan and do a tabletop exercise. I think you got to have different parties within the organization practicing the plan on a regular basis and align it to scenarios, real-life scenarios that are relevant to how your business is organized.

Drew Rose: That's really helpful. Wolfgang, what's your thoughts?

Wolfgang: Yeah, I'll dovetail on that. I also just saw in the chat a question about how often the tabletop exercises should be done. So, I'm going to bring all this together. First off, however great our plan is, it's out of date. It's out of date the minute we type the very last character. It's out of date the minute we save it up to SharePoint or however we're distributing it out, even if we just updated it. And so, why do I say that? Well, I say that because if we are going to be keeping everyone in the loop, if we're going to be making sure that stakeholders know what they're supposed to do, if we're going to be doing all that, it starts with really having an ongoing series of conversations because the plans out of a date and the human half-life of memory is six weeks.

So, six weeks from now, you're going to forget half of what's in that plan anyways. So, how do you keep it up to date? You keep it up to date with regular meetings, with regular emails, with regular communications, with tabletops, all those sort of things. And the question is, "Well, how many of those do you need to do?" And so, some of the data I saw on this most recently was five. Five is the magic number. Probably because it fits in one hand. I like that. That's simple. But five activities in any given month are correlated with the highest degree of recovering after continuity and business disruptions and security incidents. Five activities. Now, what are those five activities? Is that reviewing a [inaudible] ? Is that reviewing a plan? Is that doing a tabletop? Yes. Yes. All of those.

How many should we do? Should we do all five? Well, that comes back to why are you doing them? You're doing them to build trust. You're doing them to build muscle memory, like someone else mentioned in the questions. I love that point. Muscle memory is a must. And you're doing them to build knowledge. And you're also doing them to update your plans in your environment. [crosstalk]. I just want one of [inaudible], Drew. I'm going [inaudible].

Drew Rose: Finish, please.

Wolfgang: But the amount and pace of planning should be commensurate with how quickly you can remediate anything you find. And if you've got a really easy way to update those documents and a really easy way to update the technology, five.

Drew Rose: Yeah, no, that was a solid point. Again, not all data is created equal. Some we care about. Some we don't. Some, it doesn't matter if we recover immediately. We can wait a day or two to get it back. I think the word that comes to mind for me is empowerment. It's just like we'll get back to the fighting. I don't know why I'm constantly talking about fighting today, but when you get into a fight, you have a game plan. You don't exactly know what moves you're going to make when. You have to be able to respond to the situation at hand, so you have to feel empowered to make those decisions. And we need to be able to empower our teams that are involved in incident recovery, disaster recovery, business, continuity, to be able to go to the right places to reassess what they're supposed to do if it happens.

Instead of memorizing and having to remember exactly what to do when it happens, empower them to be like, "Oh, I don't exactly know what to do, but if I go here, it's going to tell me. And I am confident enough because I went through the five different activities to know where to go, to remember that when that time happens."

Speaking of tabletop, I'm remiss to not say as the pandemic kept going on and nobody was going back to work in the office, we recognized there was an opportunity to raise awareness around incident response to companies outside of the executive and technical teams. And so, at Living Security, we created something called the virtual tabletop experience. It's called the war room. It's built for managers and directors and VPs of all shapes and sizes to get that empowerment, to understand if they're at home woman it's midnight and something comes across their email or they see something online, they know who to talk to, who to call, who not to call, what's important information to give, and who are the different members of that team.

So, I think that panned out some of the remaining questions in the list. So, I think we have a little less than 10 minutes left. Oh no, we have more time. We have 13... What is that? 23 more minutes. We go for the hour. Awesome. So, maybe we can get a little bit more technical on some of those plans of testing for effectiveness, Vinitha. What Wolfgang was saying around five different activities, did that resonate with you? And do you have any examples from your time probably going out to organizations and helping them construct and then test these plans that you can share with the audience?

Vinitha: Yeah. And by the way, that did resonate and, Wolfgang, I like that the five doesn't necessarily mean you got to... I think I heard you say something like monthly, right? Doesn't mean you got to have five tabletop exercises a month. It means you have to do some activity related to incident response outside of your day-to-day job, like you said, reviewing the plan, reviewing the [inaudible], having people outside of the InfoSec team review it. But, Drew, to answer your question, I'll go back to saying that whether it's a technical tabletop exercise or it's an exercise for executives, I'll say I'm a huge fan of making sure that the C-suite in your company or whoever else who ends up needing to get involved in the event of a cyber crisis, that they go through this exercise, as well.

Because like the technical folks, they're human beings at the end of the day. They need to build that muscle memory. And while they all acknowledge that it's important to know what you need to do, it's really, really important to make sure that the PR piece, the executive piece, the time when the incident gets escalated to the C-suite, you can't have your C-suite sitting there and saying, "Well, what do we do now?" Or in the likelier scenario, having them having differing opinions within the C-suite. We've all heard stories of reporters standing outside the CEO's house or standing outside a board member's house. And in that event, do the executives know whether they can share, what to share, what not to share externally, when to share it? So, I'll talk about structuring a tabletop exercise from a technical standpoint, I think it's really important to make the design, or I'll call it a scenario in this case, very, very relevant to your environment.

If you are in the medical device industry, having it structured to something completely unrelated isn't going to matter. Really understanding how a particular attack can travel within your organization and how your technology is used to conduct business is super important. So, the more real the scenario is the better the outcomes are going to be. So, that's what I'll say from a technical standpoint. Now, from a standpoint, I think the biggest things that you're trying to assess, test, or evaluate in a simulation is, "What are those escalation thresholds and triggers? What are those coordination and communication triggers?" What I mean by that is once it gets to... Have you appointed someone in your C-suite who's your security champion, who's your quarterback, if you will? Or are you going to have 10 different people with 10 different opinions in a situation of chaos and crisis?

That's what I mean by escalation, communication, and coordination thresholds. If it's time for your PR team to put out something to the public, do you have a template? Do you know what you should share and what you shouldn't share? Have you engaged the right regulators and folks in law enforcement? I think you don't want it to be the first time doing this, especially even for the C-suite, when you're going through a breach. So, those are some of the things that I'll share when it comes to not just crafting, but who the audience I think you should include in different types of tabletop exercises and cyber simulations.

Drew Rose: Yeah. I think some of the things that you brought up, the items you brought up of importance were the shortcuts. Like fast forward through the incident, as we go through the response, what are the things that we could do right now? The legal response has always been an interesting one because it's always the one that we can kind of blame the lack of transparency on legal and then collecting the information and getting together. But internally from an ethical perspective, doing the right thing, we should have our ducks in a row.

If legal is important to the response, it may cost us a couple thousand dollars, maybe more, to have them involved in this whole process, depending on the firms you're working with, but they should be involved, and they should also have those templates. And when it happens, we know who to go to, and it's not a full red line review back and forth process that could take hours or days. It should be something that on the first iteration, we can get out quickly. So, that is a great response.

There's some great questions coming in. This one's really, really interesting. And we have a little bit of firsthand experience here, and I can't wait to hear the thoughts on my panelists. But the question is, "What items do you put in your DR plan to protect the mental wellbeing of the responders, and how do you review it during your planning process?" Now, my initial response before I ask for feedback is... I'm probably wrong. ... If you don't. I think the mental wellbeing is paramount. If that's not embedded in your culture already, a plan isn't going to be helpful.

And I think that comes to, like with dealing with Ukraine, we have contractors in the Ukraine, and we've just been very sensitive to the situation of, "Do they want to work? When do they want to work? How do they want to work?" And just being okay with a lack of productivity during this time period and versus like, "Oh, man. We can't afford this lack of productivity. We are going to let them go find someone new." And so I think... I don't know. I'd love to hear your thoughts on that response. Wolfgang, you want to start us off?

Wolfgang: Yeah. That was actually one of the things that I learned from Duo Security. I really appreciated when the pandemic hit because it was incredibly hard to work. It was incredibly hard to think. We were all worried. And we paused and redid our roadmaps. And I remember the head of product saying, "Yeah, when these things happen, that is not productivity as usual. It's not business as usual. We need to rethink what we're doing so that we can be cognizant and, again, empathetic and supportive of our folks." So, I really, really appreciate that. Especially coming from the consulting world where... Oh, true story. One of my colleagues reported to me, she had a baby. And then, the question was, "Well, how soon is she getting back to billable? Is that next week? Is that a... " "No, give her some time. What do you mean is that next week?" "Wha are her numbers?" "Don't ask that!"

So, I do think that technology in general as a discipline, instant response, continuity and recovery, in particular's capabilities, need to be cognizant of that human factor. And we're a SaaS company, and there's SaaS companies on this panel watching, I would encourage you to have adjustments for those plans, so when things happen, you give your people time to recover, and you redo your roadmaps, and you redo your priorities.

Drew Rose: Yeah. We've been in an active DR exercise for the last two years. As a business owner, I think through the Great Resignation where everyone's recognizing they can work kind of anywhere and at any time, it's a company that responded with empathy during this disaster to their team that is going to retain the most talent because it's incredibly easy to jump ship right now. In the technical industry, a hundred percent. So Wolf, love that response. Vinitha, what's your thought? I know you started Rubrik just a little before the pandemic, and I know we've had some conversations around how the culture has been during this disastrous time. So, I'd love to hear your response on this topic, as well.

Vinitha: Yeah. No, I think absolutely. I think you said it right, Drew. Culture and work-life fit and mental wellbeing shouldn't be something that you insert into the process at the time of crisis or during certain events. I truly think it should permeate in an organization. And honestly, I say this a lot about security teams. We, a lot of times, engage in, like we have to wear strategic hats when we plan our roadmaps, when we talk about what we want to fix in the long run, but we also have to wear very, very tactical hats. Our day jobs involve the game of Whack-A-Mole. I always tell my teams, "When you prepare your roadmap, make sure you add and just forget about, even if we weren't in a pandemic, even if we are not dealing with a cyber breach, when you prepare your roadmap, you need to have buffer every quarter, every month, in the event of a fire drill, because then it's all hands on deck."

Because during a crisis, to be honest, we aren't really thinking about mental wellbeing. It's similar to if you have a crisis at home, and if your baby hurts herself, you are not going to go take a nap. What's going to help you really execute in that situation, if your mental wellbeing was strong throughout? So, I'll say that's something we need to practice on a regular basis because just for security teams, we have very, very tough jobs. And let's just acknowledge that the way the security landscape has evolved, the hackers have already out beaten the number of security talent that's out there. We're not producing enough security talent. We, as a community, I think need to encourage and inspire more people to get into security if we even have a shot at this. So, I think security culture, company culture, that's just paramount if we want to tackle this issue as a team, collective team.

Drew Rose: Yeah. Very, very, very well said. Yeah. Our job cybersecurity is very, very much a thankless job. So, I'll take this opportunity to everybody in this call to thank you. You're doing a great job. I know you're trying your hardest, and I know you don't get thanked enough. So, that's my opportunity to put it out there. Now, back to the tactics. Okay. "Should we force our vendors to have tabletop exercises DR testing?" So, I think that's a two part question. Should we force them to do it on their own? Or should we force them to be involved in ours? And I know that depends on leverage and clout whether that's going to happen, but that is very interesting to think through what that would be like. So maybe, Wolfgang, do you want to talk through your ideas around vendors and tabletops?

Wolfgang: Yeah. So, there's a couple different ways to take that. One, I have led tabletops where we brought in key vendors to be part of that conversation and to answer how they would act. So, depending on the vendor relationship, you may want to do that. The second thing is part of your vendor risk management. If you have a more mature vendor risk management program, if you've got a mature supply chain security program, I would encourage you to have tabletops as part of that checklist. I mentioned earlier that one of the things that I went through before I joined Duo Security was auditing all our, not all of our tier one, I'd still be there. All the tier one vendors for their continuity and recovery plans. And one of the things we looked at was, "How often are you doing tabletops? Show me the evidence of your last tabletops." And we uncovered a lot of things I think they probably wished we wouldn't have, by asking those questions. So, certainly ask the questions and certainly use the power of purse via supply chain management and [inaudible] risk management to drive that.

Drew Rose: Awesome. Vinitha, what's your thoughts? And you are on the vendor side, as well. Are you hearing this in the vendor risk assessments, and I'm sure your team is completing.

Vinitha: Yeah. No, my team's involved in both. So, the customer trust piece of it where we are the vendor, and then also the supplier's program. I think it goes back to, again, which of your vendors do you consider are critical? And the reality of it is, let's just face it. We're not going to be able to demand a lot through contract with some of our larger vendors, whether it's AWS or Google. I think being able to understand what risk they pose and having those relationships is really, really important.

So, I don't have a lot else to add apart from what Wolf did. I think it's a hard problem to face. I think if your supplier security program is fairly mature, you can then start with a few vendors who you think are important to your supply chain and see how that relationship would help you figure out. Because it's hard. A lot of these things we see, sometimes vendors are willing to play ball, and they may have their own reasons for doing certain things or not. And sometimes, it's hard to get them to do things unless it's actual law.

Drew Rose: Yeah. And we're having a tabletop conversation. I think tabletops are not stationary nor are they just for assessments, so what do you post exercise?

Vinitha: Do you want to go for it, Wolf? Are you on mute?

Wolfgang: I can. I'm not sure. I wasn't following the question. What do I [crosstalk]?

Drew Rose: Yeah. So after the exercise is completed, what happened? Are you just done, or do we wait until next year for the next annual review?

Wolfgang: Oh yeah. Just one and done. So, yeah. You want to do an after-action report. You want to, oftentimes, have a change request for any updates to your code, your systems, your configurations, your IT. You want to make sure you thank the people who are involved because it's a time commitment. And so, furthering the relationship, further tuning the capabilities, and any updates to the plan.

Drew Rose: Yeah. That's helpful. Vinitha, your thoughts on anything [crosstalk].

Vinitha: The only thing... Absolutely all of those things. I think the after-action report reveals a lot. It reveals what we need to fix, both from a systemic and technology and process perspective, maybe people, but it also reveals how well we did in this exercise. And sometimes, apart from the more technical components, you may decide that, "Look, I think for our organization and for what we do and the kind of risks that we face, we need a crisis management team. We have an incident response team. We have a BCDR program. But we actually need a crisis management team. And we need a crisis champion to come and quarterback this every time an incident escalates to a crisis."

So, some of those things that come out of your lessons learned or your post incidence can be more programmatic in nature, as well. So, I think it's important to assess various facets of that and make the changes that you can and really right size it for your business.

Wolfgang: Now, I want to pull on something you just said there, cause it reminded me of the metric side of things, the measurement side of things. We, several years ago now, so this is back to hard drives, but I think the concept still applies, we gamified it. We measured how long it took to recover. And if they beat their score there were awards and prizes and a celebration. And if they didn't, well, we'll still have lunch, and we will tell a couple war stories. So, I do think there's a lot of ways for using what you just said, that is the metric side of things, the after-action report to drive culture and drive the team forward.

Vinitha: I love that, Wolf. I love... I say this all the time. For various things in security, we do that with the security culture at Rubrik. We're trying to do that with some other areas of security. Gamify it. Because I think if you insert positive actions... By that, I mean, you incentivize your reward and gamify versus slapping people on the wrist. It's just human behavior. It just gets people to respond in better ways and to sort of enjoy it through the process. I love gamifying.

Drew Rose: Yeah. No, that's a solid response. Definitely something we were trying to achieve with [inaudible], just ability to enjoy the experience as a team and go through the exercises as you would, so when you get into a response situation, when you get into this fight, flight, or freeze, you're able to kind of dismiss that very primal response, get to your core brain functions, and get to the next step as quick as possible, which is what we want in any response plan here.

I got time for about two more questions, subjects. This is something I don't know enough of. Is there a rule of thumb or some type of maybe misguidance or ISO guidance around how much time IT teams, cyber teams, should be spending in preparation and exercising these types of plans?

Vinitha: I don't think there's... And I'm probably not quoting the exact framework, but I doubt there's any guidance on how much time they should be spending. The only guidance I've seen that certain laws and regulations require is basically reporting an incident. So, something like GDPR says within 72 hours of identifying it as an incident, you need to report it. And you have different state and local laws, as well. So, I think the general guidance is for your business, for the assets that are important to you, for the way you are organized, make sure there's enough preparation.

So, I don't think there's guidance around that, but then I will say this sparked another thought, Drew. And I just read this yesterday. There's something that came out of the federal government recently. I think it came out in early March. There's a lot of chatter around it, which is SCCs are going to start... I can't remember the actual timeline. Pretty soon, they're going to start requiring public companies to start including the maturity of cybersecurity capabilities, reporting on past and present incidents, and what they're doing about it. So, they're going to start requiring that in their 10-Ks and other disclosures. And that I think is a pretty drastic shift. It shows you how important cybersecurity incident response and what you're doing about it as a company has now become to our society because we've seen the impacts they've had. So, I thought that was pretty interesting. I just read about it yesterday.

Drew Rose: Wolfgang, what's your thoughts? Or do you know anything that we don't know? You probably know a lot. We don't know, but...

Wolfgang: I think that was well said. The SCC guidance is really interesting. Just as a rule of thumb, you want to prepare enough to be ready, but you don't want to prepare too much, in so far as if you're not doing anything, you're not in a good spot. Right about here is good. When you start doing too much where you cannot absorb those changes and update the plans and make the technical changes, that's when you've gone too far. So, every organization, I think, is going to have their own sweet spot. It's just being aware that that sweet spot exists and looking for it.

Drew Rose: And to your point, you need to be doing something across the board, the bare minimum, and then focusing on the critical areas. So, yeah. We've got less than two minutes left. I think going back is one of the first poll questions. This will be the sign off. But for the people in the audience that do not properly have a plan ready or they're in the process of building one, what is like the first thing, or what do you think is the one thing they should reconsider as they're developing this plan or starting on the path of either incident response or a BCDR plan? Vinitha, why don't you start us, and then go do the little closing statement?

Vinitha: Yeah. And sorry. The question was, if they don't have a plan today, what is the first thing that they should do?

Drew Rose: Yes.

Vinitha: Right. I think it's to start building components of the plan and really identify, based on how your organization is structured, if you were to have a security incident, who are the players that need to be involved? Again, going back to the people component of this, because figuring out the how and what you need to do technically is hard, but it's easier than figuring out people's roles.

Drew Rose: Yes. Make sure in your budget, you have gift cards or gift baskets for these key essential folk. And before you even start, a little bribery never hurt. No one's getting in trouble for a little bribery in this situation. Vinitha, thank you so much for your time here. Wolfgang, you can close us out, as well, before I say goodbye.

Wolfgang: Yeah. So, once you know the people, what do you do? No one has the same plans. If you ask the people who are the technical stakeholders, they all have in their heads how they're going to recover. So, capture that information to start with. If you ask the business people that we identified, "What matters to you?" you'll be able to start getting an idea of where your crown jewels are and where your systems are. So, even if you have no plan, start with the people, and then ask folks, because everyone has an idea. So, start there.

Drew Rose: Yeah. To summarize, you're not on your own, and there's a lot of answers you're not going to get by yourself. And so, the people that you're bribing, bring them to lunch, ask them the questions, and they should help focus. Thank you, Wolfgang and Vinitha, so much for your time this afternoon. I have some really great takeaways and some great quotes. Thank you to the audience for sticking around for the last hour. I know we bounced around some different [inaudible] areas, but I hope that you are able to have some takeaways to bring back to your organization. As always, if you are looking for security awareness training and virtual tabletops, quantifying human risk, reach out to us at Living Security. We're having a lot of fun doing it. We're doing some really cool, innovative things, and we would love to have that conversation. Thank you everybody for a great afternoon, and enjoy the rest of your week.

Wolfgang: Take care.

Vinitha Thanks.



Subscribe to Learn How to Prevent Cybersecurity Breaches

Additional Reading