From weak credentials, misconfigurations, identifying private keys or API keys, and other vulnerabilities, IoT firmware can make you easy targets for bad actors. Thomas Pace, CEO and Co-Founder of NetRise, discusses the biggest firmware vulnerabilities and how to fix them before they become a problem. He breaks down how firmware analysis is done, what is software bill of materials (SBOM) and additional challenges surrounding the space.
About Thomas
Thomas is currently the co-founder and CEO of NetRise, a cybersecurity company focused on providing visibility into devices to identify vulnerabilities and risks via firmware analysis. Before NetRise, Thomas served as the Global Vice President of Enterprise Solutions at Cylance. His responsibilities ranged from conducting incident response investigations, product marketing, public speaking, and analyst relations. Thomas was also responsible for ICS security at the DOE for three years and served in the United States Marine Corps, serving in both Iraq and Afghanistan. Thomas has spoken at Black Hat, DEFCON, RSA, and was interviewed on 60 Minutes and Last Week Tonight with John Oliver for his efforts related to ransomware.
Interested in connecting with Thomas? Reach out on Linkedin!
About NetRise
NetRise provides visibility and risk identification to a class of devices (IoT, ICS, MedDev, telecommunications equipment) that historically have had no visibility with the intention of providing clear recommendations to remediate these risks efficiently.
Key Questions and Topics from this Episode:
(00:33) Introduction to Thomas & NetRise
(01:13) Firmware analysis
(02:47) XIoT vs IoT firmware
(04:38) Biggest vulnerabilities and fixing them
(06:43) Process of analyzing firmware
(08:49) Difficulties of software bill of materials
(12:22) How to approach device security
(13:53) Challenges in IoT
(15:56) Exciting aspects in IoT for 2023
Transcript:
– [Ryan] Welcome to the IoT for All Podcast, I’m Ryan Chacon, and on this episode, you’re going to get some really great insights into the biggest firmware vulnerabilities and how to fix them. And I’ll be spending time talking with Thomas Pace the co-founder and CEO of NetRise, a leading firmware security company that provides visibility and risk identification to a class of devices. Ton of value here in this episode. If you’re watching this on YouTube, we truly appreciate it if you would like, subscribe, and hit that bell icon, so you get the latest episodes as soon as they’re out. Other than that, let’s get onto the episode. Welcome, Tom, to the IoT for All podcast. Thanks for being here this week.
– [Thomas] Happy to be here, man, thanks for having me.
– [Ryan] Absolutely, let me kick this off by having you give a quick introduction about yourself to our audience, if you wouldn’t mind.
– [Thomas] Sure, so I’m Tom Pace, co-founder and CEO of NetRise. NetRise is a company that is providing visibility and risk identification to a class of devices known as XIoT, Extended Internet of Things, which includes IoT, ICS, medical devices, embedded systems in vehicles, satellites, and telecommunications equipment. And we provide that visibility and risk identification by doing scalable automated firmware analysis.
– [Ryan] Fantastic. One thing I wanna ask you real quick. So when it comes to firmware analysis, how has that kind of just historically been done? You know, why is it a focus for you all and kind of what’s the value when you’re talking about doing that at scale?
– [Thomas] Yeah, so I started my career, or rather I, you know, prior to this, I was working at Department of Energy, 2013, 2016. Doing industrial control system security. And one of the things I was tasked with is determining the impact of various vulnerabilities and risks against our ICS devices, and, you know, we had a few of those. And what was quickly determined is we basically had no technical capability to be able to answer those kinds of questions in any kind of meaningful way, either at scale or not at scale. So the only real way, that this problem has been approached historically, has been by doing like manual consulting engagements essentially where if you’re a device manufacturer, you interact with some consulting firm who rips apart the firmware, does as good of a job as they can in identifying some components, finding vulnerabilities, et cetera. Problem is, that’s a very snapshot in time kind of assessment. And then that’s wildly unscalable for a number of various obvious reasons. And then the only other way is relying on the device manufacturers themselves to publish all of the issues that are known in their devices which obviously is never going to happen.
– [Ryan] Right, right. And you mentioned XIoT that’s actually kind of a newish term from a lot of conversations that I’ve had. When it comes to the firmware side of things, how does the approach differ when it comes to XIoT, and then, you know, other IoT devices that you’ve kind of experienced or other people out there maybe using themselves?
– [Thomas] Yeah, so XIoT devices are typically running like an embedded operating system. So, you know, like a ton of these devices are just running like embedded Linux. So it has a file system, it has, you know, user directories, it has applications and configuration files, and normal things you would expect to see in just a Linux operating system. We really break it down into two or three different categories. You have embedded Linux, and then you have RTOS, realtime operating systems. So things like VxWorks, Green Hills, uCos, eCos, that is generalizing, but typically like a single binary operating system. And so much, much harder problem. I mean, we support that as well. And then you have Windows firmware which is pretty self-explanatory. So the things that are different than that would be things that are just like machine code, you know. If you look at stuff like BIOS firmware, and like UEFI stuff, graphics card firmware, network card firmware. There’s a wide range of what that can be. But we tend to focus a lot more on like the embedded operating system side of firmware. So if you look at firmware for like laptops, we typically aren’t super interested in the firmware that’s on those. But we are very interested in firmware that’s on like router, switches, security cameras, printers that kind of stuff.
– [Ryan] And what are some of the biggest vulnerabilities when it comes to firmware that, you know, our audience out there listening to this should be thinking about, if they’re, you know, they might be a little unfamiliar or disconnected from the firmware side, but what should they be thinking about when it comes to those vulnerabilities, how do you kind of approach addressing them, fixing them, making sure that they’re not vulnerabilities anymore?
– [Thomas] Yeah, I mean, number one, I think people will be very surprised to see the types of vulnerabilities that exist on these devices are the same kinds of vulnerabilities that exist on their traditional IT assets. There’s no difference. They’re typically a lot older. And so, but other issues we identify that are not, you know, directly known as like vulnerabilities or CVEs, things like weak credentials, misconfigurations, identifying private keys or API keys. So there’s a number of other things that we identify that are incredibly risky and problematic, aside from just like traditional CVEs. And so in terms of how you go about addressing those, really depends on the persona. So if you are a device manufacturer obviously you have a number of additional things that you are able to do to remediate or mitigate those vulnerabilities as compared to an enterprise customer who has purchased your device, who is not able to update the code at their leisure. Technically you could, but then you void the warranty and you cause a bunch of other problems for yourself. So if you’re a device manufacturer, you can update the components, you can add in other mitigations and things like that. If you’re an enterprise user, you can apply pressure back to your device manufacturer, you can apply the latest firmware updates, you can segment those devices in some capacity, apply, you know, additional monitoring capabilities around those devices, leverage a platform like ours for like third party risk management and procurement purposes. We’ve used that. So number of things you can do on both sides.
– [Ryan] Fantastic. And what is like, I guess, if you were to talk somebody through kind of the process you go through when it comes to analyzing the firmware what does that process look like timeline-wise? You know, how does that usually go? What’s involved from the company that you’re working with side versus what you all handle? Like just just to kind of at a high level, I’d love to learn more about that.
– [Thomas] Yeah, once again, I’ll break it into the two different categories. So if we’re working with device manufacturers, in a significant number of those scenarios, we can come to the first meeting we have with them with their firmware in the platform already, because it’s just available on the internet, or we’ve gathered from some other location, right? Or we bought a device and ripped firmware off of it, whatever. So it’s very easy to get a hold of the firmware by downloading it from publicly available sites or support portals or whatever it is. And so we just upload it into our platform and you know, there’s so many things that depend on processing time but for the average size of firmware, you’re looking at, let’s call it between 15 minutes and an hour maybe. Some stuff takes a very long time, ’cause there’s like a hundred file systems or something crazy. But so, but if we come to that without that from a device manufacturer perspective, very easy. They have all of the firmware in some repository or something, and getting that set up for them to upload firmware, I mean literally takes like five minutes. It couldn’t be an easier process. For the enterprise customers, there’s like an extra step. And that step is you need to identify the devices you care about and identify the firmware versions of those devices. And once that happens, we can work with them and say like, “Okay, we already have that firmware,” or “We need you guys to go download that firmware from the support portal and then upload it to us.” So that’s the two basic routes to get firmware to us. I mean they take very, very limited amounts of time.
– [Ryan] Fantastic. One thing I kind of, slight transition, I know we were talking about this before we jumped on here, but it was talking about software bill of materials and kind of what that means, how that’s generated, and how when you’re doing that for firmware, it is kind of tough. I think that’s something to shed some light on. I’d love it if you could kind of talk through why generating a software bill of materials for firmware is tough, what those challenges are, and then at the end of the day, like obviously it’s important to do it and what is the real value there?
– [Thomas] Yeah, so the reason it’s challenging is because you have a huge problem at the front end. And that problem at the front end is what is generally known as extraction. And what we mean by that is not extraction of the firmware from the device, that always gets confused. What we mean by that is extraction of basically, something we can analyze from the firmware image. Now some firmware images have nothing. There’s nothing to extract. It’s just like a regular old file system and we’re good to go. However, lots of firmware has like proprietary compression algorithms or proprietary file formats or this file system isn’t a file system we’ve seen before. They might be leveraging encryption, which I use that word, I’m probably giving it a bit more credit than I should. Sometimes people just like XOR things and they’re like, “It’s encrypted.” And we’re like, “Okay.” So there’s an infinite number of things that can happen from the extraction process that need to be addressed. So that’s the first really big challenge and that’s a non-trivial problem. And then once that happens, the ability to identify component, this is all done on zero knowledge analysis, right? We are not operating under the assumption that we are working with the device manufacturer. And even if we were, they don’t always know what’s what. And so things like symbols are being stripped from binaries which makes identifying things more difficult. Back ports make things more difficult. Like, “Hey, we updated that component actually,” but then didn’t change the version. And it’s like, “Okay.” So it’s super common though, happens all the time. So those things, and then you have just a bunch of proprietary binaries that might not have like a consistent versioning system or any versioning system whatsoever. And so how are you to even identify that in a unique way? So bunch of problems there that you don’t really tend to run into. And in terms of the value, I mean tons of value here. Number one, the most important thing is like, do I have this thing? Whatever this thing is, if you’re looking, “Hey, we wanna see if we have this component, because we have reason to believe that this component was contributed to by a person from this country and we don’t want that in our device.” Or, “Hey we know that this component has a critical vulnerability and we wanna know where is that component in our environment.” Answering that question right now is totally impossible for enterprise users. It’s impossible. They have no way to do it. The only opportunity they have here to get that question answered is by reaching out to every single device manufacturer of every single device they have in their environment. And the overwhelming majority of those device manufacturers will not be able to answer that question for them.
– [Ryan] Fair enough. Okay, fantastic. Let me ask you another question about when it comes to device security and kind of challenges as it connects to that, from your all’s perspective of things and the stuff you deal with it on a daily basis, where do you kind of see, or how do you kind of approach device security? I guess we can start there.
– [Thomas] Yeah, I mean we really approach this problem in the same way you approach just about any other cybersecurity problem. The first step has to be getting visibility. And the way you get visibility to this problem set is by, the best way to do that in my opinion right now is generating a software bill of materials. Now there are other things that are important that are not included in a software bill of materials, as I mentioned, like private keys, weak credentials, et cetera, et cetera. So it’s not just about an SBOM as there’s no silver bullet as always. So once you have the visibility, now you can do all of these other things. Vulnerability correlation. Are exploits available? Should we change this password? Should we delete this username? Should we update this private key? Like whatever it is. But you can’t do any of those second order actions until you have very good visibility into what’s actually present on these devices from the firmware.
– [Ryan] Gotcha, gotcha. Okay, very cool. Last thing I wanted to ask you kind of as we wrap up here is around the challenges, other challenges that you see in the space from, you have a very unique perspective that you bring coming to this, coming to, you know, IoT devices, XIoT devices, from, you’re looking at the space from probably a different lens that a lot of the people I’ve spoken with. I’d love it just to kind of hear as we’re into 2023 now, what are some of the biggest challenges you’re seeing companies that you work with kind of struggle with or battle against? And kind of just your thoughts on that.
– [Thomas] Yeah, I mean as it relates to this space, I think a big problem that companies are gonna have to come to grips with is, we are generating a significant amount of net new data that they did not know was there. So, and we know that, and we understand that. And so we’ve taken actions to attempt to soften that as much as we can by doing things like enriching our vulnerability information in a bunch of different ways to allow people to prioritize those in ways that make sense outside of just what’s available in the NVD, which is nowhere near sufficient to do proper vulnerability prioritization. So that’s gonna be one of the bigger challenges. Firmware acquisition is always going to have some edge cases that need to be addressed. So those would be a couple of them. Also combining, you know, this idea of outside in and inside out, you know? Outside companies might be like Armis, Dragos, Claroty, Nozomi, like those kind of people. And then inside out would be someone like us. And so that puts the whole picture together for you. It’s like having a network intrusion detection system and EDR, right? Which everybody agrees you should have both at this point, I think. So we’re really just applying that same kind of thought process to a class of devices that has basically been ignored forever.
– [Ryan] Fair enough. Okay, awesome. One thing just because this is like an episode we’re doing earlier in the year, I just wanna get your kind of just quick raw thoughts on things you’re most excited about going into, you know, as we’re into this new year. We talked about challenges. We talked about a lot of stuff today, but just like breaking away from that, is there anything from an industry perspective as a whole that you’re most looking forward to? Or it could be more drilled down to the kinda the world that you focus on on a daily basis, but just outta curiosity, anything that comes to mind?
– [Thomas] I mean, we are seeing a ton of traction in this space. Companies coming to us saying like, “Hey, customers are demanding this and demanding that.” So what I’m excited about is kind of the, I don’t know what the best way to say it is, the mainstream acceptance of this kind of solution. It’s a really kind of rewarding thing. I mean, two years ago, the word SBOM wasn’t exactly the most well known piece of nomenclature. And now that’s like all anyone talks about for better or worse. And so, you know, that’s what’s really exciting to me, is like to see the momentum that is picking up around this problem, is just like, you know? It’s funny when people say to me things like, “Wow, you like really timed this well.” And I was like, “Yeah, yeah, I sure did.” Like, I wish I could say I had that.
– [Ryan] Take credit for it, right? You took credit for it. Yeah, right, exactly.
– [Thomas] That’s how it goes, man. You always gotta, you know, the harder you work, the luckier you get. So I just-
– [Ryan] Yeah, you’re putting yourself, you know, from your experience and background and then, you know, finding an opportunity to start a company to solve a particular problem, and that particular problem continuing to get bigger and being something that really needs focus and attention is what you were putting yourself in the position hopefully to capitalize on, assuming that your thoughts on that were right. And seems like that there’s a lot of need for this. So that’s, but you’re right, it’s the harder you work the luckier you get. So, but I don’t think it’s all luck.
– [Thomas] There’s a little bit of other stuff in there.
– [Ryan] Yeah, for sure. Well, Tom, thank you so much, man, for taking the time. We’ve had, I think, maybe one or two guests in the past have talked about firmware, but to the level as you’ve been able to dive into it today. And I know the work your company’s doing is really much needed and very fantastic stuff. So I really, you know, push all of our audience members to check out what you have going on. And hopefully we can do more stuff together and definitely have you back to talk about further topics throughout the year.
– [Thomas] Happy to, man. Thanks a lot for the kind words and I appreciate you having me on.
– [Ryan] Sounds good. We’ll talk soon.
– [Thomas] All right, bye-bye.
– [Ryan] All right everyone, thanks again for watching that episode of the IoT for All Podcast. If you enjoyed the episode, please click the thumbs up button, subscribe to our channel, and be sure to hit the bell notifications, so you get the latest episodes as soon as they become available. Other than that, thanks again for watching. And we’ll see you next time.
Industry News
