Patch Diffing Entire Firmware Images, Squeezing Out Bugs

Firmware patch diffing is a relatively under-documented process, but one that can be really important when doing IoT security research. In this post, I’m going to pick apart a Netgear firmware image, and figure out the root cause for one of the bugs in it. We’ve also just released the first revision of our GraphQL API, which we’ve started using internally to speed up a number of repetitive tasks. One of those is patch diffing, so there will be some light references to the API in here today too. As part of our ongoing research, we often “patch diff” firmware images. We might do this to identify the root causes of bugs we haven’t discovered ourselves, or to figure out exactly how a vendor has patched an issue we disclosed to them. Patch diffing is the process of cross-comparing the vulnerable and fixed versions of a product. With this, and some light metadata, more often than not we will be able to pin down where the product has changed, and therefore insinuate exactly how it was vulnerable in the first place and how the issue was fixed. [caption id="attachment_3978" align="aligncenter" width="1113"] [/caption] Traditionally, patch diffing was done with two compiled binaries. Using tools like , , , etc. it's (relatively) simple to see exactly where and how a compiled executable has changed between revisions. However, turning this into actual insight is not necessarily a painless process. You have to be comfortable enough staring at endless assembly, figuring out exactly where the bug you are looking for might have been. And if there are lots of other non-security related changes made between revisions, you’ll also have to spend a lot of time sifting through noise. Patch diffing firmware then adds another layer to the puzzle. For a start, we often have to work with entire filesystems, rather than the single patched file in isolation. We need to be able to drill down within a firmware image to focus only on files, which have changed over a given revision. We then need to compare each of those files to figure out exactly which section within each file is relevant to the bug, and which changes are less relevant for us. This can be quite time-consuming, especially if you do a lot of this kind of work. Lucky, we’ve recently added a GraphQL API to make diffing and querying firmware images for this purpose a bit quicker and easier. Let’s identify a bug we want to figure out the root cause for. Of course, I’m interested in all bugs, but near the top of my bugs-of-interest list is exploitable WAN-side router bugs. Netgear recently released a series of patches for . The details in the official advisory are thin (read: there are absolutely no details), as Netgear has bundled 9 different fixes into one post and communicated close to nothing regarding the actual issues they’ve fixed. [caption id="attachment_3980" align="aligncenter" width="1078"] [/caption] However, many of the issues also have ZDI IDs, which mean that they were disclosed by the (ZDI). The ZDI recently published a few advisories related to their 2020 Mobile Pwn2Own , a couple of which were related to a WAN-side RCE bug in the Netgear R7800. I’ve had my eye on that one since November, so let’s look at the details provided by ZDI. The issue we’re going to focus on is the snappily-titled “”, aka CVE-2021-27251, used by “Team Flashback”. It affects firmware version (and likely many before) of the R7800, but was fixed in version . At the time we did this diff, IoT Inspector wasn’t picking bugs exactly like this, so it warranted a look into to enhance our detection capabilities. Armed with what we know from the advisory and both the vulnerable and fixed firmware versions, we can start trying to find the root causes of this issue. While some firmware update blobs may contain only the files, which have been patched, this is quite uncommon in router firmware. The firmware blobs we get from Netgear both contain pretty much the entire device filesystem. [caption id="attachment_3982" align="aligncenter" width="675"] [/caption] This means that in each firmware image, not only are we going to have the changed files we’re interested in diffing, we’re also going to be sifting through files, which haven’t been modified between revisions. In order to lighten our workload, we need a way to figure out exactly which files have changed between these two revisions. Probably the most efficient way to do this is to hash every file in the image. Then, for each comparable file, check whether the hash has changed between versions. If you’re an IoT Inspector user, you can use our GraphQL API to query firmware images. The GraphQL API exposes a firmware type, for which you can list all the files and their associated hashes. I’m going to quickly go over how you might query the API so if you’re not (yet 😊) an IoT Inspector user, you can feel free to skip this bit (or ). The query is pretty simple. For the given firmware, we can ask for a list of all the files, along with properties for each. Notably, we can also ask for the SHA256 hash for the file (SHA1 and MD5 are also available for those who like multiple hash types). The response is just a JSON object we can easily ingest into a script. Once we have the list for all files for a given firmware image, we can filter and compare them however we want. In this case, since this particular Netgear model uses , there’s quite a lot of files in , which vary very slightly between firmware versions, but which are basically only noise to us. [caption id="attachment_3986" align="aligncenter" width="1150"] [/caption] Again, we can filter out plaintext files, which have different hashes, but the same exact size, in , in our client-side script logic. With a little tweaking, we have a pretty short list of files, which may be candidates for further analysis. From what we know from the ZDI’s CVE-2021-27251 advisory, the affected component is “ready-genie-cloud", and the issue stems from “a fallback to a[n] insecure protocol to deliver updates”. There’s not an obvious “ready-genie-cloud" file in our list of changed files, but there is a . Interestingly, there wouldn’t be much use in diffing these files directly. In version 1.0.2.78, is a shell script, and in 1.0.2.80 it’s an ELF binary. We can’t do a precise diff, but we can do a quick manual overview to see if we’ve found our root cause. The old is a shell script. It contains several callable functions, however the one which looks the most promising is : This script seems very likely to be the root cause we’re looking for. A call is made to curl to download over an HTTPS connection. However, if the HTTPS download fails (the file isn’t created), then the fallback is to download over a normal FTP connection. Since it would be possible for a well-positioned attacker on the WAN-side to serve up fake DNS, force the HTTPS connection to fail, and host a malicious FTP server, this could probably be chained to force this script to download whatever files they wanted. All this correlates with the advisory description. In this case the “fallback to [the] insecure protocol to deliver updates” is likely a reference to a fallback to FTP. Let’s look a little further into the script flow and try to get a little more certainty. The file that gets downloaded to is simple. Its contents look something like the following: Each line is whitespace-separated with three fields: a filename, an OpenSSL version, and a hash. We can see some of these values being extracted in the script: The is read from based on the version installed on the device. The file is downloaded from the same root that the file was (), then ’d. The MD5 value is then checked against the md5 value read from . An attacker in control of could therefore easily force an affected Netgear device to download an arbitrary update file, which would then be verified against an attacker-controlled MD5 hash. Once the file has been downloaded and its checksum checked, the final stages of the install are done. A working folder is created and the downloaded update package is un-tar.gz’d to this folder. Then, critically, all the files that are un-tar.gz’d are copied to the system root recursively. As we can see, the update package is designed for exactly this behavior. The update package mirrors a Linux filesystem so that, for example, files within the directory will be written to (or overwrite) other files in the directory on the R7800. This means any file in the update package will simply overwrite any file on the system! So, there’s quite a few ways an attacker might choose to practically exploit the device. It also likely means that the exploit would be very persistent, perhaps even surviving a factory reset. Since 1.0.2.80’s is an ELF binary rather than a script, we have to work a little harder to find the exact patch (not that hard, we just fire up Ghidra). The logic previously handled by the shell script function is now handled by a series of compiled functions. As a quick example, the function at is responsible for downloading the file, this time to a file called . Notably, the FTP fallback is gone. HTTPS is the only option now. Diffing can be a really valuable tool for identifying root causes of bugs in embedded systems. “Traditional” techniques sometimes need to be tweaked in order to apply them to entire filesystem images, update packages, or other kinds of firmware delivery mechanisms. Sometimes, you get a small curveball, like the type of file entirely changing, but serving the exact same purpose. It’s useful to go into these kinds of diffing exercises with an open mind, because device vendors can be quite unpredictable.