Practical Modbus Fuzzing with boofuzz

So what we're going to do today is try our hand at fuzzing the Modbus protocol. We're going to go through a minimal example: we'll fuzz just one Modbus function, with the minimum number of fields that are likely to provide useful results. A full Modbus fuzzing suite is beyond the scope of this post, but we'll go through a few pointers about how to get there before we call it a day.

What we'll use: boofuzz, Sulley's successor, is a very neat fuzzing framework that's usually my first option when it comes to automating protocol fuzzing. We'll also use PyModbus to write our own trivial, crash-prone Modbus server. Both can be installed via pip.

A Crash-prone Modbus Server

Normally, you would fuzz a real-life device, but a basic introduction that begins with "go out and buy this particular PLC running this particular firmware version" wouldn't be very useful. So instead, I'll start it with emulating a device with a sloppy firmware.

If you haven't used PyModbus before, don't worry, it's pretty straightforward. In order to build a Modbus server, you instantiate a ModbusSlaveContext, passing it a number of "data blocks" which correspond to its input registers, coils and so on. Data blocks are instances of a storage mechanisms that support, at a minimum, reading and writing values at a certain address. The simplest one, ModbusSequentialDataBlock, is just a thin wrapper over a collection of serializable bytes -- essentially little more than a C array. You use the data blocks to create a data store (essentially just a collection of several data blocks), and you build a server context out of that -- a server context being a bunch of device-specific state, plus the data that it can retrieve. Add a TCP server on top of it, along with the decoding logic required to understand Modbus, and bam -- you got your own Modbus server.

(Note that Modbus' terminology is slightly reversed: a Modbus "master" hands out requests and gets back answers, which makes it a client, whereas "slave" devices get requests and hand out answers, which makes them servers).

In order to emulate a sloppily-written device, we're going to create our own data block, except ours will be really awful: when it's asked for a value at an address higher than 0xFF, it's going to crash.

This is barely 50 lines without the comments:

#!/usr/bin/env python3

from pymodbus.datastore import ModbusSequentialDataBlock
from pymodbus.server.sync import StartTcpServer
from pymodbus.datastore import ModbusSlaveContext, ModbusServerContext

import logging
import os

##
# A sloppy data block: someone forgot to check some address bounds
# somewhere, and reading input registers past 0xFF is going to cause
# this device to crash!
class BadDataBlock(ModbusSequentialDataBlock):
    def __init__(self):
        self.values = [0x00] * 0xFFFF
        super().__init__(0, self.values)

    def getValues(self, addr, count):
        # Uh-oh...
        if (addr <= 0xFF):
            return self.values[addr:addr+count]
        else:
            os._exit(1)

def run_server():

    bad_block = BadDataBlock()

    store = ModbusSlaveContext(
        di=ModbusSequentialDataBlock(0, [0xFF] * 32),
        co=ModbusSequentialDataBlock(0, [0xFF] * 32),
        hr=ModbusSequentialDataBlock(0, [0xFF] * 32),
        ir=bad_block)

    context = ModbusServerContext(slaves=store, single=True)

    StartTcpServer(context, address=("localhost", 5020))

def main():
    FORMAT = ('%(asctime)-15s %(threadName)-15s'
                    ' %(levelname)-8s %(module)-15s:%(lineno)-8s %(message)s')
    logging.basicConfig(format=FORMAT)
    log = logging.getLogger()
    log.setLevel(logging.DEBUG)
    run_server()

if __name__ == "__main__":
    main()

If you point your favourite Modbus tool at this program (PyModbus itself has an excellent REPL, but if you have a Windows machine around, Modbus Test Utility is pretty good, too!), you'll see that you'll be able to poke around coils, discrete inputs and holding registers just fine. They're all mapped at addresses between 0x00 and 0x20 (i.e. 32). If you read above that address, you'll get an error. But -- told you it was sloppy! -- you'll be able to read input register values above 0x20, too. Above 0xFF, though, the program will "crash".

Boofuzz 101

Boofuzz is very straightforward to use. In order to get the hang of it, I encourage you to read the official docs, which include a very short quickstart guide https://boofuzz.readthedocs.io/en/stable/user/quickstart.html.

The basic mechanism is beautifully simple. The basic unit of boofuzz' fuzzing process is a session. A session is essentially a "scripted dialogue" -- deep down, a sequence of send() and recv(), if you will -- where you have control over what you send(), and about when you do recv().

Each session is defined by a connection and number of fields, which can be organized into blocks. Generally, each boofuzz field maps to a protocol field. Depending on protocol, some fields are going to be "binary" fields (e.g. s_bytes -- which is one way to describe, for example, the source and destination fields of an IP packet), but others can be outright ASCII strings, if you're fuzzing HTTP, for example.

Each field is defined by a starting value and a number of parameters that describe how it's going to be fuzzed (if it's going to be fuzzed at all). For example, the transaction ID field would look like this:

s_bytes(value=bytes([0x00, 0x01]), size=2, max_len=2, name="transaction_id")

if a field should be fixed, you can pass fuzzable=False here. As we'll see in a minute, it doesn't really make sense to fuzz all the fields of a packet, or at least not in every scenario.

It's possible to gather several fields in a block -- which is basically just a collection of fields which, for some reason, make more sense together. You don't have to, but it will help you organize your code better and it will also make it easier to explore the results that boofuzz yields.

A fuzzer built with boofuzz, then, works roughly like this:

  • You instantiate a session object, pointing it at the device (or, in our case, the process) that you want to fuzz
  • You build a protocol definition for it, by defining each field of the protocol and telling boofuzz how you want to vary that field's value. Optionally, you can gather fields into blocks.
  • You connect to the fuzzing's victim using the session object above, call fuzz(), and wait for things to happen

In our case, it'll take just a few seconds for something useful to happen. That's because our bug is obvious and monumental and we'll only be fuzzing a few fields of the protocol -- partly because, well, we cheated, so we know where to look. In real life, with real devices, this can take hours, sometimes days or even weeks.

The Simples Modbus Fuzzer in Existence

A Basic Modbus Protocol Map

Well then! Let's build a minimal protocol definition using boofuzz!

We're going to build a protocol definition that includes just one packet type -- Read Input Registers. The structure of this packet is trivial -- in addition to the typical Modbus header and the function ID, it includes just two fields: the starting address, and the number of input registers to read, starting from that input address, both of them two-byte. The whole packet looks like this:

Modbus packet structure

Modbus Read Input Registers (function code 04) packet structure

Before we move on to writing the description, let's see which fields we want to fuzz.

The obvious candidates are the Starting Address and Number of Registers fields. Clearly, if there's any flaw in the logic that handles this request, it's going to be tied to these two values, so we definitely need to fuzz those.

We're only going to fuzz the Read Input Registers function today, so we're going to keep the Function ID field fixed to 0x04. In general, I prefer to fuzz each Modbus function separately, because it makes it easier to explore fuzzing results. However, if you're fuzzing a "black box" device, it's a good idea to add a fuzzing session for the function ID as well: some vendors implement vendor-specific commands with function codes outside Modbus' standard, and fuzzing will, at the very least, help you discover them more easily.

If we were to fuzz a real-life device, rather than work out a minimal fuzzing example, it would also be a good idea to fuzz the Length and Unit ID fields. Crafting packets where a length field in the data (e.g. the number of registers to write in a Write Multiple Registers request) doesn't match the length of the header is a gateway to a bunch of basic exploits. And, for historical reasons that I won't bore you with this time, some devices -- I've seen two so far which -- don't check the Unit ID field and attempt to handle requests that weren't meant for them.

However, in this post, we're going to keep these two fields fixed to speed up our fuzzing session, and in order to make it easier to explore the results that we get.

The protocol ID field has to remain fixed if this is to be a meaningful exercise -- it's supposed to be 0x0000 for Modbus/TCP. As for the Transaction ID field, whether you fuzz it or not is up to you. Mishandling this field can have security implications, but it's pretty difficult to find that via fuzzing. On the other hand, having a non-constant transaction ID is closer to the real-life behaviour of the protocol.

To recap:

  • We're going to fuzz the Starting Address, Number of Registers, and Transaction ID fields
  • We're going to keep the Protocol ID, Length, Unit ID, and Function ID fields fixed.

This leads us to the following, very simple protocol definition:

    ##
    # Modbus TCP header:
    # * 2-byte transaction id
    # * 2-byte protocol id, must be 0000
    # * 2-byte message length
    # * 1 byte unit ID
    #
    # followed by request data:
    # * 1 byte function ID
    # * 2-byte starting address
    # * 2-byte number of registers
    s_initialize(name="Read Input Registers")

    if s_block_start("header"):
        # Transaction ID
        s_bytes(value=bytes([0x00, 0x01]), size=2, max_len=2, name="transaction_id")
        # Protocol ID. Fuzzing this usually doesn't provide too much value so let's leave it fixed.
        # We can also use s_static here.
        s_bytes(value=bytes([0x00, 0x00]), size=2, max_len=2, name="proto_id", fuzzable=False)
        # Length. Fuzzing this is generally useful but we'll keep it fixed to make this tutorial's
        # data set easier to explore.
        s_bytes(value=bytes([0x00, 0x06]), size=2, max_len=2, name="len", fuzzable=False)
        # Unit ID. Once again, fuzzing this usually doesn't provide too much value.
        s_bytes(value=bytes([0x01]), size=1, max_len=1, name="unit_id", fuzzable=False)
    s_block_end()

    if s_block_start("data"):
        # Function ID. Fixed to Read Input Registers (04)
        s_bytes(value=bytes([0x04]), size=1, max_len=1, name="func_id", fuzzable=False)
        # Address of the first register to read
        s_bytes(value=bytes([0x00, 0x00]), size=2, max_len=2, name="start_addr")
        # Number of registers to read
        s_bytes(value=bytes([0x00, 0x01]), size=2, max_len=2, name="reg_num")
    s_block_end()

Defining our fuzzing session

There's just one bit left to do, and that's to plug all this into a fuzzing session. In order to make what happens next comprehensible, though, we'll have to take a quick detour to explain the matter of monitoring.

Fuzzers aren't exactly magical CVE machines. Fuzzing a program with a hidden privilege escalation bug won't get you root -- it will just crash the program. It's up to you to figure out which of these crashes are just annoying, and which ones can be exploited. Furthermore, most programs can be crashed in a depressing number of ways. Ours is no exception: there are 65280 different addresses for which it will crash.

This raises two questions:

  • How do you know when a program has crashed?
  • How do you restart it?

This is what a Monitor is for. In short, a Monitor object is what you use to figure out when your device crashed (as opposed to just being slow to respond -- sometimes it happens), and to restart the target and bring it, and its environment, to the desired initial state, so that you can continue fuzzing it from where you left off when it crashed.

I'm not going to cover that here because -- and this is somewhat counter-intuitive -- monitoring is a remarkably complex problem, especially when you're fuzzing a remote device, as opposed to a local process. Boofuzz has a simple example that uses a process monitor but, in general, you'll need to write your own monitor when fuzzing against external devices.

For illustrative purposes, we can do without a monitor, though. Instead, we'll tell boofuzz to stop after one unsuccessful reconnection attempt, by passing a restart_threshold of 1.

This is what our complete fuzzing program will look like, then:

#!/usr/bin/env python3

from boofuzz import *

def main():
    session = Session(target=Target(connection=TCPSocketConnection("127.0.0.1", 5020)),
                      restart_threshold=1, restart_timeout=1.0)

    ##
    # Modbus TCP header:
    # * 2-byte transaction id
    # * 2-byte protocol id, must be 0000
    # * 2-byte message length
    # * 1 byte unit ID
    #
    # followed by request data:
    # * 1 byte function ID
    # * 2-byte starting address
    # * 2-byte number of registers
    s_initialize(name="Read Input Registers")

    if s_block_start("header"):
        # Transaction ID
        s_bytes(value=bytes([0x00, 0x01]), size=2, max_len=2, name="transaction_id")
        # Protocol ID. Fuzzing this usually doesn't provide too much value so let's leave it fixed.
        # We can also use s_static here.
        s_bytes(value=bytes([0x00, 0x00]), size=2, max_len=2, name="proto_id", fuzzable=False)
        # Length. Fuzzing this is generally useful but we'll keep it fixed to make this tutorial's
        # data set easier to explore.
        s_bytes(value=bytes([0x00, 0x06]), size=2, max_len=2, name="len", fuzzable=False)
        # Unit ID. Once again, fuzzing this usually doesn't provide too much value.
        s_bytes(value=bytes([0x01]), size=1, max_len=1, name="unit_id", fuzzable=False)
    s_block_end()

    if s_block_start("data"):
        # Function ID. Fixed to Read Input Registers (04)
        s_bytes(value=bytes([0x04]), size=1, max_len=1, name="func_id", fuzzable=False)
        # Address of the first register to read
        s_bytes(value=bytes([0x00, 0x00]), size=2, max_len=2, name="start_addr")
        # Number of registers to read
        s_bytes(value=bytes([0x00, 0x01]), size=2, max_len=2, name="reg_num")
    s_block_end()

    session.connect(s_get("Read Input Registers"))

    session.fuzz()

if __name__ == "__main__":
    main()

Firing it up

This is all very simple at this point. First, fire up the PyModbus program we wrote earlier. Then run the fuzzer we just wrote.

At this point, you'll be treated to a very Matrix-esque sequence of scrolling junk, that will generally look like this:

$ ./modfuzz.py                                                                                                                            [266/274][2020-07-11 14:26:12,010]     Info: Web interface can be found at http://localhost:26000
[2020-07-11 14:26:12,011] Test Case: 1: Read Input Registers.transaction_id.1
[2020-07-11 14:26:12,011]     Info: Type: Bytes. Default value: b'\x00\x01'. Case 1 of 195 overall.
[2020-07-11 14:26:12,011]     Info: Opening target connection (127.0.0.1:5020)...
[2020-07-11 14:26:12,011]     Info: Connection opened.
[2020-07-11 14:26:12,011]   Test Step: Monitor CallbackMonitor#140605128657936[pre=[],post=[],restart=[],post_start_target=[]].pre_send()
[2020-07-11 14:26:12,011]   Test Step: Fuzzing Node 'Read Input Registers'
[2020-07-11 14:26:12,011]     Info: Sending 12 bytes...
[2020-07-11 14:26:12,011]     Transmitted 12 bytes: a5 00 00 00 00 06 01 04 00 00 00 01 b'\xa5\x00\x00\x00\x00\x06\x01\x04\x00\x00\x00\x01'
[2020-07-11 14:26:12,012]   Test Step: Contact target monitors
[2020-07-11 14:26:12,012]   Test Step: Cleaning up connections from callbacks
[2020-07-11 14:26:12,012]       Check OK: No crash detected.
[2020-07-11 14:26:12,012]     Info: Closing target connection...
[2020-07-11 14:26:12,012]     Info: Connection closed.
[2020-07-11 14:26:12,027] Test Case: 2: Read Input Registers.transaction_id.2
[2020-07-11 14:26:12,027]     Info: Type: Bytes. Default value: b'\x00\x01'. Case 2 of 195 overall.
[2020-07-11 14:26:12,027]     Info: Opening target connection (127.0.0.1:5020)...                                                                                                                                                                                                       [2020-07-11 14:26:12,027]     Info: Connection opened.
[2020-07-11 14:26:12,027]   Test Step: Monitor CallbackMonitor#140605128657936[pre=[],post=[],restart=[],post_start_target=[]].pre_send()
[2020-07-11 14:26:12,027]   Test Step: Fuzzing Node 'Read Input Registers'
[2020-07-11 14:26:12,027]     Info: Sending 12 bytes...
[2020-07-11 14:26:12,027]     Transmitted 12 bytes: 00 01 00 00 00 06 01 04 00 00 00 01 b'\x00\x01\x00\x00\x00\x06\x01\x04\x00\x00\x00\x01'
[2020-07-11 14:26:12,027]   Test Step: Contact target monitors
[2020-07-11 14:26:12,027]   Test Step: Cleaning up connections from callbacks
[2020-07-11 14:26:12,027]       Check OK: No crash detected.
[2020-07-11 14:26:12,027]     Info: Closing target connection...
[2020-07-11 14:26:12,028]     Info: Connection closed.

And, sooner or later, boofuzz will run into one of the lucky combinations and crash our silly program.

Now, because we've told it to give up after one failed reconnection attempt, our program has exited. But it has deposited a results file in the current directory, under boofuz-results/. That's a simple SQLite database which you can actually browse with an SQLite browser, but Boofuzz has a nice web UI that you can use to explore what happened. To use it, just run:

$ boo open ./boofuzz-results/<result file>

and go to localhost:26000.

The last case that you see is the one which boofuzz aborted, because it couldn't connect to the target anymore. The previous one is the one that resulted in the crash.

In my case, that was the 22nd attempt (this is not representative of what happens IRL). It looks like this:

And, sure enough, if you look at what got transmitted, it's clear what happened:

Transmitted 12 bytes: 00 01 00 00 00 06 01 04 a5 00 00 01 b'\x00\x01\x00\x00\x00\x06\x01\x04\xa5\x00\x00\x01'

This tried to read (04) 1 register (00 01) from address 0xA500 (a5 00), which ended up triggering our bug.

Where to take it from here

Congratulations, you just got your first Modbus fuzzer to crash something.

This is where the hard part begins, of course. Our minimal fuzzer isn't bad but it's still:

  • Fuzzing only one function
  • Giving up at the first crash, which it detects based on nothing but how long it takes to get an answer back (if it's more than 5 seconds, that's a crash).

This isn't going to be enough for real-life devices. The journey to "enough" is way longer than there's room for in a blog post, but I can sketch out some of the things that you'll want to do to get there.

  1. The biggest hurdle is simply monitoring and restarting your device. More often than not, that will involve some off-the-shelf hardware to allow you to power-cycle your device. Many devices that speak Modbus today (especially clients) are running a general-purpose OS, or are at least running code that you can compile and run on a Linux machine, and fuzz locally. I warmly recommend you resist the urge to do so. It makes little sense to fuzz another program than the one you're shipping. If there's no other way, it's better to just augment the code with features that make it easier to monitor -- for example, you can get your device to send keep-alive packets at regular intervals. Power-cycling a device -- if it gets to it, even by using a serial-controlled PDU like this one -- is a fifty-line script that takes half an hour to cobble together.

  2. Extending the protocol definition is where most of the analysis work lays. The good news is that this is generally pretty reusable. Depending on your requirements, it can make sense to maintain several of these, with various fuzzing ranges, for example.

  3. While you'll want to regularly fuzz your firmware and that can take very long, it can make sense to run a subset of your fuzzing tests on nightly builds, for example. If you do, I encourage you to have a look at boofuzz's results database: it uses a very straightforward schema that's very easy to get reports out of.

  4. After a certain level of complexity, it's usually a good idea to get packet dumps of the entire traffic as well. boofuzz does give you packet dumps but they're a little tedious to work with. On the other hand, packets in a PCAP file can be easily replayed, so that you can then reproduce a crash in isolation.

At the end of the day, though, it's useful to remember that fuzzing isn't magical security dust. You can't throw it at a device and wait for the security problems to reveal themselves. Most of the work is not in the fuzzing itself, but in triaging its results and in figuring out which ones are exploitable and how.

Happy fuzzing!