[PT008] Fuzzing Linux kernel with Syzkaller

 

Syzkaller is a very effective fuzzer for Linux kernel that has found a lot of bugs in recent years. You may have heard of names like Dirty Cow, Meltdown, Spectre, BlueBorne… but syzkaller have found thousands of such bugs over the past few years. Here is the link to the bugs that syzkaller reported

Figure 1. Wellknown kernel bugs [1]

Google also built a system called syzbot that automatically and continuously fuzz Linux kernel. Their bugs mainly thank to this system.

Figure 2. Syzbot dashboard [2]

The special thing I see is that all bugs are public, including bugs that have not been fixed. Serious bugs will be fixed first. I notice that real systems don’t get kernel updates as often as it should, especially PCs and servers. If a hacker wants to perform an LPE or RCE attack on a target, he can wait for syzbot to find a serious bug, and then write the exploit code for that target.

Then I set my own challenge: given a specific version of Linux kernel, find an LPE bug of it and write the exploit code. Checking some servers and PCs which I used, I see that their kernels are built at quite long time ago (it’s Dec 2020 now):

• CentOS: 3/2020
• 2 Ubuntu Desktop 18.04: 10/2020
• Ubuntu server Ubuntu 18.04.3: 7/2020
• Ubuntu server 1: 2/2020
• Ubuntu server 2: 6/2020
• Ubuntu server 3: 8/2019

So, the potentiality of successfully exploiting a target is quite high: Linux kernels out there are quite out-dated. Normally, if you want to find an N-day bug, you search on the Internet for available exploits, or CVEs and try to write your own exploit code. With Linux kernel, it is even more difficult because the number of public exploits and CVEs are very few. But now, when there is a syzbot that continously publishes new bugs, that difficulty seems to be solved, and my approach is as follows:

1. Find the Linux kernel version of the target.
2. Compile it with suitable configuration for fuzz.
3. Build a qemu image.
4. Crawl all syzbot’s found bugs and form input corpus for syzkaller.
5. Fuzz the target with syzkaller until finding exploitable bug.
6. Minimize crash log into a smallest program that crashed the target.
7. Write the exploit code.

My target is: kernel 4.15.0-60-generic on Ubuntu 18.04 (compiled in Aug 2019).

Figure 3. target kernel version

Following video is my result of months learning about fuzzing with syzkaller and exploiting Linux kernel (successfully found an LPE bug of the target and wrote exploit code for it):

In this article, I will present the first 6 steps.

1. Find kernel version of target

I learned two methods to download the kernel version corresponding to the target.

Method 1: Based on the kernel version map [3], we find the corresponding Mainline Kernel Version, and then download the source code from this link. The Mainline Version of 4.15.0-60 is 4.15.18.

Figure 4. kernel version map [3]

However, the mapping between kernel version and mainline version is not 1-1. Many kernel versions have the same mainline version. If using version 4.15.18 to fuzz, there are many bugs are patched. This method is not as good as the following.

Method 2: many people sugguest downloading Linux source code with following command: apt-get install linux-source

However, to run this command, you have to install the same OS version as the target. In addition, this command installs the latest source code that is compatible with the OS, so many bugs are patched. So, this command is not suitable for our purpose. By capturing traffic while running the above command, I find out that the package is fetched from this link.

Figure 5. traffic captured while installing linux source

So, we can get the source code directly from this link, and it will be completely the same as the kernel of the target.

Figure 6. target linux version found [4]

Using ar, tar commands to extract the downloaded .deb, you will have the linux kernel source code.

2. Compile source code for fuzzing

To compile source code for fuzzing, read following guide. However, using default config, many drivers are built into loadable modules (CONFIG_XXX=m), not statically linked into vmlinux file. But according to the current support of syzkaller, it can fuzz only vmlinux. Therefore, to fuzz any module, you have to set it as a built-in module (CONFIG_XXX=y). It takes a lot of time to figure out which modules you should fuzz, so using a config file of syzbot should be the best choice. I use one of syzbot’s .config file [5].

Run command “make olddefconfig”.

Note: check if the following options are enabled:

# Coverage collection. CONFIG_KCOV=y   # Debug info for symbolization. CONFIG_DEBUG_INFO=y   # Memory bug detector CONFIG_KASAN=y CONFIG_KASAN_INLINE=y   # Required for Debian Stretch CONFIG_CONFIGFS_FS=y CONFIG_SECURITYFS=y

Then run: make bzImage

When compiling, I encounter some errors. I have to remove following options to successfully build the code:

• CONFIG_MODVERSIONS=y
• CONFIG_BATMAN_ADV=y

3. Build qemu image

Guideon building qemu image. Basically, it uses debootstrap to install a Debian system into a folder, and create an image, copy all files into that image. Run following command to build the default image: ./create_image.sh

Note:

• Root password is empty.
• As default, it is a 2G Stretch image.
• Network interface is eth0. When start it with qemu, maybe the iface name is not the same. We can fix it after booting it on.
• There are ssh keys: ssh stretch.id_rsa and stretch.id_rsa.pub. The public key is added into the config of the image, so we can use the private key to ssh onto the machine.

Run the virtual machine with the created image and kernel:

qemu-system-x86_64 \         -m 2G \         -smp 2 \         -kernel $KERNEL/arch/x86/boot/bzImage \         -append "console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \         -drive file=$IMAGE/stretch.img,format=raw \         -net user,host=10.0.2.10,hostfwd=tcp:127.0.0.1:10021-:22 \         -net nic,model=e1000 \         -enable-kvm \         -nographic

Figure 7. target fuzzing machine

Note:

• Not all hosts support the option enable-kvm that speeds up fuzzing.
• Virtualbox host only support -smp 1 while enable-kvm=true.
• By forwarding port 22 -> 10021, you can ssh or scp with the configured ssh key.

4. Make corpus

Input of syzkaller is a corpus.db file, including syz programs. Each syz program is basically a ordered list of syscalls. Its format is defined by syzkaller. For example:

# https://syzkaller.appspot.com/bug?id=a839269752ca27e5a69938064ba906ac8cf3fb36 # See https://goo.gl/kgGztJ for information about syzkaller reproducers. #{"procs":1,"sandbox":"","fault_call":-1,"close_fds":false} r0 = syz_usb_connect$hid(0x0, 0x36, &(0x7f0000000080)=ANY=[@ANYBLOB="12010000000018105e04da070000000000010902240001000000000904000009030000000921000000012222000905810308"], 0x0) syz_usb_control_io$hid(r0, 0x0, 0x0) syz_usb_control_io$hid(r0, &(0x7f00000001c0)={0x24, 0x0, 0x0, &(0x7f00000000c0)=ANY=[@ANYBLOB="00222200000096031306e53f070c0000072a9000a7c900be0017cf6643a30b09007a1583"], 0x0}, 0x0) syz_usb_ep_write(r0, 0x0, 0x1, &(0x7f0000000000)='B')

Lets look at a bug on syzbot [6], you will see following information:

• Commit: Linux kernel version.
• Syzkaller: syzkaller version.
• Config: the .config file used for building Linux source.
• Syz repro: syz program that causes the crash. This is what we have to crawl.
• C repro: C program that causes the crash.

Figure 8. a bug found by syzbot [6]

The fields syz repro and C repro can be empty, because not all crashes can be reproduced. This is my scripts used for crawl all those syz programs.

• Script parses all bug pages, gets all available syz programs.
• Script keeps a cache of all fetched pages.

After fetching all syz programs, you use syz-db to pack them into a corpus file:

Figure 9. pack corpus with syz-db

5. Fuzz target with syzkaller

Follow the guide. Note:

• Put the file corpus.db (do not modify its name) into workdir. Syzkaller will recognize it as input.
• Set reproduce = false in syzkaller config file. If not, syzkaller will use crash log (list of programs executed), to perform reproducing step, to find the program that causes the crash. This step is very time-consuming. We should ignore it.
• Set sandbox = setuid, to fuzz as user nobody but not root. Crashes caused by root have little/no meaning.
• Syzkaller runs qemu with snapshot option, so the image is not affected by fuzzing/crashing.

My config file is as follows:

{       "target": "linux/amd64",       "http": "127.0.0.1:56741",       "reproduce": false,       "sandbox": "setuid",       "workdir": "/root/linux-data/qemu/blog/workdir",       "kernel_obj": "/root/kernels/linux-source-4.15.0-60",       "image": "/root/linux-data/qemu/blog/stretch.img",       "sshkey": "/root/linux-data/qemu/blog/stretch.id_rsa",       "syzkaller": "/root/gopath/src/github.com/google/syzkaller",       "procs": 4,       "type": "qemu",       "vm": {             "count": 1,             "kernel": "/root/kernels/linux-source-4.15.0-60/arch/x86/boot/bzImage",             "cpu": 1,             "mem": 2048       } }

After run syz-manager, go to http://127.0.0.1:56741 you will see fuzz dashboard:

Figure 10. Fuzzing dashboard

After about half a day, I found many bugs:

Figure 11. Fuzzing result

There I see that the bug KASAN: use-after-free Read in bcsp_close should be exploitable. Look at the report, you see a skb is double-freed: it is allocated and freed in bcsp_recv, and freed again in bcsp_close. A double-free bug is a typical exploitable LPE.

BUG: KASAN: use-after-free in kfree_skb+0x2b6/0x340 net/core/skbuff.c:659 Read of size 4 at addr ffff88802b4fa2a4 by task syz-executor.0/8945   CPU: 0 PID: 8945 Comm: syz-executor.0 Not tainted 4.15.18 #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1ubuntu1 04/01/2014 Call Trace:  …  skb_unref include/linux/skbuff.h:960 [inline]  kfree_skb+0x2b6/0x340 net/core/skbuff.c:659  bcsp_close+0xce/0x160 drivers/bluetooth/hci_bcsp.c:763  hci_uart_tty_close+0x1af/0x240 drivers/bluetooth/hci_ldisc.c:551  tty_ldisc_close.isra.2+0x91/0xd0 drivers/tty/tty_ldisc.c:506  tty_ldisc_kill+0x46/0xc0 drivers/tty/tty_ldisc.c:652  tty_ldisc_release+0xfa/0x210 drivers/tty/tty_ldisc.c:819  tty_release_struct+0x14/0x50 drivers/tty/tty_io.c:1611  … RIP: 0033:0x417e77 RSP: 002b:00007ffca8899c00 EFLAGS: 00000293 ORIG_RAX: 0000000000000003 RAX: 0000000000000000 RBX: 0000000000000003 RCX: 0000000000417e77 RDX: 0000000000000000 RSI: 00000000a31766cd RDI: 0000000000000003 RBP: 0000000000000001 R08: 00000000a31766d1 R09: 0000000000000000 R10: 00007ffca8899d40 R11: 0000000000000293 R12: 000000000076c980 R13: 000000000076bf00 R14: 000000000076bf21 R15: 000000000000dc8d   Allocated by task 102:  kmem_cache_alloc_node+0x69/0x400 mm/slab.c:3640  __alloc_skb+0xa4/0x500 net/core/skbuff.c:193  alloc_skb include/linux/skbuff.h:988 [inline]  bt_skb_alloc include/net/bluetooth/bluetooth.h:339 [inline]  bcsp_recv+0x82c/0x13f0 drivers/bluetooth/hci_bcsp.c:685  hci_uart_tty_receive+0x1eb/0x4b0 drivers/bluetooth/hci_ldisc.c:616  tty_ldisc_receive_buf+0x12f/0x160 drivers/tty/tty_buffer.c:460  …   Freed by task 102:  …  __kfree_skb net/core/skbuff.c:646 [inline]  kfree_skb+0xb6/0x340 net/core/skbuff.c:663  bcsp_recv+0x531/0x13f0 drivers/bluetooth/hci_bcsp.c:623  hci_uart_tty_receive+0x1eb/0x4b0 drivers/bluetooth/hci_ldisc.c:616  tty_ldisc_receive_buf+0x12f/0x160 drivers/tty/tty_buffer.c:460  …

Anaylizing the source code, you see the skb is allocated at drivers/bluetooth/hci_bcsp.c:685 of function bcsp_close:

Figure 12. skb allocation

Skb is freed at drivers/bluetooth/hci_bcsp.c:623 bcsp_recv:

Figure 13. free skb

After that, it is freed again at drivers/bluetooth/hci_bcsp.c:763 bcsp_close:

Figure 14. double-free skb

In bcsp_close, bcsp->rx_skb is checked againtst NULL before freeing. However, before that, on line 623, bcsp->rx_skb is freed without being set into NULL. This is the root cause of the bug.

Checking Linux source code on github [7], I discovered the vulnerability was caused by a commit on 7 Jul, 2019 which patches a memory leak in rx_skb [8]:

Figure 15. commit fixing memory leak but causing double free [8]

On 4 Nov, 2019 the bug was fixed by nullifying bcsp->rx_skb every time it is freed [9]:

Figure 16. fixing double free in bcsp_close [9]

6. Reproduce crash

View the guide here. Reproduce is the step of find the minimized program that causes the crash. Following is an example of crash log:

Figure 17. log crash

During fuzzing process, the programs are executed multithreadedly (depends on the option procs in config file), so the program that caused the crash does not necessarily immediately precedes it. I use the following method:

1. Run the image in qemu.
2. scp the files syz-execprog and syz-executor onto the virtual machine.
3. Use syz-execprog to run crash log on the VM. Gradually remove unrelated programs from the log, until there is only one program left. You can remove syscalls in the program to make it smaller.
   ./syz-execprog -executor=./syz-executor -repeat=1 -sandbox=setuid -enable=none -collide=false ./log0

Program I found:

03:47:02 executing program 0: r0 = openat$ptmx(0xffffffffffffff9c, &(0x7f0000000280)='/dev/ptmx\x00', 0x0, 0x0) ioctl$TIOCSETD(r0, 0x5423, &(0x7f0000000040)=0xf) ioctl$KDADDIO(r0, 0x400455c8, 0x1) r1 = socket$inet6(0xa, 0x400000000001, 0x0) bind$inet6(r1, &(0x7f0000000600)={0xa, 0x4e20, 0x0, @loopback}, 0x1c) sendto$inet6(r1, 0x0, 0x0, 0x20000008, &(0x7f00008d4fe4)={0xa, 0x4e20, 0x0, @loopback}, 0x1c) r2 = open(&(0x7f0000000240)='./bus\x00', 0x100000141042, 0x0) ftruncate(r2, 0x10099b7) sendfile(r1, r2, 0x0, 0x8000fffffffe)

Use syz-prog2c to convert that program into C code: syz-prog2c -prog crash.syz -repeat=1 -enable=none > crash.c

To check if the above C code works, I create a test user, login and run the C program, successfully crash the sytem:

Conclusion

I successfully solve the problem with kernel 4.15.0-60-generic. Without syzbot, with me it is almost impossible (I tried some public exploits but failed). However, more targets should be tested before we can confirm whether this method can be used in practice. If it can, I think the security of Linux kernel needs more attention, such as being more regularly updated to latest version.

References

[1] https://events19.linuxfoundation.org/wp-content/uploads/2017/11/Syzbot-and-the-Tale-of-Thousand-Kernel-Bugs-Dmitry-Vyukov-Google.pdf.

[2] https://syzkaller.appspot.com/upstream.

[3] https://people.canonical.com/~kernel/info/kernel-version-map.html.

[4] http://us.archive.ubuntu.com/ubuntu/pool/main/l/linux/.

[5] https://syzkaller.appspot.com/text?tag=KernelConfig&x=fac7c3360835a4e0.

[6] https://syzkaller.appspot.com/bug?id=a839269752ca27e5a69938064ba906ac8cf3fb36.

[7] https://github.com/torvalds/linux.

[8] https://github.com/torvalds/linux/commit/4ce9146e0370fcd573f0372d9b4e5a211112567c.

[9] https://github.com/torvalds/linux/commit/cf94da6f502d8caecabd56b194541c873c8a7a3c.

Click here for Vietnamese version

ManhND

Service Center, VinCSS (a member of Vingroup)