Syzkaller is a very effective fuzzer for Linux kernel that has found a lot of bugs in recent years. You may have heard of names like Dirty Cow, Meltdown, Spectre, BlueBorne… but syzkaller have found thousands of such bugs over the past few years. Here is the link to the bugs that syzkaller reported
![]() |
Figure
1. Wellknown kernel bugs |
Google also built a system called syzbot that automatically and continuously fuzz Linux kernel. Their bugs mainly thank to this system.
![]() |
Figure
2. Syzbot dashboard |
The special thing I see is that all bugs are public, including bugs that have not been fixed. Serious bugs will be fixed first. I notice that real systems don’t get kernel updates as often as it should, especially PCs and servers. If a hacker wants to perform an LPE or RCE attack on a target, he can wait for syzbot to find a serious bug, and then write the exploit code for that target.
Then I set my own challenge: given a specific version of Linux kernel, find an LPE bug of it and write the exploit code. Checking some servers and PCs which I used, I see that their kernels are built at quite long time ago (it’s Dec 2020 now):
- CentOS: 3/2020
- 2 Ubuntu Desktop 18.04: 10/2020
- Ubuntu server Ubuntu 18.04.3: 7/2020
- Ubuntu server 1: 2/2020
- Ubuntu server 2: 6/2020
- Ubuntu server 3: 8/2019
So, the potentiality of successfully exploiting a target is quite high: Linux kernels out there are quite out-dated. Normally, if you want to find an N-day bug, you search on the Internet for available exploits, or CVEs and try to write your own exploit code. With Linux kernel, it is even more difficult because the number of public exploits and CVEs are very few. But now, when there is a syzbot that continously publishes new bugs, that difficulty seems to be solved, and my approach is as follows:
- Find the Linux kernel version of the target.
- Compile it with suitable configuration for fuzz.
- Build a qemu image.
- Crawl all syzbot’s found bugs and form input corpus for syzkaller.
- Fuzz the target with syzkaller until finding exploitable bug.
- Minimize crash log into a smallest program that crashed the target.
- Write the exploit code.
My target is: kernel 4.15.0-60-generic on Ubuntu 18.04 (compiled in Aug 2019).
![]() |
Figure 3. target kernel version |
Following video is my result of months learning about fuzzing with syzkaller and exploiting Linux kernel (successfully found an LPE bug of the target and wrote exploit code for it):
In this article, I will present the first 6 steps.
1. Find kernel version of target
I learned two methods to download the kernel version corresponding to the target.
Method
1: Based on the kernel
version map
![]() |
Figure
4. kernel version map |
However, the mapping between kernel version and mainline version is not 1-1. Many kernel versions have the same mainline version. If using version 4.15.18 to fuzz, there are many bugs are patched. This method is not as good as the following.
Method 2: many people sugguest downloading Linux source code with following command: apt-get install linux-source
However, to run this command, you have to install the same OS version as the target. In addition, this command installs the latest source code that is compatible with the OS, so many bugs are patched. So, this command is not suitable for our purpose. By capturing traffic while running the above command, I find out that the package is fetched from this link.
![]() |
Figure 5. traffic captured while installing linux source |
So, we can get the source code directly from this link, and it will be completely the same as the kernel of the target.
![]() |
Figure
6. target linux version found |
Using ar, tar commands to extract the downloaded .deb, you will have the linux kernel source code.
2. Compile source code for fuzzing
To
compile source code for fuzzing, read following guide. However,
using default config, many drivers are built into loadable modules (CONFIG_XXX=m), not statically linked into vmlinux file. But
according to the current support of syzkaller, it can fuzz only vmlinux. Therefore,
to fuzz any module, you have to set it as a built-in module (CONFIG_XXX=y). It takes a lot of time to figure out which
modules you should fuzz, so using a config file of syzbot should be the best
choice. I use one of syzbot’s .config file
Run command “make olddefconfig”.
Note: check if the following options are enabled:
# Coverage collection. CONFIG_KCOV=y
# Debug info for
symbolization. CONFIG_DEBUG_INFO=y
# Memory bug detector CONFIG_KASAN=y CONFIG_KASAN_INLINE=y
# Required for Debian Stretch CONFIG_CONFIGFS_FS=y CONFIG_SECURITYFS=y |
Then run: make bzImage
When compiling, I encounter some errors. I have to remove following options to successfully build the code:
- CONFIG_MODVERSIONS=y
- CONFIG_BATMAN_ADV=y
3. Build qemu image
Guideon building qemu image. Basically, it uses debootstrap to install a Debian system into a folder, and create an image, copy all files into that image. Run following command to build the default image: ./create_image.sh
Note:
- Root password is empty.
- As default, it is a 2G Stretch image.
- Network interface is eth0. When start it with qemu, maybe the iface name is not the same. We can fix it after booting it on.
- There are ssh keys: ssh stretch.id_rsa and stretch.id_rsa.pub. The public key is added into the config of the image, so we can use the private key to ssh onto the machine.
Run the virtual machine with the created image and kernel:
qemu-system-x86_64 \ -m 2G \ -smp 2 \ -kernel $KERNEL/arch/x86/boot/bzImage \ -append "console=ttyS0 root=/dev/sda
earlyprintk=serial net.ifnames=0" \ -drive file=$IMAGE/stretch.img,format=raw \ -net user,host=10.0.2.10,hostfwd=tcp:127.0.0.1:10021-:22 \ -net nic,model=e1000 \ -enable-kvm \ -nographic |
![]() |
Figure 7. target fuzzing machine |
Note:
- Not all hosts support the option enable-kvm that speeds up fuzzing.
- Virtualbox host only support -smp 1 while enable-kvm=true.
- By forwarding port 22 -> 10021, you can ssh or scp with the configured ssh key.
4. Make corpus
Input of syzkaller is a corpus.db file, including syz programs. Each syz program is basically a ordered list of syscalls. Its format is defined by syzkaller. For example:
#
https://syzkaller.appspot.com/bug?id=a839269752ca27e5a69938064ba906ac8cf3fb36 # See https://goo.gl/kgGztJ
for information about syzkaller reproducers. #{"procs":1,"sandbox":"","fault_call":-1,"close_fds":false} r0 = syz_usb_connect$hid(0x0,
0x36,
&(0x7f0000000080)=ANY=[@ANYBLOB="12010000000018105e04da070000000000010902240001000000000904000009030000000921000000012222000905810308"],
0x0) syz_usb_control_io$hid(r0,
0x0, 0x0) syz_usb_control_io$hid(r0,
&(0x7f00000001c0)={0x24, 0x0, 0x0,
&(0x7f00000000c0)=ANY=[@ANYBLOB="00222200000096031306e53f070c0000072a9000a7c900be0017cf6643a30b09007a1583"],
0x0}, 0x0) syz_usb_ep_write(r0, 0x0,
0x1, &(0x7f0000000000)='B') |
Lets
look at a bug on syzbot
- Commit: Linux kernel version.
- Syzkaller: syzkaller version.
- Config: the .config file used for building Linux source.
- Syz repro: syz program that causes the crash. This is what we have to crawl.
- C repro: C program that causes the crash.
![]() |
Figure
8. a bug found by syzbot |
The fields syz repro and C repro can be empty, because not all crashes can be reproduced. This is my scripts used for crawl all those syz programs.
- Script parses all bug pages, gets all available syz programs.
- Script keeps a cache of all fetched pages.
After fetching all syz programs, you use syz-db to pack them into a corpus file:
![]() |
Figure 9. pack corpus with syz-db |
5. Fuzz target with syzkaller
Follow the guide. Note:
- Put the file corpus.db (do not modify its name) into workdir. Syzkaller will recognize it as input.
- Set reproduce = false in syzkaller config file. If not, syzkaller will use crash log (list of programs executed), to perform reproducing step, to find the program that causes the crash. This step is very time-consuming. We should ignore it.
- Set sandbox = setuid, to fuzz as user nobody but not root. Crashes caused by root have little/no meaning.
- Syzkaller runs qemu with snapshot option, so the image is not affected by fuzzing/crashing.
My config file is as follows:
{ "target":
"linux/amd64", "http":
"127.0.0.1:56741", "reproduce":
false, "sandbox":
"setuid", "workdir":
"/root/linux-data/qemu/blog/workdir", "kernel_obj":
"/root/kernels/linux-source-4.15.0-60", "image":
"/root/linux-data/qemu/blog/stretch.img", "sshkey":
"/root/linux-data/qemu/blog/stretch.id_rsa", "syzkaller":
"/root/gopath/src/github.com/google/syzkaller", "procs":
4, "type":
"qemu", "vm":
{ "count":
1, "kernel":
"/root/kernels/linux-source-4.15.0-60/arch/x86/boot/bzImage", "cpu":
1, "mem":
2048 } } |
After run syz-manager, go to http://127.0.0.1:56741 you will see fuzz dashboard:
![]() |
Figure 10. Fuzzing dashboard |
After about half a day, I found many bugs:
![]() |
Figure 11. Fuzzing result |
There I see that the bug KASAN: use-after-free Read in bcsp_close should be exploitable. Look at the report, you see a skb is double-freed: it is allocated and freed in bcsp_recv, and freed again in bcsp_close. A double-free bug is a typical exploitable LPE.
BUG: KASAN: use-after-free in kfree_skb+0x2b6/0x340
net/core/skbuff.c:659 Read of size 4 at addr ffff88802b4fa2a4 by task
syz-executor.0/8945
CPU: 0 PID: 8945 Comm: syz-executor.0 Not tainted
4.15.18 #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS 1.10.2-1ubuntu1 04/01/2014 Call Trace: … skb_unref
include/linux/skbuff.h:960 [inline] kfree_skb+0x2b6/0x340 net/core/skbuff.c:659 bcsp_close+0xce/0x160
drivers/bluetooth/hci_bcsp.c:763 hci_uart_tty_close+0x1af/0x240
drivers/bluetooth/hci_ldisc.c:551 tty_ldisc_close.isra.2+0x91/0xd0
drivers/tty/tty_ldisc.c:506 tty_ldisc_kill+0x46/0xc0
drivers/tty/tty_ldisc.c:652 tty_ldisc_release+0xfa/0x210
drivers/tty/tty_ldisc.c:819 tty_release_struct+0x14/0x50
drivers/tty/tty_io.c:1611 … RIP: 0033:0x417e77 RSP: 002b:00007ffca8899c00 EFLAGS: 00000293 ORIG_RAX: 0000000000000003 RAX: 0000000000000000 RBX: 0000000000000003 RCX:
0000000000417e77 RDX: 0000000000000000 RSI: 00000000a31766cd RDI:
0000000000000003 RBP: 0000000000000001 R08: 00000000a31766d1 R09:
0000000000000000 R10: 00007ffca8899d40 R11: 0000000000000293 R12:
000000000076c980 R13: 000000000076bf00 R14: 000000000076bf21 R15:
000000000000dc8d
Allocated by task 102: kmem_cache_alloc_node+0x69/0x400
mm/slab.c:3640 __alloc_skb+0xa4/0x500 net/core/skbuff.c:193 alloc_skb include/linux/skbuff.h:988
[inline] bt_skb_alloc
include/net/bluetooth/bluetooth.h:339 [inline] bcsp_recv+0x82c/0x13f0
drivers/bluetooth/hci_bcsp.c:685 hci_uart_tty_receive+0x1eb/0x4b0
drivers/bluetooth/hci_ldisc.c:616 tty_ldisc_receive_buf+0x12f/0x160
drivers/tty/tty_buffer.c:460 …
Freed by task 102: … __kfree_skb
net/core/skbuff.c:646 [inline] kfree_skb+0xb6/0x340 net/core/skbuff.c:663 bcsp_recv+0x531/0x13f0
drivers/bluetooth/hci_bcsp.c:623 hci_uart_tty_receive+0x1eb/0x4b0
drivers/bluetooth/hci_ldisc.c:616 tty_ldisc_receive_buf+0x12f/0x160
drivers/tty/tty_buffer.c:460 … |
Anaylizing the source code, you see the skb is allocated at drivers/bluetooth/hci_bcsp.c:685 of function bcsp_close:
![]() |
Figure 12. skb allocation |
Skb is freed at drivers/bluetooth/hci_bcsp.c:623 bcsp_recv:
![]() |
Figure 13. free skb |
After that, it is freed again at drivers/bluetooth/hci_bcsp.c:763 bcsp_close:
![]() |
Figure 14.
double-free skb |
In bcsp_close, bcsp->rx_skb is checked againtst NULL before freeing. However, before that, on line 623, bcsp->rx_skb is freed without being set into NULL. This is the root cause of the bug.
Checking
Linux source code on github
![]() |
Figure
15. commit fixing memory leak but causing double
free |
On 4 Nov, 2019 the bug was fixed by nullifying bcsp->rx_skb every time it is freed
![]() |
Figure
16. fixing double free in bcsp_close |
6. Reproduce crash
View the guide here. Reproduce is the step of find the minimized program that causes the crash. Following is an example of crash log:
![]() |
Figure 17. log crash |
During fuzzing process, the programs are executed multithreadedly (depends on the option procs in config file), so the program that caused the crash does not necessarily immediately precedes it. I use the following method:
- Run the image in qemu.
- scp the files syz-execprog and syz-executor onto the virtual machine.
- Use
syz-execprog to run crash log on the VM. Gradually remove
unrelated programs from the log, until there is only one program left. You can
remove syscalls in the program to make it smaller.
./syz-execprog -executor=./syz-executor -repeat=1 -sandbox=setuid -enable=none -collide=false ./log0
Program I found:
03:47:02 executing program 0: r0 = openat$ptmx(0xffffffffffffff9c,
&(0x7f0000000280)='/dev/ptmx\x00', 0x0, 0x0) ioctl$TIOCSETD(r0, 0x5423, &(0x7f0000000040)=0xf) ioctl$KDADDIO(r0, 0x400455c8, 0x1) r1 = socket$inet6(0xa, 0x400000000001, 0x0) bind$inet6(r1, &(0x7f0000000600)={0xa, 0x4e20, 0x0,
@loopback}, 0x1c) sendto$inet6(r1, 0x0, 0x0, 0x20000008,
&(0x7f00008d4fe4)={0xa, 0x4e20, 0x0, @loopback}, 0x1c) r2 = open(&(0x7f0000000240)='./bus\x00',
0x100000141042, 0x0) ftruncate(r2, 0x10099b7) sendfile(r1, r2, 0x0, 0x8000fffffffe) |
Use syz-prog2c to convert that program into C code: syz-prog2c -prog crash.syz -repeat=1 -enable=none > crash.c
To check if the above C code works, I create a test user, login and run the C program, successfully crash the sytem:
Conclusion
I successfully solve the problem with kernel 4.15.0-60-generic. Without syzbot, with me it is almost impossible (I tried some public exploits but failed). However, more targets should be tested before we can confirm whether this method can be used in practice. If it can, I think the security of Linux kernel needs more attention, such as being more regularly updated to latest version.
References
[2] https://syzkaller.appspot.com/upstream.
[3] https://people.canonical.com/~kernel/info/kernel-version-map.html.
[4] http://us.archive.ubuntu.com/ubuntu/pool/main/l/linux/.
[5] https://syzkaller.appspot.com/text?tag=KernelConfig&x=fac7c3360835a4e0.
[6] https://syzkaller.appspot.com/bug?id=a839269752ca27e5a69938064ba906ac8cf3fb36.
[7] https://github.com/torvalds/linux.
[8] https://github.com/torvalds/linux/commit/4ce9146e0370fcd573f0372d9b4e5a211112567c.
[9] https://github.com/torvalds/linux/commit/cf94da6f502d8caecabd56b194541c873c8a7a3c.
Click here for Vietnamese version
ManhND
Service Center, VinCSS (a member of Vingroup)
Thank you very for this post, I am learning syzkaller, I am very interested at your poc, not sure that you mind to share the code or not?
ReplyDeleteDo you mean the PoC for the Bluetooth bug? You just have to convert the program in syzkaller syntax into C program, using syz-prog2c.
ReplyDeletesorry, I mean how to get root shell from this bug , I want to learn how to write this, and I think this is really a good example for me, if you can share it, that's will be cool!
DeleteThanks versy much
I mean this one: https://www.youtube.com/watch?v=-MTW1KNESQc&t=17s
ReplyDeleteGreat thanks
And one more question: May I have your email? It is good for me to send email to ask question about syzkaller :)
ReplyDelete