Rootkit blocking using virtualization
Rootkits can affect both Unix style and Windows systems. They are modifications to the kernel that are made by modifying the operating system kernel in memory, often modifying the system call table or some syscall kernel code, to hide the fact that the system has been changed, or to block certain functionality.
Virtualization gives us a unique avenue to stop rootkits from infecting virtual machines: when bootup of the virtual machine is almost done, we can simply mark all the kernel memory of the virtual machine read-only. That way rootkits will not be able to infect the kernel in the virtual machine. Since untrusted applications only run in the virtual machine, and not in the host OS, there is no way to circumvent this security mechanism.
Of course, there are a number of issues that make this project difficult:
- Memory remapping: a rootkit could be changed to, instead of overwriting kernel memory, modify the kernel page tables to point to other memory. The security mechanism needs to make sure that kernel page tables of every process, including recently forked processes, point to the read-only memory.
- Unmodified guest OS: a paravirtualized guest OS can tell the virtual machine monitor when to make memory read-only and what memory. With an unmodified guest, the virtualization software will have to determine somehow when guest kernel memory can be made read-only. Maybe at the first switch to userland?
- Security software: legitimate security software sometimes uses the same kinds of tricks used by rootkits.
Difficulty: 7
Contact: Rik van Riel - riel (at) redhat.com