Intro
During development of our Hypervisor Internals course, we rediscovered N-days in Hypervisors like VirtualBox via fuzzing. Reportedly, the original bug was discovered through static analysis which involved the usual tasks of following control flow through multiple functions and mentally tracking the state of the program to identify incorrect behaviors.
(For an (unrelated to the bug) example, tracking parameters through function calls)
/* Copy common bits from request message frame to reply. */
1333 pReply->Header.u8Function = pMessageHdr->u8Function;
1334 pReply->Header.u32MessageContext = pMessageHdr->u32MessageContext;
1335
1336 lsilogicFinishAddressReply(pDevIns, pThis, pReply, fForceReplyPostFifo);
(And into the functions implementation, mentally tracking how controllable variables are used)
static void lsilogicFinishAddressReply(PPDMDEVINS pDevIns, PLSILOGICSCSI pThis, PMptReplyUnion pReply, bool fForceReplyFifo)
772 {
773 /*
774 * If we are in a doorbell function we set the reply size now and
775 * set the system doorbell status interrupt to notify the guest that
776 * we are ready to send the reply.
777 */
778 if (pThis->enmDoorbellState != LSILOGICDOORBELLSTATE_NOT_IN_USE && !fForceReplyFifo)
The bug itself isn’t overly complex but does require multiple handlers in VirtualBox to be called with various arguments, triggered by executing a specific set of instructions in the VM that cause VM exits, transferring execution to the Hypervisor to parse and act on data controlled by the VM.
The result of the short exercise demonstrated how even rudimentary fuzzing could have discovered the RCE (implying fuzzing was not performed on this codebase, or at least on the vulnerable code blocks).
As an extension to this, I assisted a Junior with retargeting our rudimentary fuzzing harness against a different target within VirtualBox, which led to the discovery of additional new bugs.
Rudimentary Fuzzing?
Fuzzing as a standalone term is broad and tends to abstract away key details around the approach or implementation.
If I’m tasked with fuzzing a component, ask a developer to fuzz the same component, and ask another security researcher to fuzz the same component, it’s likely all 3 of our approaches would be quite different and yield varying results.
The more technical approaches to fuzzing may include:
- Coverage guidance
- Cmplog feedback
- Deterministic execution and input replay (typically via snapshots)
- Structure-aware input generation / mutation
- Optimized execution (eliminating execution of fluff code)
However, fuzzing also encapsulates approaches akin to throwing random bytes at a target and seeing what happens:
#include <time.h>
#include <stdlib.h>
char decode_char(int idx, char* buf) {
return buf[idx] << 20 & 0xfa;
}
int main() {
srand(time(NULL));
char buf[100];
while (true) {
int arg1 = rand();
decode_char(arg1, buf);
}
return 0;
}
You’ll note that the `decode_char` function would crash with even this level of fuzzing, and surprisingly even more complex, real targets fall over with this approach if you can reach them (sometimes the capability to reach the target gets you 80% towards triggering a bug).
That said, this level of basic fuzzing is almost never an approach I’d take for a serious target as prior investment in tooling and a consistent workflow makes the process of advanced fuzzing a minimal effort add.
VirtualBox as a Target
VirtualBox is open-source, making it fairly easy to identify attack surfaces.
Hypervisors also tend to share attack surface enabling the reuse of fuzzing harnesses across multiple Hypervisors, in which case starting with an open-source target may ease the initial development effort.
Architecture Overview
I tend to fuzz Hypervisors from a custom OS, such that the components look like the below:
This shows the following layout:
- Physical Windows Machine
- Windows VM
- VirtualBox running
- Custom OS under VirtualBox
With the idea that the custom OS can target the VirtualBox Hypervisor by performing operations (e.g. behaviors that cause vmexits) that will be forwarded to VirtualBox’s Hypervisor.
Custom OS
The custom OS is Rust based and fairly simple for this target.
We boot as a UEFI-based OS, setup our MMU and paging then jump to fuzzing.
As mentioned earlier, this was a super simple harness developed as an example for a Junior member. We add a little knowledge of various targets in our OS, e.g.:
pub struct ScsiPort {
pub port: IoPort,
pub state: ScsiState,
}
#[derive(Copy, Clone, Default)]
pub enum ScsiState {
#[default]
NoCommand,
ReadTxDir,
ReadCdbHi,
ReadBufSzLsb,
ReadBufSzMid,
ReadCommand,
CommandReady,
}
We don’t even generate inputs with structure awareness or gather any feedback, we simply generate random inputs for randomly chosen operations (from a defined set) and execute them on a live system, this looks something like the below:
loop {
// Read a random fuzz choice
let fuzz_choice: FuzzChoice = x86::read_rand_max(2).try_into().unwrap();
match fuzz_choice {
FuzzChoice::Read => {
// Get a random read operation
let read_op: ReadOperation = x86::read_rand_max(5).try_into().unwrap();
// Get a random length up to and including BUF_SZ
let len = x86::read_rand_max(BUF_SZ as u64 + 1) as usize;
// Execute the read op
read(&mut bport, read_op, Some(&mut buf), len);
}
FuzzChoice::Write => {
// Get a random write operation
let write_op: WriteOperation = x86::read_rand_max(5).try_into().unwrap();
// Get a random length up to and including BUF_SZ
let len = x86::read_rand_max(BUF_SZ as u64 + 1) as usize;
// Get a random u8 to write
let val_u8 = x86::read_rand_max((u8::MAX as u64) + 1) as u8;
// Execute the write op
write(&mut bport, write_op, val_u8, Some(&mut buf), len);
}
}
}
We also implement an interrupt handler to catch any faults that may occur, and in our handler we perform a cpu reset to automatically restart fuzzing in our VM.
Catching Bugs
VirtualBox doesn’t make it simple to catch/debug crashes, attaching a debugger to the running VMM process won’t work / be permitted by the Hypervisor.
One solution then is to leverage kernel debugging of the Windows OS running VirtualBox, we can then break on functions like NtRaiseHardError (which is called by VirtualBox when the VMM process crashes) or on associated exception handling routines.
Of course for this particular target we always have the source to modify too.
Outro
Rather than dive into the details of a particular bug or release of a tool as we’ve done before, this exercise was originally to begin exposure of Hypervisor fuzzing to a Junior member, others may find the exposure to the high-level architecture useful.
A lot of Hypervisor bugs I’ve seen in the public tend to be triggered by hooking or modifying a running Linux or Windows kernel and fuzzing/triggering the bug from there (can be less effort, especially if the target is behind a protocol like VMBUS which from a custom OS approach requires implementing support for the protocol), though there has been a few notable publications of utilizing a custom OS for Hypervisor research (also useful for research targeting CPU bugs!)
