Usercorn – a versatile kernel+system+userspace emulator | Hacker News Reader

Usercorn – a versatile kernel+system+userspace emulator

github.com

37 pointsmre13 comments

One of the many practical neat things you can do when you're emulating all the software of a machine, is visually analyzing memory relatively easily: https://www.youtube.com/watch?v=25nLk1Z8wd0

Author here. Usercorn at a basic level is like qemu-user in that it can load a userspace binary like ELF, MachO, etc into a CPU emulator and provide a syscall emulator interface.

It's built as an API that allows instrumentation of the resulting binary. You can hook instructions, basic blocks, syscalls, memory access... You get information like stacktraces in a cross-platform manner. There are helpers to compute and display diffs of memory and registers throughout execution.

There are many differences from qemu-user:

Usercorn attempts to emulate several OS/syscall ABIs all from one binary (Linux, BSD, XNU, CGC, more to come like Redox, maybe weird stuff like Windows and DOS). qemu-user just translates arguments to your host kernel so to run a Linux binary you need a Linux host. qemu-user also has a different binary for each arch, like qemu-i386 and qemu-arm. The primary usercorn binary can emulate every single supported OS/arch combination and aims automatically detect the correct arch/kernel when you run a program.

Usercorn is being built as a framework around the Unicorn Engine. It's not just a way to run a binary on the command line. It can be loaded as a library (`NewUsercorn("binary").Run(args, env)` will run an arbitrary supported app). It allows hooking many things and I've been building a lot of features around static trace/analysis. I'm extremely excited about some of the features and supporting tools I have planned.

This twitter thread shows examples of a tool called imgtrace I built with Usercorn. imgtrace can visualize memory writes done by a program. The Hilbert curve view of MD5 is nice, as it makes the memory containing the internal MD5 state as well as the iterators very obvious at a glance: https://twitter.com/lunixbochs/status/717945725024280578

The underlying code of imgtrace is also extremely simple (basically all of it is to create the image data), which shows off the power of Usercorn's simplicity: https://github.com/lunixbochs/usercorn/tree/master/go/cmd/im...

Usercorn is much less mature than qemu-user. It supports around ~50/400 Posix syscalls and is missing many architecture-specific features. Some architectures still need work on memory segmentation and thread-local storage. I've split up functionality internally in a way that makes sense, but I'm still building the base functionality and haven't designed the public API at all yet (even though it's possible to use Usercorn from external code, that's just happenstance due to the functions/types I exposed while building it).

----

Architecture support is tied to Unicorn. Usercorn supports x86_64 best, and has various levels of support for ARM, MIPS, sparc, and m68k.

It's still very much WIP, but can run many binaries at this point, even Linux binaries dynamically linked to glibc. Host support is best on OS X and Linux. Guest support is best on Linux.

Wow this is seriously cool. Took a while of going through it and the links to Unicorn to understand what it was (your top level comment wasn't up yet).

I think this snippet from your Examples section sums it up best:

    usercorn bins/x86.linux.elf
    usercorn bins/x86_64.linux.elf
    usercorn bins/x86.darwin.macho
    usercorn bins/x86_64.darwin.macho
    usercorn bins/x86.linux.cgc
    usercorn bins/mipsel.linux.elf

That only explains the qemu-user part. The ability to run binaries in Usercorn is just a means to an end - I want to build tools in the vein of `maybe` (VFS layer will make this easy, and so much safer), `imgtrace`, `strace`, and some stuff that's 1000x more ambitious than these.

Running binaries is just... I'm recreating the universe so I can change just a few things, like logging memory access or doing taint analysis.

This is similar to tools like DynamoRIO, DrMemory, PIN, valgrind, but it not only covers all of those spaces, I could probably spend a few hours listing interesting stuff outside those scopes I want to build on Usercorn (for example, you could ignore the CPU emulation part of of Usercorn and use just the kernel emulation interface to implement the Linuxulator or Windows10+Linux compat layer natively for an OS).

I'll probably write real docs sometime after a proper API, but both are lower priority than core features atm.

My rough view of emulating userspace and the syscall ABI is that the three main tricky parts are (1) signals (2) threads (3) mmap and generally handling the guest address space; most of the rest is just a big pile of fiddly details. How does Usercorn do there currently? (Among other things there are some pretty tricky race conditions to avoid between signals and syscalls to make sure you return to the guest and don't acidentally block forever in a syscall.)

(I work on qemu, including its linux-user stuff...)

Posix mmap is generally working, because that's easily mapped to a core API in Unicorn. I exposed the qemu mem_map_ptr API for mapping files as well, but I haven't integrated it yet (I emulate file mmap for now by copying the file into guest memory).

Signals are a pretty high priority, but are currently unimplemented. Usercorn has a trampoline mode that allows stopping the CPU, executing a section of code, and returning to the previous location when it's done. I can use this for signal dispatch. sigaction/signal syscalls will just register the address to call.

fork/clone/threads in general are not yet implemented, but the groundwork is mostly there - I'm mainly worried about memory synchronization, and might need to add APIs to Unicorn for this. Unicorn supports running an arbitrary number of CPUs in parallel in the same process, so I'm actually planning on making both forked processes and threads run in the same host process, and do a sort of PID namespace/transparent layer.

This will make stuff like emulating IPC or a process ptracing another process much easier to control, and I won't need an external process like wineserver to negotiate stuff the native system doesn't know how to do.

Unicorn engine home: http://www.unicorn-engine.org/

Is Unicorn a fork which will eventually grow into another Qemu or just a fork to remove stuff and make it into something else that doesn't have the same goals. I'm confused because the website on the one hand says it's less functionality but then also that it does more than qemu. So which is it? Or is it just that they provide bindings and an API which they tout as "more than qemu".

Unicorn is a massive patch on top of QEMU 2.2 [1] that approximately does the following:

- Removes a couple of architectures (because they're not supported yet for hooks/register access).

- Removes hardware support, as well as the qemu-system and qemu-user targets.

- Modifies the JIT and all supporting APIs to not use globals. This allows you to compile JIT support for more than one guest architecture into the same binary, as well as run more than one instance of the JIT at once (so you can emulate two CPUs in parallel, even of different architectures).

- Adds a hook infrastructure to call callback function(s) with metadata on many different events, including memory access, memory fault, interrupt, basic block entry, instruction execution (so like per instruction address). These calls are inserted into the JIT stream only when you add a hook, so they're reasonably efficient.

- Adds register read/write APIs (this is actually a ton of code in the form of a switch/case table for each arch)

- Exposes all of this using a simple C library / API:

    uc_open(arch, mode)
    uc_mem_map()
    uc_mem_write/read()
    uc_reg_write/read()
    uc_emu_start/stop()
    uc_hook_add/del()

[1] I'm currently porting it to QEMU 2.5 because I need one new ARM NEON instruction. This is probably overkill, but it's a good cause.

Thanks for the detailed explanation. Does that mean it's not a replacement for QEMU and not intended to be?

Regarding qemu-user and qemu-system, I've had bad luck with it all the time. In 2.5-rc2 I got qemu-system-x86_64 but when I build 2.5 with the same configure flags, I don't get qemu-system-x86_64 at all. You wouldn't happen to know how to work around that, would you? As I use it just for kvm functionality for vms, I configure it like this, but qemu-2.5 final's build system seems to be broken.

    --target-list=x86_64-linux-user
    --enable-pie
    --enable-seccomp
    --enable-curses

The last known version that was able to build qemu-system-x86_64 is 2.4.92.

Any idea?

1) It really depends on the features of QEMU you use. Unicorn and friends could eventually support every feature of QEMU, but currently are nowhere near that. Usercorn should be able to replace qemu-user eventualy.

2) You should really go to the QEMU team for support on that.

2) Seems like the configure (and/or build) script was broken at some point. Letting it build all targets does not omit the amd64 kvm executable.

1) But I don't really understand the end goal if you're always forking of the next qemu relese to patch stuff back in. Is there no interest in qemu upstream for those features? I've seen it for the first time 2.6-rc, so is qapi some kind of API you might adapt or is it too limited?

At this point Unicorn is a hard fork with a very specific goal. It accomplishes this goal much better than upstream QEMU. I believe combined TCG / no-globals should be upstreamed (I think I actually saw commits in 2.5 about supporting multiple architectures in one QEMU binary?), but I don't know if anyone working on Unicorn has talked to anyone upstream about this.

I believe Unicorn also contains QEMU bugfixes.

To me, in an ideal world, we might have something like this:

libqemu-cpu - Basically Unicorn. C library/API to easily create/hook/manipulate one or more CPUs.

libqemu-system - All hardware/peripheral support from QEMU. Attach any component(s) to a supported CPU at runtime.

libqemu-user - The current qemu-user, except as a library with syscall/whatever hooking support. There'd probably be a good amount of code generation here.

Reply on news.ycombinator.com