Shadow Simulator – run real applications over a simulated Internet topology
shadow.github.io
shadow.github.io
I can see this was developed for NRL research, so not trying to say it's a knock off or anything. Seems to have been developed a few years later, but close to the same time, one of those things like Deep Impact and Armageddon both coming out at the same time.
Anyway, having used mininet quite a bit back in the day in grad school, this kind of thing is great. Not sure exactly the extent of networking you can simulate with this toolset, but mininet lets you put entire simulated autonomous systems on a single laptop and you can mess around with BGP policies. It's really cool, stuff you would normally need to work for AT&T with access to millions of miles of fiber to have a chance to do.
Mininet looks cool!
It looks like Mininet lets programs run pretty close to natively, using network namespaces to pipe data over its simulated network, and some other lightweight techniques for isolation and some extent of reproducibility (e.g. using the fifo scheduler).
Shadow intercepts the libc API via LD_PRELOAD, allowing finer grained control over the process's view of the world. In particular it lets us emulate time - we can run simulations with tens of thousands of endpoints on a single physical machine; even if we don't have the CPU power to run it in real-time, the simulation behaves as if we do. Conversely, small simulations can be run faster than real-time.
The down-side is that it's more difficult to support arbitrary software, since there are various ways of "escaping" LD_PRELOAD, and we don't support the full libc API. We're in the process of switching to a ptrace-based approach that is more robust, at the cost of some additional runtime overhead.
> Not sure exactly the extent of networking you can simulate with this toolset, but mininet lets you put entire simulated autonomous systems on a single laptop and you can mess around with BGP policies.
It sounds like mininet has a lower-level model of the network itself. Shadow simulates links between endpoints, including bandwidth limitations and drop rates. It doesn't model individual routers along the path, so yeah I don't think you could use it to simulate BGP.
I do absolutely have horrendously little networking experience :) so I was probably fiddling with the knobs wrong (and fiddling with wrong expectations from the mechanics), but I didn't see any tunables for ramp-up et al.
For a while I've been idly curious how to iterate on applications that need to respond favorably to everything from 10Gbps and "what latency?" to flaky 3.5G with "give up" latency and worse jitter. At the end of the day I know I need real networks to play with, but I've been curious what software will get me the closest approximation in terms of microsecond-to-microsecond network behavior.
TBH this question is me being somewhat lazy :) - I'm not yet ready to start working on the above ideas, so I'm yet to actually sit down and do some basic prerequisite research... but I thought I'd jump on the opportunity to ask while I could.
We recently added a discussion section to Shadow's GH repo: https://github.com/shadow/shadow/discussions. That'd be a good place for your questions. More knowledgeable folks are pretty responsive there, but aren't on HN :)
The discussion section sounds like exactly the place to follow up on this sort of thing, thanks for the reference. (Once I verify what I'm looking for is actually exotic :P)
Thanks for replying!
Wouldn't eBPF be a better alternative?
We have recently learned of an interesting technique that dettrace [1] uses of combining seccomp with an eBPF filter and ptrace. Instead of generating a ptrace-stop for every syscall (as we do now, using PTRACE_SYSEMU), they use a seccomp policy with an eBPF filter, s.t. a ptrace-stop is only generated for syscalls that violate the policy, allowing them to emulate the result of those syscalls. syscalls that don't violate the policy are allowed to execute natively, saving a lot of overhead.
[1]: https://github.com/dettrace/dettrace
This works great for them since they want to emulate a relatively small subset of syscalls. In our case we want to emulate most syscalls, so it's not as clear-cut of a win. We have found though that if we use an LD_PRELOAD'd shim in the target process to intercept syscalls and then service them via IPC, that's substantially faster than catching them with ptrace. That runs back into the problems with LD_PRELOAD in general of there being various ways of missing syscalls. but, we may be able to use that technique along with ptrace+seccomp+ebpf to intercept any syscalls that we'd otherwise miss. The seccomp technique would allow us to exempt the syscalls that our shim itself is making to do the IPC.
Mininet is a network emulator designed to run real network software. It emerged as a leading tool to evaluate software defined networking solutions. Mininet leverages openvswitch which is the same software that would run on the networking boxes. So the TCP/IP stacks are emulated. This means that the system has very high fidelity making it behave close to the real system. Mininet runs in 'wall-clock' time. (or real-time)
On the other hand, simulation has a more difficult time achieving the same fidelity due to the difference in the execution modes. Discrete event simulation works by maintaining a queue of events. Each event has a timestamp that advances the simulation clock forward. Each event can create 0 or more events. The behavior of the network (adding delays / dropped packets / etc.) are simulated with models. This simulation allows a process to run real application code by invoking it from an event handler. A discrete event simulation uses virtual clocks which can be faster or slower than wall-clock time.
As a general rule we can say that simulation is more scalable than emulation (10s vs 1000000s of nodes) while emulation has higher fidelity (more accurately captures behavior of real system).
Spinning up a network topology simulation using mininet was effortless, and it proved to be an invaluable tool for network research. It's good to see other projects such as Shadow being developed in this space as well.
Shadow's primary use-case is simulating the Tor network. Shadow's tor plugin has a step-by-step for getting a tor network simulation in particular running: https://github.com/shadow/shadow-plugin-tor/wiki
It then uses the an event queue to transfer the packets between virtual hosts, and applies rate-limiting, bandwidth, jitter, packet loss, and TCP stack delays and CPU processing delays, among other things I'm sure. Very cool!
The previous update has links back to the whole series; I stopped including it in the most-recent update since it was getting a bit cumbersome: https://github.com/shadow/shadow/discussions/1060
Ah, ok, without virtualizing/emulating.
That's really a different use case where you're designing the back-haul for the company or university. But that's super powerful as well.