DTrace and Python | Hacker News Reader

DTrace and Python

github.com

129 pointsmyautsai20 comments

Great deck! In particular, for the incredible magic on slide 14, a debt of thanks is owed to John Levon[1] and to whomever has maintained that work and brought it forward.

On slide 28, the presentation asks "What is 'Speculative Tracing'?" It's unclear if that's a rhetorical question, but just to answer it here: speculative tracing is a DTrace facility that allows for data to be traced speculatively, and only committed to the trace buffer if and when some other (later) condition is met.[2] My original inspiration for this was a case that we had back in the day at Sun on the Performance and Application Engineering (PAE) team, when Yufei Zhu (now at Facebook) described a case she had in which one out of every 10,000 mmap()'s was failing with EINVAL -- and it was really tough to use DTrace when she was only interested in its output 0.01% of the time. For Yufei's (motivating) example, speculative tracing offered a way of capturing all of the necessary data on every mmap request, but only emitting that data when the entire operation was found to have failed. Speculative tracing is one of those you-don't-need-it-until-you-need-it features of DTrace (to which I would certainly add anonymous tracing) -- but when you need it, it's a lifesaver, and I have it used it as recently as last week to nail a particularly nasty bug that very much needed it.[3]

[1] https://blogs.oracle.com/levon/entry/python_and_dtrace_in_bu...

[2] http://dtrace.org/guide/chp-spec.html

[3] https://twitter.com/bcantrill/status/769225926726918144

Shameless plug: if you are looking for something similar to DTrace but for Windows that can be also used in Python or any other COM capable programming language... you can take a look at our Deviare Hooking Engine: https://github.com/nektra/Deviare2

Additionally, we have other open source instrumentation engines like https://github.com/nektra/Deviare-InProc (better and more secure than Detours, check [1]), RemoteBridge: https://github.com/nektra/RemoteBridge. SpyStudio will also be open sourced very soon: http://www.nektra.com/products/spystudio-api-monitor/

[1] https://www.blackhat.com/docs/us-16/materials/us-16-Yavo-Cap... and https://www.blackhat.com/docs/us-16/materials/us-16-Yavo-Cap...

interesting.

by the way, i got a google security warning on chrome trying to get to one of the nektra blog pages: Google Safe Browsing recently found harmful programs on blog.nektra.com.

also, have you considered an open source anti-virus program at nektra ? One based on stuff like what's available here: http://www.nsrl.nist.gov/nsrl-faqs.html

Hi @naveen99.

We upload precompiled samples but time to time Google marks us as harmful when they aren't. Feel free to download the source code from https://github.com/nektra/. Visual Studio projects are provided too.

Thanks. Will do.

Yes, we are working on that. Basicly Google is giving that notice because the samples were not signed.

We don't plan to make an antivirus since we don't see a business there but some anti virus companies are using our products.

Can Deviare handle rewriting RIP-relative instructions over 5 bytes long starting within the first 5 bytes? E.g.:

  0:000> uf python27!Py_GetPath
  python27!Py_GetPath [c:\minonda\conda-bld\work\python-2.7.12\pc\getpathp.c @ 688]:

    688 00000000`1e1054f0 4883ec28        sub     rsp,28h
    689 00000000`1e1054f4 488b055dc51d00  mov     rax,qword ptr [python27!module_search_path (00000000`1e2e1a58)]

That second mov instruction will get trampled if you overwrite the `sub rsp,28h` with a jmp, so you need to relocate it as well, which requires re-calculating the RIP-relative IMM32 address embedded within the instruction.

I've found a lot of hooking libraries couldn't handle that.

Hi @threntnelson, yes. The generated code will consist in a copy the sub opcode and the mov will be converted to an absolute mov. Sometimes there is no 1-1 conversion so they are converted to several instructions preserving register values if they must be temporarily used.

Does anyone have any inside knowledge of when, if ever, DTrace might become a standard/official part of Linux? Seems like right now you need to compile a kernel module which will taint your standard kernel in order to use it.

> Does anyone have any inside knowledge of when, if ever, DTrace might become a standard/official part of Linux?

Never. Due to dtrace being licensed under the CDDL it will never be mainlined into the upstream kernel. I would be keen to point out this does not mean there is a conflict of licenses from a legal point of view. Before anyone starts making claims like "you're not allowed to do this", see the use of the ZFS kernel module in recent releases of Ubuntu. I shall say no more on the matter as a refuse to be drawn into a licensing debate...

Getting back on topic, those seeking more on user space providers should check out this video from Bryan Cantrill.

https://www.youtube.com/watch?v=6chLw2aodYQ

Full disclosure I never had to run Solaris in production so I can look back on Sun with rose tinted glasses. That said I still long for many of the features it pioneered. Seriously ZFS would eliminate half of my production problems and dtrace would make the remaining half solvable.

> I would be keen to point out this does not mean there is a conflict of licenses from a legal point of view. Before anyone starts making claims like "you're not allowed to do this", see the use of the ZFS kernel module in recent releases of Ubuntu.

AFAIK it's only packaged as a dkms-package:

  Package: zfs-dkms
  Version: 0.6.5.6-0ubuntu8
  Priority: extra
  Section: universe/kernel
  Source: zfs-linux
  Origin: Ubuntu
  Maintainer: Ubuntu Developers...
  Original-Maintainer: Darik Horn ...
  Bugs: https://bugs.launchpad...
  Installed-Size: 8 325 kB
  Provides: lustre-backend-fs,
    lzfs, lzfs-dkms
  Pre-Depends: spl-dkms (>= 0.6.5.6)
  Depends: dkms (>= 2.2.0.2), zfs-doc (= 0.6.5.6-0ubuntu8)
  Conflicts: lzfs, lzfs-dkms
  Replaces: lzfs, lzfs-dkms
  (...)
  xenial/universe amd64 Packages

Note that this is in "universe" not Ubuntu proper.

From DKMS(8): dkms is a framework which allows kernel modules to be dynamically built for each kernel on your system in a simplified and organized fashion.

> I shall say no more on the matter as a refuse to be drawn into a licensing debate...

I really don't want to start that debate, but I also think the "Ubuntu distributes (binary) ZFS" isn't quite accurate.

Now, how this all shakes out in terms of whether or not ZFS is bundled with Ubuntu ("on the ISO so to speak"), and if it then really makes a difference that it's distributed as source-code that can be automatically compiled -- I don't know. I wouldn't think shipping pythons source (to be later compiled to byte code, pyc) is enough to dodge GPL restrictions on bundling other code/binaries as part of a distribution.

But, yeah, whichever it is, CDDL code isn't likely to ever be upstreamed to the kernel proper.

That's actually only half the story and mostly not relevant... the dkms package is still there, but ZFS is now shipped with the kernel package:

lathiat@ubuntu:~$ dpkg -S zfs.ko linux-image-4.4.0-35-generic: /lib/modules/4.4.0-35-generic/kernel/zfs/zfs/zfs.ko linux-image-4.4.0-36-generic: /lib/modules/4.4.0-36-generic/kernel/zfs/zfs/zfs.ko

Announcement is here: https://insights.ubuntu.com/2016/02/16/zfs-is-the-fs-for-con...

There is a lot of debate about the licensing legality of it, and there are opposing positions from various parties. Here's the official post summarising it: http://blog.dustinkirkland.com/2016/02/zfs-licensing-and-lin...

Thank you for taking the time to reply with those two links. Very interesting (and directly helpful as one of my use-cases is led+zfs. Not needing the dkms module makes things simpler).

That's Linux today. Upgrade. ZFS + DTrace capabilities.

FreeBSD already has DTrace :)

Enhanced BPF is integrated in Linux 4.x series -- as in, it is part of the kernel, and everyone who deploys on Linux will be getting it once they upgrade -- and provides similar capabilities.

I have not yet looked at the new BPF functionality at all, so I desperately hope we are heading towards something more integrated. I really should set aside some time to catch up on some of the recent presentations about it... including yours Brendan :)

The real shame of linux tracing utilities from my time using them, is that there is a large disjointed set of tools and frameworks that you need to use each in different circumstances. And there is no single language for using those tools and then processing the output.

The great thing about dtrace, from using it personally on Solaris, is that it is a single integrated tool where 90% of the time you can get sufficiently summarised output from a single file dtrace script to debug most things. And this lends itself to sharing those scripts for re-use by others, which I have done extensively working on ZFS, iSCSI and NFS. This also makes it much easier for the less informed to take those scripts and run with them into their own modifications, rather than needing to figure a lot more out from the outset.

This doesn't seem overly surprising based on the way the linux kernel is developed, by way of multiple parties mostly implementing there own interests and large projects not commonly being planned out in a co-ordinated fashion. You also have the luxury of being able to view, modify and run your own kernel with custom changes to debug these problems if you need to.

By contrast, on Solaris generally you were stuck with what you got from SunOracle and probably didn't have source or the ability to hack your own modifications in... and there was great business case for them to create a single tooling from the outset.

Unfortunately it will be years before myself and many others get to use BPF on our production systems as we have to run an "enterprise" distribution like RHEL or god forbid SLES. But yes it certainly is good to see steps being made in this direction.

Just give FreeBSD a shot!

Video: https://www.youtube.com/watch?v=HwqvHUGyGTE

Reply on news.ycombinator.com