Hexyl: A command-line hex viewer | Hacker News Reader

Hexyl: A command-line hex viewer

github.com

456 pointsokanesen113 comments

I admit, my first thought on seeing the title was "What's wrong with xxd?"

This is nice! I like the colorization a lot -- it's like with source code, you don't realize how much you rely on the colors until you get a new computer and need to download syntax definitions for your editor again. Only feature I really think is missing is line numbers.

Nicely done!

haha, had exactly the same thoughts. the colours really makes it much much easier on the eyes to recognise things. well done!

Same thought except, I wonder why this isn’t a patch to xxd?

I'm struggling to even find the canonical source code for xxd, let alone a way to provide a patch. You could probably email the address in the man page, but that's from 1997 so I wouldn't be too optimistic about getting a response. And even then it's quite likely that the original author considers xxd finished and would be reluctant to accept patches for it anyway. On top of all that the hard part of this work (cross-platform terminal colours) had already been done for the author, so the implementation was probably far easier than adding colour output to xxd would have been.

Cross-platform terminal coloring was solved in the 1980s with termcap and then terminfo, for every terminal that's even come close to seeing mainstream use and which had colors in the first place. The libraries are part of the ncurses library and are shipped with every OS this stuff would build on to begin with.

This also solves the problem of moving the cursor and drawing a screen in general.

Aren't termcap and terminfo linux (or maybe unix) only?

termcap is supported on FreeBSD, at the least.

Both of them long predate Linux. They're definitely Unix-flavored, but so's the Berkeley sockets library, and that's available everywhere, too. Ultimately, they're a library which is packaged with a database, and nothing about them is very system-specific except the terminal information in the database.

https://en.wikipedia.org/wiki/Terminfo

termcap was written by Bill Joy in 1978. terminfo dates back to 1982.

Eh, A lot of modern terminal emulators don't seem to bother providing termcap/info entries, or setting their own $TERM variables. In reality, most code just fires off ANSI sequences and hopes for the best. Looking quickly at the source for Hexyl, I think that's what the ansi_term library it uses does.

If anything, I've found doing this works better then trying to muck around with ncurses, which often gets actively mislead by incorrect terminfo entries.

ncurses wins once you use terminals beyond xterm.

termcap/terminfo styling is limited to features like underline, bold and reverse. No coloring, per se.

"man terminfo | grep color" shows that pretty clearly to not be the case. There's plenty of codes for setting colors.

It's entirely possible termcap and terminfo have been allowing applications to color their output for longer than you've been alive.

The downvotes prove how much HN hates history, I suppose.

xxd is distributed with Vim.

I've never thought of xxd as being anything but a "Vim thing".

That it had its own history (1990-1998) before being pulled into Vim is a TIL for me.

Here you go:

https://github.com/vim/vim/tree/master/src/xxd

There might still be other versions of this out there, but I suspect most OS distros pick up xxd by way of packaging Vim.

So if you work on some other xxd, you're probably in a dead fork.

The man page is also from Vim:

https://github.com/vim/vim/blob/master/runtime/doc/xxd.1

My first patch to xxd, if I were to work on it, would be to fix the man page to indicate clearly that it's the xxd bundled with Vim.

Because the author wanted to do his own, personal thing?

I get the idea behind your question, but it irks me a bit since it implies a sort of responsibility to contribute to an existing piece of software. This is a net positive of course, but the beauty of open source is the freedom to work on the things you want to work on.

It's illegible in my terminal. Not a fan of the trend to hardcoded unicorn vomit.

I have to admit, my first question in mind was how to disable colors because I do not use syntax highlighting at all.

but if you want to disable colours in this why not use hexdump or xxd :D . it's like the only feature it adds to standard hexdump tools

Me neither.

I did use it religiously for a long, long time; then tried without for a week and never went back.

It's not like telling the difference between string literals and keywords was ever a major issue for me. I guess it could help short term when learning a new language, but I'm pretty sure it slows down overall progress.

How does it slow down progress?

For me it doesn't. It's a quick way to see that I made a typo when a keyword doesn't turn the right color, or that I closed a strong with the wrong quote Mark.

I find it's a bit like navigating using GPS, once I get there I still have no clue because I was mostly acting on cues.

Slow compiles have the same effect for me, I get pushed out of flow and into reactive mode.

The only way to write good code is to be proactive, and any features that interfere with that process are worse than whatever "problems" they "solve".

I'm also not a fan of the colouring, especially in this application, because when you're looking at a hexdump chances are you don't care about ASCII or whitespace --- and it's not as if you couldn't tell whether something is in the ASCII range, given the character representation in the right column anyway.

The border lines don't help much either and seem more like decoration; addresses to the left and an indexing header on the top, like the traditional "canonical" hexdump format, would be far more useful.

How can you tell the difference between an unprintable character and whatever encodes them (I've most often seen '.'), in a single character cell, without color?

What are line numbers in context of a binary?

Addresses.

File offsets; not addresses :) I assume the tool isn't reading ELF format files and working out the memory address.

Right, I was being a bit loose there. I meant something like how `xxd` displays the number of bytes up to the current output row. Just something to give context of where you are in the file.

    00006980: 0000 0000 0000 0000 0000 0000 0000 0000

You can quickly find position of byte number X using:

line number = floor(X/(bytes/line)) + 1

column offset = X % (bytes/line) + 1

Or it could just print byte numbers instead of line numbers.

> I admit, my first thought on seeing the title was "What's wrong with xxd?"

My first thought too. Then I looked and saw it was nice, and then I noticed it's by sharkdp. He really has a knack for improving on things you thought were fine (fd and bat being other examples).

Looks like the screenshots on Github are outdated. I just ran it locally and it does print out offsets in a new left-most column.

edit: Added in this commit: https://github.com/sharkdp/hexyl/commit/91a119f4537f746045c8...

Liked that though the boxes seem to be a distraction. Perhaps they could use the dim ansi code or be turned off.

Just looked at manpages on someone else's account with no colour highlighting, definitely missed it.

Since back in my DOS programming days, I've lived in several different Hex editors when analysing data streams. Some great, some ordinary. I would have killed for colour highlighting like Hexyl seems to have.

I wonder though, whether I can colour code 'blocks' to differentiate them? One of the things I seemed to do a lot in hex editors was to check out the differentiation in, for example, 72 byte blocks of data, so to be able to delineate 72 byte blocks in different colours in the hex editor would make it easier to see where each block starts/ends.

One ludicrous nitpick: the logo doesn't show anything hexyl, that's 1,2,3,5-tetramethylbenzene. If it was a functional group, it might be 3,4,5-trimethylbenzyl or something like that.

Hexyl would look like:

  R
   \/\/\/

The logo shows a small part of this molecule: https://en.wikipedia.org/wiki/Hexanitrodiphenylamine

.. which is apparently also called "Hexyl". Granted, I was just looking for a short word that starts with "Hex" and I was never good at organic chemistry :-)

Many of your projects have cool logos, how do you make them?

Thank you. Not all of them are by me, but I typically just use Inkscape.

While reading this my hope was it not being written in JavaScript. I'm not disappointed ;)

261 lines of pure Rust by the look of it.

I was hoping that it's written in Rust. I wasn't disappointed either.

Yeah! -- Because prejudice rocks and JavaScript sucks right?

Every problem requires appropriate tools. Why would you use a slow and resource-intensive language for a thing as simple as hex viewing?

It's comparably slow though :p

    xxd `which hexyl` > /dev/null  0.12s user 0.09s system 99% cpu 0.219 total
    hexdump `which hexyl` > /dev/null  0.19s user 0.05s system 91% cpu 0.256 total
    hexyl `which hexyl` > /dev/null  1.69s user 0.39s system 95% cpu 2.175 total

It takes a second to compute all that color. Nice tool though - but what I would really like to see is a --color argument added to xxd as well, as I'll probably forget about hexyl down the line and start looking for xxd again (which is conveniently already installed on most *nix systems).

The same author also wrote hyperfine, a tool to compare performance of various program runs.

    hyperfine './target/release/hexyl ./target/release/hexyl' 'xxd ./target/release/hexyl' 'hexdump ./target/release/hexyl'
    Benchmark #1: ./target/release/hexyl ./target/release/hexyl
      Time (mean ± σ):      1.529 s ±  0.028 s    [User: 1.476 s, System: 0.050 s]
      Range (min … max):    1.491 s …  1.581 s    10 runs

    Benchmark #2: xxd ./target/release/hexyl
      Time (mean ± σ):      70.5 ms ±   0.5 ms    [User: 68.0 ms, System: 1.2 ms]
      Range (min … max):    69.5 ms …  72.3 ms    41 runs

    Benchmark #3: hexdump ./target/release/hexyl
      Time (mean ± σ):     262.4 ms ±   2.8 ms    [User: 260.1 ms, System: 1.5 ms]
      Range (min … max):   259.8 ms … 268.8 ms    11 runs

    Summary
      'xxd ./target/release/hexyl' ran
        3.72 ± 0.05 times faster than 'hexdump ./target/release/hexyl'
       21.70 ± 0.43 times faster than './target/release/hexyl ./target/release/hexyl'

Currently hexyl seems nearly 22x slower than xxd.

... and I have already used hyperfine to benchmark hexyl as well :-)

Yes, it's a shame. But I don't think there is too much we can do about it. We have to print much more to the console due to the ANSI escape codes and we also have to do some conditional checks ON EACH BYTE in order to colorize them correctly. Surely there are some ways to speed everything up a little bit, but in the end I don't think its a real issue. Nobody is going to look at 1MB dumps in a console hex viewer (that's 60,000 lines of output!) without restricting it to some region. And if somebody really wants to, he can probably spare 1.5 seconds to wait for the output :-)

We have to print much more to the console due to the ANSI escape codes and we also have to do some conditional checks ON EACH BYTE in order to colorize them correctly.

A few extra comparisons and output for each byte shouldn't be that much slower; fortunately the function of this program is extremely well-defined, so we can calculate some estimates. Assuming a billion instructions per second, taking ~1.5s to hexdump ~1 million bytes means each byte is consuming ~1500 instructions to process. In reality the time above is probably on a faster CPU, so that number maybe 2-3x more. That is a shockingly high number just to split a byte into two nybbles (expected to be 1-3 instructions), convert the nybbles into ASCII (~3 instructions), and decide on the colour (let's be very generous and say ~100 instructions.)

The fact that the binary itself is >1MB is also rather surprising, especially given that the source (not familiar with Rust, but still understandable) seems quite small and straightforward.

Rust binaries can be large because unlike C, the standard library is statically linked, as well as jemalloc. Jemalloc will no longer be the default as of the next release, so that will shave off ~300k...

What's replacing Jemalloc?

The system malloc implementation. Users who want to use jemalloc have to opt in, but doing so is relatively easy (using the jemallocator crate from crates.io).

Why was this done?

Did rust become less dependent on allocator performance, or did system allocators improve enough? IIRC glibc malloc has improved a lot over the last few years, particularly for multithreaded use, but I don't know about windows / macOS.

So, long ago, Rust actually had a large, Erlang-like runtime. So jemalloc was used. Over time, we shed more and more of this runtime, but jemalloc stayed. We didn't have a pluggable allocator story, and so we couldn't really remove it without causing a regression for people who do need jemalloc. Additionally, jemalloc was already removed on some platforms for a long time; Windows has been shipping the system allocator for as long as I can remember.

So, now that we have a stable way to let you use jemalloc, the right default for a systems language is to use the system allocator. If jemalloc makes sense for you, you can still use it, but if not, you save a non-significant amount of binary size, which matters to a lot of people. See the parent I originally replied to for an example of a very common response when looking at Rust binary sizes.

It's really more about letting you choose the tradeoff than it is about specific improvements between the allocators.

It seems I was wrong. The new hexyl version is significantly faster (see my other comment)

You may be able to speed things up by using a lookup table instead of branching.

(If it's spending a lot of time in Rust's format function you could also use a (or the same) lookup table to convert to hex/dec/oct.)

The format function is going to end up allocating a string for every single byte. That's a huge overhead.

Edit: Turns out to be about 22% overhead, see https://github.com/sharkdp/hexyl/pull/23. Also it was 2 strings per byte, not 1.

Thanks to that PR, hexyl is now slightly faster than hexdump. Both are about a factor of 2-3 slower than xxd:

    Benchmark #1: hexyl $(which hexyl)
      Time (mean ± σ):     169.8 ms ±   8.2 ms    [User: 152.5 ms, System: 17.1 ms]
      Range (min … max):   162.2 ms … 189.1 ms    16 runs
     
    Benchmark #2: hexdump -C $(which hexyl)
      Time (mean ± σ):     188.5 ms ±   4.4 ms    [User: 186.2 ms, System: 2.2 ms]
      Range (min … max):   184.1 ms … 198.2 ms    14 runs
     
    Benchmark #3: xxd $(which hexyl)
      Time (mean ± σ):      72.8 ms ±   2.7 ms    [User: 71.9 ms, System: 1.1 ms]
      Range (min … max):    71.0 ms …  87.8 ms    40 runs

I made a little clone for fun and got a bit carried away optimising. Now at about 3x the speed of hexyl 0.3.1:

https://github.com/sjmulder/hxl

Most of the improvement came from not using printf, fputs, and putchar in favour of operating directly on an array for the line that can be fwritten in one call.

Fixed in v0.3.1 (https://github.com/sharkdp/hexyl/releases/tag/v0.3.1) :-)

What difference would that make?

It is possible to use also radare2[1] for this, along with radiff2. It provides more options and has zero external dependencies.

[1] https://github.com/radare/radare2

This is great. The same author also wrote insect, which is cool, too. https://github.com/sharkdp/insect

Very cool! I greatly appreciate the author's inclusion of a `Pros and cons` section in the README, particularly the cons.

Wow! That guy is a monster, so much great stuff

For something with a little more firepower, I like: https://github.com/evanmiller/hecate

the author appears to be a true rust master, he wrote quite a few neat programs in rust, very impressive and that actually got me interested in rust lang

Same here, I like small and neat Rust projects like this, the source code is concise and easy to read.

He also authors https://github.com/sharkdp/bat, which is also worth to check out.

bat is awesome, I'm using it daily since I found that, better that 'cat'

261 lines of Rust. Hot damn, I couldn't do it in Python.

As someone who is considering learning Rust, this may just have pushed me over the edge.

I use hexer for this, which is an editor too.

http://blog.metaclassofnil.com/?p=757

There's also a vim mode for hex, which I use less often:

https://vi.stackexchange.com/questions/2232/how-can-i-use-vi...

One of the problems with vim+xxd (and many other hex editors) is that it loads the entire file, so you can't easily use it to look at large files (archives, disk images, etc).

Many moons ago I wrote a very bare bones ncurses hex viewer/editor, after not really being able to find one that fit my needs. I still use it pretty regularly: https://github.com/quo/hex.py

There's hexl-mode for emacs that's pretty cool as well:

https://www.gnu.org/software/emacs/manual/html_node/emacs/Ed...

Needs a diff mode so that it can also replace vbindiff.

I do not use a hex editor/viewer often. Not at all. But when I do, I am painfully aware of how sparse my toolbelt just became.

This little program should fit nicely into that category. An area I would like to explore, but have little-to-none incentives.

I've personally found vbindiff excellent for viewing files as hex: https://www.cjmweb.net/vbindiff/

The colorized output is a nice incremental improvement on xxd. And, I think that it's nice that the README doesn't unnecessarily emphasize that it's written in Rust.

Slightly off tangent: One tool I have used to dump data (and more) is cxmon. A very nice monitor if one grew up using monitor cartridges on C64, Amiga etc:

http://cxmon.cebix.net/

https://github.com/cebix/macemu/tree/master/cxmon

Has anyone compared this to dhex, hexer, or ht?

The colors remind me of editing MS-DOS files & drive structures with Norton Disk Doctor! Fantastic!

The repo author has some pretty interesting rust projects. Worth checking out.

Cool tool :) having a hex dumper that understood UTF-8 and dumped the characters as one of the columns would be awesome too. These days, ASCII is a small subset of strings.

Thank you! See: https://github.com/sharkdp/hexyl/issues/6

Just a minor thing while you're eyeballing this thread: I noticed your fork of lsd has the wrong build instructions.

https://github.com/sharkdp/lsd#from-sources

Good work on these tools though. You make me want to learn Rust :)

I forked lsd just to fix these build instructions: https://github.com/Peltoche/lsd/pull/33

I removed my fork...

Ahhh yes. Apologies.

Sorry to ask, but what is he use case for a hex viewer like this? What are the kinds of projects you might be working on when you use something like this?

You come across an undescript binary. It might be a jpeg with an altered extension, malware or basically anything. You open it in a hex viewer and try to gather some info.

Your lab manager goes "hey, can you generate these proprietary binary files (currently made with a horrible gui) to control this robot programmatically".

But shouldn't one use an editor, like vim, in those cases?

Uh, vim doesn't open binary files very well, and the initial goal is to just display them to figure out how they work not to edit them.

Afterwards you will probably want to run experiments with a hex editor, but I found that xxd + vim was better for reading then doing so directly in a hex editor at the time. If I did it again I'd likely start with this because of the color.

So i have been working on an embedded project lately where i had to export the framebuffer to an sd Card. I used a raw Format first (netppm) and had to use vim in hex Mode to validate the header. So this is really neat. I was looking for an padding option in the readme but i guess i ll have to install it later

This isn't very common, but I was trying to build a common CLI tool in the browser to use in an internal application at work. I compared output from my implementation with the CLI tool until I got to feature parity (and to clear up confusing things in the spec/source code).

For other things, I've used it for examining network packets (binary protocols), mystery files, and proprietary data formats. I've also used a hex editor to hack executables (most notably hacking in a dark theme into Unity3D).

One (of many) uses I used xxd for: viewing a byte stream that is supposed to be utf-8 text, but the non-ascii characters in the file are not displaying properly. Hex viewers will let you see exactly what bytes are present, without interpretation. As for the utf-8 stream, it turned out to have been double encoded into utf-8.

I've been looking for a command line hex editor too.

bpe: https://sourceforge.net/projects/bpe/

Or the Joe editor in -hex mode: https://joe-editor.sourceforge.io/

Would have loved being able to run it on windows. I use hex viewers for embedded work and have been looking for a decent one for Win.

It works just fine on Windows (you probably need PowerShell for the ANSI escape codes)

Ok, been a lurker on HN for awhile, but created an account to post.

This is freaking awesome. Been looking for something like this for awhile!

I wish it would work with less or have less like features, taking the first n bytes is okay, but pretty minimal.

Is anyone aware of a React component that implements a decent hex viewer?

Hope this gets into distros

It will.

e.g. https://tracker.debian.org/pkg/rust-hexyl

I guess I am too old, but I use mcview. Or HIEW.

why am i seeing ugly background-colors with rxvt-unicode https://imgur.com/a/qkzAkqR instead of the good foreground colors I see in xfce4-terminal https://imgur.com/a/LY6bPn9

i do seem to have 256 colors in urxvt my $TERM is rxvt-256color and i have rxvt-unicode-256color installed

Please see: https://github.com/sharkdp/hexyl/issues/9

I miss hiew.

If I can make a suggestion - I don't mind the program being slow because of all the logic - how would it be if you didn't give special status to ascii but instead tried to intelligently assume a string encoding for strings and accordingly colored "binary" vs "text" (utf-8, utf-16)

What's wrong with hexdump?

It's not written in Rust.

Nothing. It's doing fine. Thanks for asking.

Reply on news.ycombinator.com