Reverse engineering the 1988 NeXT keyboard protocol | Hacker News Reader

Reverse engineering the 1988 NeXT keyboard protocol

journal.spencerwnelson.com

239 pointsspenczar561 comments

> When I told my Arduino to sleep for 51 microseconds, it was generally sleeping for either 56.2 or 61.3 microseconds. That seemed to easily be bad enough to cause lots of problems.

This is why I've never quite liked the Arduino ecosystem, the way it exists. Sure, it works and it's accessible to newcomers... but it doesn't encourage understanding the inner workings of the platform enough, which is something that is extremely valuable and accessible when you're working with simple 8-bit microcontrollers. Learning about hardware registers and clocks and cycle accuracy really isn't that scary, that's the point of using a tiny (and frankly obsolete) 8-bit platform!

But when you give people an API that takes an integer number of microseconds, and then approximates it, people are going to think they can bit bang an async serial protocol in C using it, and they will be very confused when it doesn't work (as happened here), and you end up in cargo cultish land of trying to work with bad data instead of developing an actual understanding of the hardware you're working with.

I've seen so many people thinking they need to design their bespoke board around an Arduino shield connector because the thought of just throwing on the ATMega directly scares them, even though the thing only needs 5V, GND, and crystal (if that), because what would they do outside the comfortable ecosystem? And it just makes me sad. If you're at the point you're designing your own semi complex boards and you haven't graduated from Arduino yet, you're not letting yourself grow.

(Or maybe I'm just an old millennial and I think these zoomers should be learning microcontrollers by programming in assembly like I did when I was 10 and using the PIClist delay routine generator and counting my cycles before Arduino existed, now get off my lawn? You decide :-) )

Right, it's complicated, isn't it?

Like, I couldn't (or at least wouldn't) have gotten started on this project if I had to read the ATMega reference manuals from the very first step. But I did feel very frustrated that the Arduino "delayMicroseconds" function is so far off from accurate, and I felt frustrated also whenever I looked for deeper explanations of almost anything. It's a very copy-and-paste culture.

I eventually did find programming things more directly to be quite rewarding and worthwhile. My code is still in a ".ino" file, and uses some Arduino stuff, but it's basically built around ISRs in some frankencode middle state.

There's a lot of room for someone to create good tutorials and material for ramping up and out of Arduino. Would have helped me a lot, at least.

I wish the Arduino IDE actually tried to be educational instead of just being a dumbed down wrapper around avr-gcc with automatic #includes. There are so many things that could be done... stuff like showing side by side disassembly of functions, with register names displayed symbolically. Or even just basic IDE stuff, like built in contextual function documentation.

And then it should have an "export self-contained project" feature that gives you a directory containing a Makefile and all the dependencies to build your .ino, as source code form (and none of the ones you don't use), so you can easily see exactly what is getting built under the hood and can use that as a bridge to working outside the ecosystem. At the scale of 8-bit micros there's no reason not to work with copies of your dependencies, and it's very educational seeing everything in one place (and being able to hack on it) instead of having it scattered in a bunch of global package paths.

The frameworks and libraries are useful (if opaque and quirky and underdocumented), but the IDE is just so underwhelming... heck, it doesn't even manage to be a decent text editor.

Getting started with Arduino makes total sense, nobody's expecting newcomers to start off with the IC datasheet... but they should be able to eventually read the parts they need and work with it directly, instead of relying only on software abstractions. That's the beauty of these systems.

I have found PlatformIO to be a good step up from the Arduino IDE: it generates a new environment for each project with a sane folder structure, and it is possible to dive into the implementation of any function just with the usual Ctrl+click. It helped me a lot to understand how the Arduino libraries worked under the hood and to make changes in them when I needed it.

>There's a lot of room for someone to create good tutorials and material for ramping up and out of Arduino. Would have helped me a lot, at least.

Good news they do exist (at least if i understood your statement correctly), You just need to search with the keywords "AVR", "atmega328", or whichever version you intend to use instead of "Arduino".

I attempted to make a simple PLC this summer and these resources were a great help :

[book] Make: AVR Programming https://www.amazon.com/AVR-Programming-Learning-Software-Tec...

and this complementary (or introductory) video to the book by the author i believe https://www.youtube.com/watch?v=ERY7d7W-6nA

[Youtube-videos] A playlist of Cornell's ECE 4760 AVR Lectures by the very chill "Bruce Land" (from 2012) : https://www.youtube.com/playlist?list=PLD7F7ED1F3505D8D5

The course website (labs, exercises and readings) : https://people.ece.cornell.edu/land/courses/ece4760/index_20...

Homeworks : https://people.ece.cornell.edu/land/courses/ece4760/homework... Labs : https://people.ece.cornell.edu/land/courses/ece4760/labs/old...

Although the course was centered around an atmega16 or something? i really don't recall sorry but the knowledge i gained was invaluable and easily transferable to a 328p.

Fundamentals of Microcontrollers - Arduino bare-metal breakdown (A playlist of a guy "Mitch Davis" exploring an atmega328 barebone and using tools like AVRdude etc..) https://www.youtube.com/playlist?list=PLNyfXcjhOAwOF-7S-ZoW2...

Atmel Programming Tutorial (by Chris Dahms ) https://www.youtube.com/playlist?list=PLoLaqVexEviMZu55Y4JO6...

Hope this helps.

Thanks; these are helpful but not quite what I meant. I was thinking of something that speaks in terms like “you have an existing Arduino project; how do you migrate off of Arduino and to gcc-avr?” In other words - explicitly build off of what you learn from Arduino, rather than starting from “scratch.” Maybe that’s harder, though.

Aah trying to bitbang with a Sleep? I agree, it's a case of "get off my lawn and go back to node.js" :)

Or, more accordingly, don't think the computer is magical.

Edit: it's a bit funny seeing someone trying to discover things by themselves but saying "it's difficult to measure pulse width with an oscilloscope" just triggered me - pun intended

Bitbanging like this is fine for synchronous protocols... but if you're trying to work with an async one like this one, there's just no safe way to do it in C unless your have a lot of leeway in your timing. Even if you get it to work today, there's no telling how a compiler update might change the instruction generation enough to change the timing and screw it up.

I know it's cliché to tell people to go code in assembler but... this is where you do it, and why you do it. And it's easy. Seriously. These things are so simple it is not hard at all to learn how to code for them in asm. And it opens up a whole world of cool timing hacks you just can't do in C.

I haven't done much microcontroller stuff recently, but here's a thing I wrote for an ATtiny a while ago. This kind of tight timing bit banging hack is just not possible to do reliably in C.

https://github.com/marcan/sigmafix

I would absolutely love to write assembly for the most time-sensitive parts, while keeping C for the parts that act as a USB HID library interfacing with the host computer. It really does feel like it would be so much simpler to be able to drop down into a little assembly sequence for reading back the response from the keyboard.

But - to your point about Arduino's gaps - I have just no idea whatsoever how to build something like that. Not so much how to write the assembly - I've written a bit, and I'm sure I could work that out. No, I mean literally how I link and assemble the final program and get it to work on the chip.

I don't know whether that's just an Arduino problem, either. I find this stuff hard to google, but maybe I'm using the wrong terms - these are the perils of learning as you go.

Heh, I wrote a USB stack for PICs in assembler way back and... yeah, I wouldn't recommend that :-) (the official stack was written in C, but I refused to use C compilers for PICs at the time because they were expensive and terrible; things got better with SDCC, but oh god C18 was bad - thankfully C works very nicely on AVRs).

But yes, mixing asm and C should be a lot easier and well documented. The repo I linked is a simple example of how to do it stand-alone, without any libs or Arduino anything, but... yeah.

This isn't exactly an Arduino problem; the "template project" issue is pretty common across the embedded industry. But for something as popular as Arduino not to have done a better job is unfortunate.

You'll figure it out, I'm sure! You can probably continue to piggy-back on the Arduino environment by using inline assembly (the asm("") statement), but it's annoying since you need to keep the assembly in a C++ string literal.

In the meantime, please read the manual or look up some tutorials on Youtube on how to use the oscilloscope. They're complicated for sure, but I think you had a popular hobbyist model and there should be help to be found. Failing that, look for the word "cursor" on the front panel; cursors is usually how you measure things and it's like at the core of scope functionality so it should not be hard.

> Bitbanging like this is fine for synchronous protocols

Yes, it might be (easier if the timings are more flexible), but a naive sleep (that might be trying to use an internal timer) will throw you off

That's exactly my point, they would need assembler for that (or at least "predictable" C instructions)

What frustrates me about arduino is how limited the peripherals are. I can see why they chose to begin with that obsolete processor - the arduino model of programming is simple & approachable precisely because the processor is so dumb (no complex interrupt controller, no DMA, no complex power gating and clocking setups).

For reliable embedded software I try and get the peripherals of the microcontroller to do all the timing critical work and my code sits up at a high level handling complete messages etc. Kind of the equivalent of "not blocking the event loop" in node, or functional vs imperative programming.

But the arduino libraries etc don't really fit well with this style, because they have to accommodate lowest common denominator chips like the ATMega. At least it's not PIC 8s I suppose...

For getting into embedded, I would strongly recommend the Teensy series (the 4.1 is joyful overkill for virtually any project). They are far more functional processors which are supported very well by the arduino ecosystem and have their own excellent set of libraries maintained by Paul which make full use of the hardware in the MCU. Being ARM based means that on chip debugging is relatively painless if it comes to that.

The beauty of using such a simple MCU is that it is very easy to learn how to program it directly...

... and yet Arduino isn't actually teaching people how to do that :(

I'm a staunch proponent of using up to date MCUs, which means ARM these days, but I'd be completely on board with still using an ATMega as the default in 2022 for this kind of educational purpose if the point were actually being educational.

At least it's not a PIC16F84, I suppose. All the microcontroller books spent a decade teaching literally the lowest end, crappiest, and more expensive PIC in the series because it was the first model with Flash memory, and ignoring everything that came after it. Even those that were backwards compatible for all intents and purposes, and not any harder to incrementally learn.

I like to think that there's an alternate universe where the entire industry adopted NeXT's idea of replacing the dedicated Caps Lock key with a Command+Shift key combination, because it's such a smart and obviously correct thing to do.

Sadly, I'll never live in that universe.

I understand it was quite popular in the MSX world.

That world does exist with programmable keyboards. Fn + Capslock = Capslock On / Off toggle. Capslock = Esc. Makes coding in VI / VIM quicker on the touch and lighter on the fingers. Just wish laptop manufacturers would include programmable keyboards so multi-boot allows retention instead of having to run a service to override the OS input device.

But then whenever you use somebody else's keyboard you're in a different world

On Windows I use AutoHotkey to swap Caps Lock and Escape which makes Vim much smoother. I also set it up so that it automatically disables itself whenever a remote desktop window has focus.

I kind of do - I have caps lock remapped to lcontrol and lcontrol remapped to lwin, so the leftmost column of my keyboard looks like ~, tab, lcontrol, lshift, lwin from top to bottom. If I need control+shift I just hit control with the flat of my pinky instead of the tip so it also catches shift, and if I need shift+command I hit shift with the flat of my pinky instead of the tip so it catches command too. It's not perfect, but it a lot easier on my wrist than other configurations I've attempted.

As others have mentioned, programmable keyboards will give you this — I have a keyboard powered by QMK, and I programmed it such that lshift + rshift = caps

After your post and looking at the keyboard I finally understand how mechanical keyboard enthusiasts can use winkeyless keyboards. You probably won’t need caps lock and it’s enough to have command and alt at the bottom.

Too bad windows and Linux don’t come with standard layouts that are more like Mac and probably next.

Sinclair Spectrums (and possibly the older ZX81s) used a keypress combination (CAPS SHIFT + SYMBOL SHIFT) as well. It's not invented by NeXT.

I have a custom keyboard from WASD that is mostly* the NeXT layout, and I use Karabiner (macOS) to program it to behave like one. It's wonderful.

*The keyboard in the post has | in the right place but all my NeXT keyboards had it in some wackass position over the ten-key. Ridiculous.

Chromebooks use the ALT+Search combination for caps lock instead of a dedicated key.

Nelson seems to have been using a Rigol benchtop scope and some proprietary software that came with his generic logic analyzer. But from long experience I'm wary of investing my time in proprietary vendor tools.

Does anyone have a good overview of the free-software signal-analyzer-software/oscilloscope-software landscape? I've played with sigrok a little bit, enough to get it to decode some PS/2 signals I captured on an Arduino. OpenHantek looks pretty great over in oscilloscope-land. What else is out there, and what's better or worse, or why?

Haven’t used them ever, but Red Pitaya[1] had a compelling pitch.

[1] https://redpitaya.com/

Thanks for the link.

Check out this DSO: https://www.tindie.com/products/earth_people_technology/open...

btw, some people tried to reverse engineer and rewrite the firmware for this Rigol https://www.eevblog.com/forum/projects/rigol-ds10xxz-firmwar...

You might try asking your question over at the eevblog forum. I have a Rigol scope and when I wanted to use a linux host I found a driver for it over there. So in terms of host side options there is quite a bit. I think this was the one I used[1]. For myself the cost and performance of the hardware usually outweighs anything else. I also have an analog discovery 2 (diligent) and it seems like they have some open source offering as well[2] (which I haven't tried). They do have a linux client for their host side which was good enough for me.

[1] https://github.com/wd5gnr/qrigol

[2] https://github.com/Digilent/waveforms-live

I've read nice things... a long time ago about the Bus Pirate wrt. protocol analysis/exploration (maybe not for high speeds): http://dangerousprototypes.com/docs/Bus_Pirate

Check out Sigrok: https://www.sigrok.org

https://sigrok.org/wiki/Kingst_LA2016 to be exact. LA1016 is supported, but LA1010 seems to be even more cut down version.

Indeed teardown reveals https://blog.lvu.kr/kingst-la1010-logic-analyzer/ is using older FPGA and no dedicated memory buffer. Compare to LA1016 https://www.cnx-software.com/2014/12/30/la1016-la2016-and-la...

> 100 m @ 3 channels, 50 m @ 6 channels, 32 m @ 9 channels, 16 m @ 16 channels

all <40MB/s, so its purely streaming LA like original Salea.

Edit: ah, so its further optimized https://sigrok.org/wiki/KingST_KQS3506-LA16100 direct Salea logic16 clone, everything makes sense.

I have a Siglent SDS1104X-E scope, it's a step up from the typical Rigol DS1* in my opinion in both use and features. I've also had the Rigol DS2x and Keysight DS3x series at work, and various Picoscope PC scopes.

Stuff I find better with benchtops: - display update rates (I've caught random glitches I would never have been on a PC scope) - sample buffers - display options (persistent, colouring etc) - segmented capture modes (Siglent call it "history) - protocol triggering, which is incredibly useful for working with UART/CAN/SPI etc. Keysight nailed this. - glitch triggering - just triggering in general - realtime online protocol decode - user interface (in terms of having knobs and buttons rather than clicking on stuff)

Basically, I use my scope a lot as a more powerful logic analyzer for signals using less than 4 channels. I do have a backup LA, and Sigrok/Pulseview are fantastic when required (and support a lot more protocols). But a logic analyzer is never going to show you if you've got signal integrity issues.

As to why there aren't any open source benchtop scopes, good point. At least my Siglent is Zynq based with Linux for the UI/rendering/networking and FPGA for the quicker bits. The Keysight's have custom ASICs.

Bunny's Novena could be a good start towards the hardware.

Never did electronics, so when I've read of logic analyzers in a book, it sounded like something between an oscilloscope and a complex gadget out of a story from Isaac Asimov. It's surprising they're so simple, cheap and effective.

Benchtop logic analyzers used to be more complicated. They are still technically superior with better probe termination, higher sample rates, and sophisticated hardware triggering.

The big down side is of course the limited sample memory. Many (especially the very popular, because cheap, fx2la) pc-based ones simply stream the data, which enables the use of all PC RAM and essentially infinite capture times.

Last time I tried a cheap ($8) saleae fx2la clone using pulseview/sigrok (open source combo for visualization, with open source fx2lafw), I didn't find a continuous option, but that might have been a GUI issue, thanks for the tip!

You can just set the sample count really high (up to 1 trillion). The samples are not compressed in RAM, so the maximal capture time is determined by the size. For 8 GiG and 24 MHz this means you can capture for ~6 minutes.

I tend to simply set the sample count higher than needed and stop the capture prematurely.

If you have a decent oscilloscope with logic/protocol analyzer features, you can trigger the analog scope side of the tool on data. This is a very powerful debugging tool for finding the physical cause of odd data. Yes, with a pc-based tool you can capture everything to disk and go back and search, but that usually works out to be kind of clumsy.

Tektronix introduced some nice LA units 20+ years ago that could be configured with analog mux outputs to pass probe channels over to a scope. The instruments could be cross triggered if you wanted to but it was fun to cycle through a whole address and data bus in real time and look for SI problems.

A while back Cypress made a programmable chip that had USB capability, programmable general purpose IO and DMA for efficient data transfer. This turned out to be just what as needed to make cheap logic analyzers for hobbyists.

https://www.eevblog.com/forum/testgear/logic-analyzer-that-i...

> A while back

Still in production, still incredibly useful.

Hey spenczar5, https://www.reddit.com/r/ReverseEngineering/ would like this a lot :)

Good idea, I've just posted it there :)

Had one of these with my original NeXT Cube that I bought new while in grad school.

Still one of my favorite keyboards ever. And the Cube itself was so far ahead of everything else out there at the time that it felt like pure magic.

> And the Cube itself was so far ahead of everything else out there at the time that it felt like pure magic.

Ah, the good old days, when I used to argue with my buddy that my SGI Indigo was more magical than his NeXTcube. It was even more fun than the old Mac/PC arguments. The Android/iOS fights today just don't have the same lustre.

Maybe because it had an undertone of friendly rivalry, in good old Unix hacker fashion?

I admittedly enjoy the passion shown on both sides of any divisive issue in this peculiar realm: vi vs Emacs, sysd vs sysv, xorg vs Wayland, etc.

>I don’t really understand all of the details, but from what I can gather, AM radios require a little clock component, and it’s desirable for this component to not be too close to any actively-used broadcast frequencies.

https://en.wikipedia.org/wiki/Intermediate_frequency#Uses

ADB? This means I can use an old Mac keyboard on NeXT, or vice-versa?

On models with ROM v74 or above. On NeXT hardware, you can also go the other way.

Pretty good thread: http://www.nextcomputers.org/forums/index.php?topic=3428.0

He says it is DIN of the "non-ADB" flavor, so probably not.

I used to use an Apple Extended Keyboard II on my NeXTStation Turbo Color. The later NeXT models had full ADB support. The early Cubes did not.

I also sometimes used an ADB NeXT keyboard with various Macs in the 1990s.

What a lovely tale! These are just the best. :)

That gigantic 8.5u space bar. Trying to hit Command using your thumbs must be painful. Compare with a modern US Mac keyboard at 5u.

No copy and paste keys? Seems like another badly designed keyboard.

It is for all intents and purposes a copy of the Mac layout. Which shouldn't be surprising given who founded NeXT.

It has them, actually; legends are printed on the front of the keycaps. Here's a picture: https://kuritakey.zone/next-non-adb-keyboard-aae/

Appreciate the negativity though!

You can always purchase it as an addon:

https://drop.com/buy/stack-overflow-the-key-macropad

off-topic: any chhance for an RSS feed?

This is a nice suggestion, and fairly flattering. Maybe I'll get around to adding one in the next week or so. My site is put together with a weird pile of goofy Python scripts and pandoc; generating an RSS file doesn't sound too hard.

Reply on news.ycombinator.com