* Guy sees service running on 32764 and thinks that's quite odd
* Gets very frustrated at useless forum posts from people with no clue
* Downloads a copy of the firmware from the (horrible) modem-help.co.uk
This is where it gets interesting:
Binwalk ------
First off, a program called Binwalk is used. This is a combo of a really cool python script, the libmagic database (if you've used the file utility in 'nix you've used it) and some C to deal with some compression types.
When binwalk works, it's pure brilliance. As you can see in the screenshot on slide 12, binwalk is able to detect strings and filesystems inside the firmware.
References to Texas Instruments and Telogy Networks would be a good starting point if you wanted to google for datasheets or some more background information
I haven't ever seen a reference to Igor Pavlov in a firmware binary before but his name popping up is a good indication that you might be hitting 7zip or LZMA compressed data - http://en.wikipedia.org/wiki/Igor_Pavlov_%28programmer%29
The filesystem -----
Squashfs is commonly used (and horribly broken) by manufacturers of modems and routers.
Luckily the author is able to extract the filesystem out by looking at the offsets printed out by binwalk - this doesn't always work!
Slide 13 is pretty typical when reversing modem firmware - someone thought they were being smart and broke the squashfs format. It looks like no one else will be able to replicate this as the source code for the modified format has been taken down (?) It's possible the author is using the name of the folder in the last screenshot of slide 13 as a hint of what to do next, slide 14 and 15 are magic to me.
I think the author has changed the LZMA sourcecode to look for LZMA compressed data with a gzip header but this seems really strange to me
Slide 15 just shows the unsquashfs tool with what seems like sensible output (if you'd done something wrong at this point you might see huge numbers of inodes or a silly number of files and you'd have to go back to the compression and keep looking)
Now the author has a filesystem to look through - I would have used strings and grep instead of just grep at this point. He's grepping for the name of the service he found in the forum posts, I think.
From here he's used a very expensive program called IDA Pro (https://www.hex-rays.com/products/ida/support/orderforms/nam...) to help. You may also be able to use the objdump command or The Online Disassembler.
I have no knowledge of MIPS assembly so I'm unable to help you there - IDA provides some flow diagrams like you can see in slide 19. I'm not sure if the author annotated the fucntion names or if IDA was able to find them.
He identifies a buffer overflow and writes a small script to exploit it in slide 20 (I'd love some extra explanation of this if anyone can give it)
When the script is run it seems to dump the current configuration of the modem, but then it crashes or resets (slide 21)
The comments on slide 23 suggest that the code has jumped to the "restore_default" function - sorry, I don't understand this part. References to nvram would suggest it's going something to the config though, which would explain slide 21.
Slide 26 shows an (hopefully LAN-side only) exploit that's able to enable the HTTP management interface of the modem and reset the password at the same time.
Final edit: I'm sorry for the formatting of this post. I'd love to collaborate with someone on a blog post about this exploit. I'm really just getting started with firmware reversing (and most of the time I don't get past the binwalk stage). My contact details are in my profile.