Hacker News Reader

Comment by gregpilling | Hacker News Reader

I am surprised that there is no automated alert to tell the webmaster that his code has changed on his website. Especially on the payments page!

With 50,000,000 users a month, surely they have a whole team working on checkout, all the time?

View on HN

Code is supposed to change on the website all the time, though, when they run a deploy. Surely they do have a team working on checkout, but it isn't obvious how this would be detected. The article leaves out how they got the malicious javascript onto the page in the first place, though, so it's hard to say.

Do you have such automated alerts set up yourself? Do you know anyone with such alerts set up?

Whenever someone complains about another company's product, code, features, security....I always wish it was mandatory to include a link to the kind of software the poster is putting into production.

I can dream.

Glass houses and all that.

I have. I deploy in ASP.NET and get a hash of the uploaded DLL. I check it twice a day. Never had any incidents to this day but as the saying goes, it's better to be safe than sorry.

And how does the hash of the DLL on some server relate to the content seen by users?

I would assume the DLL is responsible for the output of the content seen by users.

In ASP.NET pretty much all output, including html and js is included in the DLL. Only external js files are left out. Sure they could hack them which means my solution isn't bulletproof but I could also produce a hash for them too.

I do.

I have a tiny $5 Onion Omega2 on an independent cellular connection that checks file integrity on the production web servers every 15 minutes.

If the content of any of the files change, I get an e-mail.

If the alerts start coming in when I know I've just pushed a new version to production, the mail has a link that I can click that will re-scan all of the files and build new checksums.

If the alerts start coming in in the middle of the night, then I know something is up.

Obviously, this only works in small environments like mine where I'm the only one capable of updating the production servers. But it managed to catch a backdoor left in by the previous developer, who for some reason stored and updated his resume on the production server.

Ingenious, Have you considered a blog post on this approach?

It would be interesting to deploy a few of them in different places and check that they all see the same as well maybe.

Also did you do this as a belts and braces thing or is the system you are auditing particularly high security/risk in some way?

You can just set up Tripwire to do this sort of thing. It's in most distro package managers.

https://github.com/Tripwire/tripwire-open-source/#open-sourc...

What that won't do is save you from malicious code inserted into 3rd party content (script libraries, etc.) that you load from a CDN. If you're worried about that, you should make a copy of a verified version and serve it yourself.

You can just set up Tripwire to do this sort of thing

I wanted something that was completely independent of the machine. Separate box, separate network, separate architecture, etc...

I don't CDN on work projects. It's not worth the risk. If something goes wrong, I'd rather it be my fault and something I can understand and fix, whenever possible. Farming stuff out just leads to layers of things that can break, be compromised, or simply go wrong.

Again, it works at my scale (about 15 sites). It won't work for everyone.

It would be interesting to deploy a few of them in different places and check that they all see the same as well maybe.

That's a great idea. And since they're only $5 each (I think I spent $15 with the power shield), it's not a big deal.

Also did you do this as a belts and braces thing or is the system you are auditing particularly high security/risk in some way?

When I got here, most of the web sites were riddled with worms and trojans and spambots other bad stuff. One by one I just nuked them and started over. This was a deliberate isolation to keep an eye on things in case the sites or the dev machines ever got compromised.

What if a developer's machine was somehow compromised and the bad version of the file was put into the actual repo or deploy? I have no idea how likely this is or if it is a possibility in NewEgg's environment but that would be a case where only scanning for changes on the servers wouldn't catch.

What if a developer's machine was somehow compromised and the bad version of the file was put into the actual repo or deploy?

In a complex environment, that's a complex problem. In mine, it's not a big problem. Keeping the security routine on an external device with no other function I think helps. And since the device is on a completely different network, and a cellular connection with changing IP addresses, if someone was targeting the company they'd never find it.

That's the theory, anyway. So far, so good!

Reply on news.ycombinator.com