Hacker News Reader

Comment by gouranga | Hacker News Reader

What a crock of shit:

after boot camp, all engineers get access to live DB

I can understand on a startup or small org but an organisation of that size, there should be very tight access control.

Despite what anyone says, the probability that someone does something bad increases in larger groups. Security should be on a simple need-to-know basis and nothing else.

I build BIG financial software and we have certain audit requirements, access control requirements, data protection requirements etc and that is exactly how it should be.

I'd never put my data near FB. They are simply irresponsible.

View on HN

You finance and billing guys will never understand.

These are rockstars.

They work for facebook.

They would never ever type UPDATE users SET email = username || '@facebook.com'; WHERE username == 'john.smith';

Of course not. Much faster to type

    UPDATE users SET email = username || '@facebook.com';

Then let the users sort out the exceptions.

I think he was being sarcastic - you can see the semicolon in the middle of the command which looks like pretty much what they did with the whole e-mail debacle. You simply removed the second incomplete statement.

I now realise that I am underqualified to work at Facebook. :(

>implying this was an engineering team decision

Right. FB doesn't get that all they have is the user experience, and it is bad user experience if I drop out of a conversation I'm having because their DBA-free database has decided to simply, silently drop my last comment. Or if it takes me several attempts to post a photo. Or if I invite someone to an event, and they never get the invite.

You can do all sorts of amazing things if you just don't care if your code actually works or not. Grown-up companies have full-time professional DBAs, and not just for separation of concerns.

> Grown-up companies have full-time professional DBAs

I don't know if I'd call them "professional," but they definitely have full-time DBAs. ;)

What do you mean? I've worked with some of our DBAs, and they're quite good. In fact, I can't think of a single site issue that was caused by a DBA. I work on fighting spam at FB, and we make use of mysql quite a bit.

But Facebook is not a financial institution, and I would put the responsability in the user if financial data is on the news feed.

One size (of engineering practices) fit all mentality is not the correct approach either.

You are correct - it is not a financial organsiation.

It does know who you are, what you eat, where you've been, who you're friends with, what you're interested in, what your political allegiances are, what your bowel movements are like etc.

A list of investments or your mortgage statement is way less important.

>I build BIG financial software and we have certain audit requirements, access control requirements, data protection requirements etc and that is exactly how it should be.

Funny, I had the same argument with a colleague today. Since FB is listed and their business model is based on customer data they should follow the same legal requirements as customer finance companies.

"they should follow the same legal requirements"

Perhaps. But they certainly do not have the restrictions of financial audit requirements and things like Dodd Frank etc. which is overwhelming all the major banks right now and is a big way to earn money for consultants if they know ABC of audit/compliance in 2012. Anyway, the point is that fb even though not ideal in how they store/use user data (which I personally am not a fan of as well), it does not matter as much as it matters to a bank/financial institution. I say this while I work for clients that are major banks.

Note that it doesn't say whether they all have write access, or whether all the data they have access to is encrypted in some way.

And any of those statements should be interpreted as one response he got from talking to lots of different people, and be subject to some amount of skepticism.

Either way, it simply doesn't matter.

It's a little irresponsible, but how else do you expect them to debug live code?

"how else do you expect them to debug live code"

I've actually spent quite a long time writing code where, for various legal reasons, you are never going to get direct access to the production data. (And indeed some cases where accessing the data in question would be a serious criminal offence).

Makes you really keen on defensive programming and comprehensive logging - although you do then have to be careful about what you log.

I actally found it kind of fun to try and debug complex systems based only on a log file and QA with production operations folks.

[NB Of course, none of this would be applicable to Facebook]

Thou who uses unit tests, UI tests, precondition, postcondition checks, heavy logging, knows the language and toolchain and knows arse from elbow NEVER debugs in production out of necessity or desire.

It just works.

In the last 5 years, we've never hooked a debugger to production instance or given access to a developer.

In that time, we've pushed approximately 912 billion SQL transactions through the system and 24 billion page hits (most of our stuff is backend non OLTP).

That's how it's done.

No please tell me, how did you learn to write unit test which cover all the crazy data configurations users are able to come with.

Boundary value analysis? Machine state transitions testing? Equivalence partitioning?

There are heaps of techniques one can use to identify which data points and combinations are useful to test. Some of these techniques even take a peek at your code to highlight the possible pain points.

LOL! You think someone who's drunk the unit testing kool-aid has even heard of any of that?

You have to start somewhere. If I have to choose between a world where people never test and one where they test but aren't very good at it yet, I'll definitely take the latter.

Of course, I'm still filled with rage when I find a cargo-cult suite of unit tests. But it's so much easier to convince those people to write smarter tests than it is to deal with the team that has no time to write tests because they're spending all their time debugging.

By being able to design software that doesn't need or allow crazy data configurations.

My whole mantra is don't manage complexity: avoid it.

You nailed it - thanks for the absolutely spot on comment.

Design is the key word here. It requires thought, it requires intelligence and it requires multiple people's input.

You're right.

Now that I think about it, I doubt Facebook is that careful, considering how often they break things.

You don't. Reproduce the issue on a development instance.

I first took TazeTSchnitzel's comment to be sarcasm, but now I'm not so sure.

(Poe's Law says hi.)

Not sarcasm. Of course you should reproduce in a development environment, but for some issues, looking at live DB is the only way to see what is wrong.

A system I work on has grown a lot over the last few years (data for 2000 users is now data for 50,000, rules have been added, etc).

Example story: We had an issue that was only coming up on production. I could not reproduce it in a dev environment. Worse, we didn't even notice it for a long time because it was a nightly job, and people were not reporting an absence of their notifications. (nightly job to email reports).

Finally got a dump of live data to dev system (it's a lot of data, so I don't pull it all the time). Someone had updated their email address to something invalid, and the system threw an uncaught exception in the middle of the loop (yay java). So... half the people were getting their stuff, the second half didn't - guilty record was a user with last name starting with L.

Yes, we should have prevented a bad email from going in with validation, but it slipped my testing (and the client's). It's just shocking to me today that professional people whose job it is to send and receive email can mistype their email. "john smith @yaho" is not valid, yes I should catch that, but someone typed that in. Adding on top of that was my own dynamic language background not mapping well to the JVM - one bad address in the middle of a loop doesn't just get skipped and logged, but the entire process now stops.

Multiple lessons learned from that one certainly (logging, validation, exception handling, etc) but... it would have been a lot longer for me to even consider putting in an invalid email address (it worked in dev - it was working for end users, etc) - pulling live data was the only thing that made it apparent.

I agree totally with your broad point, but wanted to respond to the "I can't believe a user typed..."

Even if your 50,000 people are all pretty smart, you're well out into the range of exceptional circumstances. If I do something once a day my whole life, that's still only 25,000 times. It's pretty easy to imagine somebody on their worst day doing something like that. E.g., You go to visit family, so you're jet lagged. You had a couple of late beers with your brother, so you're hung over. His baby is screaming with colic and the toddler is banging pots together. You're VPNing in from their kitchen table, trying to fill out some form, and in the middle somebody knocks over a glass of juice. You yank your laptop up, help clean up the mess, and then sit back down to finish. While the pot-banging proceeds at full volume.

Madness, sure, but common madness.

Adding on top of that was my own dynamic language background not mapping well to the JVM - one bad address in the middle of a loop doesn't just get skipped and logged, but the entire process now stops.

Makes me wonder what dynamic language you're used to. An uncaught exception in both python and ruby aborts the program, not just the tiny inner loop that happens to be running.

Primarily PHP, although I seem to remember classic ASP VBScript not dying if you passed a bad email to CDO (but that's been a long time, so i might not remember that correctly).

PHP would not throw an exception if the mail function got a bad email address. well, using some mailing libraries might, but the core mailing function (basically just a wrapper for sendmail/postfix CLI stuff) wouldn't.

Oh, you expected silent failure from a crappy library. :) I think Java (or rather, the library you're now using) made a much better decision to not blunder along.

"on error resume next" redux?

> Adding on top of that was my own dynamic language background not mapping well to the JVM

There's your problem...

It's a problem, but it certainly wasn't my only - groovy, to be precise.

How do you do that, if issue is caused by some corner case with user data you did not foresee and hence not covered?

For some applications, the number of corner cases that can not be resolved by reading log files is minimal. (you are writing logs, right?) That is, too, assuming the problem is actually in the code itself and not the infrastructure.

Reply on news.ycombinator.com