Hacker News Reader

Comment by 1letterunixname | Hacker News Reader

Disclaimer: I'm an engineer at Meta AMA.

I don't know what anyone else in the non-MPK/-FRE buildings here is doing, but 95% chance it's not engineering. Mostly, it seems like most workers in the building chit-chat and have meetings rather than do anything resembling actual work that's building product going in front of users. Country-club, cult-of-toxic-positivity lots of relaxing going on, in my view.

In my view, you're either a hustler, a hacker, or a manager of such to be essential to a tech business. You don't need any strategic hiring coordinating coordinators for higher-education. Also, the spend on R&D needs to be more focused on productization. Finally, I'm still wondering why there's no consultantish enterprise sales arm monetizing and discovering opportunities for lucrative SaaS products like Workplace.

View on HN

> In my view, you're either a hustler, a hacker, or a manager of such to be essential to a tech business. You don't need any strategic hiring coordinating coordinators for higher-education.

I think it’s somewhat telling that you didn’t list “engineer” here.

Meta isn’t a quirky startup, and hasn’t been for a decade: it’s one of the largest corporations (by market value) on the planet. By all rights, there ought to be a lot of people performing boring, turn-the-crank work that doesn’t immediately surface to you.

A hacker is an engineer.

A hacker is a kind of engineer. Large companies require all sorts; expecting all of your engineers to be “hackers” is the kind of prima-donna-mentality-encouraging behavior that I would expect Meta to have grown out of by now.

To that point: there is a lot of boring, eye-rolling -- but totally essential -- things like requirements management that are important for engineering. "move fast and break things" makes sense when you're small and agile, but when you have a massive platform that many other individuals and business depend on you just can't do that.

Sometimes I need a wizard with multi-domain knowledge to whip up a POC system, but most of the time I need someone to ensure all of the pieces work, have specs that meet reqs, and won't cause an outage when we deploy them.

The person was just using the term "hacker" to mean "any kind of engineer".

They say otherwise directly below.

> A hacker is a kind of engineer.

I've always found the term "hacker" pretty eye-rolly.

We’re literally on hacker news right now

I think they _should_ have outgrown it by now, but I'm not sure whether I'd honestly expect it

Admittedly they seem to be overlapping circles; but I don't believe they actually are.

Only if they have a degree allowing themselves to call that.

Software engineer isn't a protected title in the US.

My cat can legally be a software engineer - no degrees required.

Unfortunely, it only devalues the qualities that the word actually means in practice.

Honestly probably wouldn't even be in the bottom 10% skill-wise, either.

Exactly. A subset of engineers who deliver much and fast.

Also at Meta. The software stack is a shit show - xplat debugging in Vscode can’t even print the variables properly. This hacker culture is good at shipping shitty stuff and getting your bonus/promo and moving on.

Snide putdowns of other groups of people just convince me your opinions are not worth listening to.

Much, fast, maintainable: pick two.

haven't read the hacker manifesto yet?

Isn't a lot of this silliness due to them feeling like they needed to "poach" talented developers, so that the competion wouldn't get them?

That, of course, is absurd on the surface. Hiring people for the sake of hiring ... so they can stand around the watercooler discussing the weather.

That led to double or triple the headcount! 85,000+ employees to work on Instagram, Facebook, and pet projects.

That number alone boggles my mind. I work with three other senior developers and we run an entire phone company with five-nines uptime. Yes, three.

And now suddenly everyone is shocked about layoffs!

4 engineers for an entire phone company sounds scary. I’m sure your engineering is robust enough such that outages are minimal, but that still sounds like a lot of on call (rotation of 4 = once a month?). Even if you only get paged once every few months, you still need to worry about getting paged until your shift is over! Even if you don’t worry in the psychological sense, you still have to schedule around it.

I actually screwed that up, we only have 3 total software engineers. Including myself.

We do have other employees who maintain the hardware, on-call DBAs to manage issues, etc. I'm only speaking to the software engineers.

And we have lots of hardware and lots of open-source solutions that handle the actual calls.

I'm full-stack but lead on the front-end, a complex Angular application to manage everything from huge call centers to small restaurants.

We have C# for APIs, cloud Oracle for the database, and a whole slew of other software and services to manage the actual calls.

Each of us is specialized in specific parts. Our up-time is tremendous given the amount of code we've written. It's extremely stable.

I've been at this 20+ years, as have the other 2. We know enough between us to get this done.

I've never been called after normal work hours. We release the updated front-end every week and haven't had any issues. And a lot of changes/improvements go into that ... that's a lot of my job.

It's well-architectured, well-tested, fault-tolerant software.

Sorry, but I'm skeptical that you actually run "an entire phone company" in the way that most people understand the term. You mention that you use lots of open-source solutions, and I'm guessing you outsource the build and operation of the network to a real phone company, probably similar to an MVNO. Am I wrong?

Meta is very different from that, they build the products that users interact with, but also build things at the bottom of the tech stack. At their scale, this makes business sense to do so, and comparing your headcount with theirs makes no sense.

I don't really understand why software engineers keep dunking on each other like this. I get that people want to broadcast how smart they are, but in reality we're just giving the general public a warped sense of how much work is actually involved in building large-scale software systems.

I’m also skeptical any time someone mentions 5 nines uptime at a small scale. For one, it takes a lot of engineering to be able to actually monitor and detect that with precision (that is, how do you know you are 99.999 and not 99.995?) and with so few people there may be holes in what’s monitored (there are so many places you can drop requests/whatever and lose availability). There’s also tail risks like datacenter incidents (if your servers are on three racks in two data centers) or dependencies like power outages that you may be getting lucky on avoiding due to small scale, rather than amortizing over a huge fleet - that is to say, if there is a 1% risk per year that one of your racks goes down and takes you 3 9s when that happens, you are really at slightly under 4 9s, but with only a few racks it doesn’t happen most years.

That last one is I suspect what makes it so small scale operators can achieve “5 9s” with a fraction of the engineering of larger operators. You can get a lot of 9s most years because you dodge infrequent risks.

> I’m also skeptical any time someone mentions 5 nines uptime at a small scale. For one, it takes a lot of engineering to be able to actually monitor and detect that with precision (that is, how do you know you are 99.999 and not 99.995?)

We have that in place. We run phone systems that businesses depend on, and we have SLAs that guarantee this uptime in order to secure customers. We have network engineers dedicated to everything from guaranteeing it on the cloud side to checking Wireshark traces for any hint of abnormalities, every day. When I say 3 people, I mean those of us writing the front-end, back-end, database procs, and code that the open-source libraries require, including forking and custom patches. We have other team members ensuring our HA pairs, load balancing, redundancy, fail-overs, and all the other associated technology is working as expected.

I won't get into the details, but we have not violated our SLAs, ever.

And you'd be surprised at what open-source software we are using to drive parts of this system. Kudos to them, they are helping us maintain this with some rock-solid software.

I think you cannot really salvage this argument. The way you describe it makes it sound to me that your company's success is ore likely due to luck and not just competence. It's also not clear if you really think this is a model that can scale to the size of Meta or if you just wanted to slide in a humblebrag.

> It's also not clear if you really think this is a model that can scale to the size of Meta or if you just wanted to slide in a humblebrag.

It can't. It depends on "software services" hosted and coded by "other software companies" so his infra's SLA is basically outsourced to either Amazon, Microsoft or Oracle.

> It depends on "software services" hosted and coded by "other software companies" so his infra's SLA is basically outsourced to either Amazon, Microsoft or Oracle.

It also depends on an incredible amount of in-house custom code.

If I deploy a change that breaks our Angular front-end, or a C# API change that has a typo that routes calls to the wrong places, or a configuration file for our open-source software that handles the phone systems, how exactly do we tell our thousands of customers they can't run their call centers? Or our restaurant customers can't take orders because their phone systems are dead?

Let's not be ridiculous. I'm not humble-bragging. I'm telling you what I do at my job.

Arent most peoples at this point?

> I don't really understand why software engineers keep dunking on each other like this.

I'm not dunking on anyone, I'm explaining my day-to-day job with a very small number of senior engineers.

We run a highly complex system. There's no way we could handle millions of calls with five-nines otherwise.

I won't get further into the details because I don't want to reveal too much PII.

I fully understand how much "behind the scenes" work goes on at a place like Facebook. I'm not sitting here imagining rooms of graphic designers thinking what the CSS button radius should be (although I'm sure with 85,000 employees, those happen also).

But note that Musk walked into Twitter, fired en-masse, and it still seems the same to me.

Yes, there are senior engineers who keep the core functionality working.

That is still far less than 85,000 employees.

In our case, it's 3 of us handling all software development. And we write a lot of mission critical code.

> I've been at this 20+ years, as have the other 2

> We release the updated front-end every week and haven't had any issues. And a lot of changes/improvements go into it

In 20 years zero issues? I don’t believe you.

> Even if you only get paged once every few months, you still need to worry about getting paged until your shift is over!

I'm not sure what you mean. A page every few months will be considered world-class achievement in a FAANG-like company. Take Amazon for instance, the oncall is brutal and getting several pagers per day is normal. Other companies may be better, but not one pager per few months better.

When you're on-call, you need to be prepared for a page. That means you can't get drunk with your friends, or get on a plane, etc. Being on-call means being ready to answer a page, whether it comes or not.

> A page every few months will be considered world-class achievement in a FAANG-like company.

Except maybe it's a different beast entirely. At amazon, they're constantly pushing new features at most teams. A stable phone company may just be handling pages for when hardware fails. Presumably there's bugs in fast-moving new code more frequently than hardware failure of a tiny org.

Also, fwiw I've been at amazon and had on-call rotations where we didn't get paged monthly. Your manager/team isn't allowing you to allocate resources to fixing your alarms or bugs if you're getting paged that often and not a crazy critical service.

I think all those big corps do a lot of change for the sake of change.

That's why we have so much churn in frameworks, etc.

If you're not "innovating" somewhere, you're not good enough to work there.

Traditional companies are more long term and stable with their tooling and decision making.

You started with an absurd conspiracy theory and just ran with it.

> Isn't a lot of this silliness due to them feeling like they needed to "poach" talented developers, so that the competion wouldn't get them?

No! No one does this!

Oh, come on. Yes they do.

> Is Apple throwing enough money at key engineers to prevent defection to Meta?

https://www.idownloadblog.com/2021/12/29/apple-meta-poaching...

Paying enough to keep key engineers is not the same as hiring new ones they don't need

I love how people assume that a poorly written and tangentially related article, with a question mark in the title nonetheless, becomes the official proof of <wild speculation that bigcorp is doing X>.

Wait until we see the flood of ChatGPT-generated articles. Oh, wait...

Are you not familiar with the world-class investigative tech journalism coming out of iDownloadBlog.com? I mean the guy lying about his day job posted it so clearly it's a very legitimate and prestigious publication.

Supposedly, there is software that uses "internet-based signals" behavioral predictions to identify which key technical people are being courted and/or are considering moving to a new gig.

Poaching is a big part of the game. You can't get a raise compared to what you can playing the game.

“Hey this guy just updated his linked-in for the first time in two years!”

> Isn't a lot of this silliness due to them feeling like they needed to "poach" talented developers, so that the competion wouldn't get them

People say this from time to time, but have we ever got evidence from any company? A director or VP gave such mandates, a leaked email that discussed such strategy?

No one ever seems to explain how the companies are coordinating to suck up talent. Otherwise, there would be a free rider problem. Why not let Google suck it up instead?

With three people there are limits with what you can build and how fast you can build it. At large tech companies there is a gigantic amount of scope. People will make up scope to justify their job, or to justify why they need to grow their teams to accomplish goals. If a company wants to invest into doing everything over prioriting what should be worked on then headcount will explode.

Perhaps one way to view hiring (at the scale of big tech) is a bit like a VC, you hire 100 people knowing 99 of them won't do anything special, but that 1/100 will make it worth it.

And like VCs the math makes more sense when money is cheap and there's a bull market.

An SAP consultant once told me that they had a term that can roughly translate to "make a crowd" (sorry, no idea about the actual phrase used by consultants in English speaking countries, if there is one). It basically means that whenever a big customer have an issue, they send more people than actually needed, and one senior consultant works on solving the problem while others just walk around and look busy. It gives the impression that they are taking the issue seriously.

'Tis true with Incident Mgmt and some highly sensitive projects, too. Nothing SAP related at all, but gives the appearance to PMs and Ops leaderships that we're going extra hard on the issue.

And to be fair, it's useful for incident mgmt since you don't know what you'll find, and will often have lots of people up your ass; having supernumerary bodies means I can throw them at something if I find an artifact or concern, and it also gives me a distraction-shield: I communicate only to Senior Engineer X, and all inquiries from Mgmt and Customers go to X; I only have to update X instead of 10+ irritated users/owners.

I’ve seen a version of this in sales where you have a welcoming call with the new client with the entire service team acting all cheery. Leaves the impression that their spend gets all those people when in reality it does not.

this would explain the cost surrounding SAP :)

In engineering, it's much closer to 2 than 100.

To be more precise, to e.

The public already knows: the headcount graph trended upwards and accelerated in recent years until recently.

I joke with people in another department that we're a poaching ground for TikTok. Mostly, I work with people who have been at the company many years and are senior production engineers, so I can't appreciate dotcom-style spending in non-engineering business units. For the same space, Alphabet has 10x headcount compared to us.

Is this actually true though?

The data seems to suggest that for most companies the trend was constant, and they just reached a point where it was no longer economical, which was exacerbated by hiring freezes halting attrition.

https://stratechery.com/2023/tech-layoffs-big-techs-hiring-r...

> Isn't a lot of this silliness due to them feeling like they needed to "poach" talented developers, so that the competion wouldn't get them?

Rumor.

A team of 3 star engineers can take on the world. It's a killer setup.

> I don't know what anyone else in the non-MPK/-FRE buildings here is doing, but 95% chance it's not engineering.

Are you saying there aren't engineers working outside of MPK/FRE (which is wrong) or that they're lazy (which is also wrong)?

Yeah I laughed at this. Whether or not you like the business model, Quest and AI are in Burlingame and is obviously very hard groundbreaking work.

A lot of engineers working hard up in the PNW offices as well.

Woah, backup and chill. :)

Edit: I guess you don't work at Meta because there's "the assume the innocent take" thing. ;) I merely had the audacity to state a fact that the building where I work demonstrably contains few engineers because I sampled titles in the seating app.

Facebook felt culturally incapable of building good enterprise software products. Which is weird for a company that makes 90%+ of its revenue B2B.

I’m not sure it’s a bad thing, though. I’ve never seen an engineering-first enterprise software group. The business is just so sales-and-relationship driven.

Look at Amazon: you can plot the enshittification of their retail business with the rise of AWS. That’s not coincidence, it’s culture.

There's 3 parts to the problem:

0. Lack of an enterprise sales org (think how VMware started or EDS or Perot Systems). I know an ex-Oracle sales guy who's itching to be an enterprise sales manager at Meta.

1. Lack of thoroughness in solving bugs and polish. Workplace video conferencing is damn good. Workplace the web and mobile apps have inconsistent and troublesome comment formatting bugs. I've sent task after task but they never get fixed. Basic things that should work don't.

2. Impact creates a cult that rewards new features while punishing better engineering and fixing things.

(My neighbor works for AWS high up. They treat most of their employees with borderline abuse.)

Having been on the other end of that B2B relationship, it's not surprising at all. FB treats their paying clients (advertisers) like dog shit and so far they've only gotten away with it because of their dominant position in the social media advertising market. Trying to break into an established market with their current attitude is not gonna work.

Oh really? Damn. Like an independent retailer or megacorp?

Are there any specifics to DM me worth running up the flag pole?

Once your spend is high enough to get an account rep things get ooookaay. But in general the ad tools are bloated, buggy and loaded with dark patterns. And have been from the beginning.

Most serious advertisers and agencies use expensive third party tools that manage your ads using the advertising APIs.

I don’t think it’s worth “running up the flagpole”.

Agencies yes, most serious advertisers probably not.

But even the b2b parts of Facebook are run like a b2ç business, because that's the culture.

Ha haaaa. I'm sure mega-corps get a much better treatment.

They had Workplace, which was a fantastic product that got shelved externally when they couldn't get a big payout early in the project.

It was the perfect Slack replacement for companies with over ~1000 employees, and I wish my current company was using it.

I worked for a company that had both Slack and Workplace and... Workplace wasn't a Slack replacement at all? It felt like a separate instance of Facebook for our company. How would it act like a Slack replacement?

WP Chats could be vaguely similar to Slack -- there can be chats associated with a group, which is sort of like Slack named channels. But I agree it's not a perfect replacement.

I think people immediately cringe commingling their work and social lives. Personally I think it's a huge over-reach that Meta requires their employees use their personal account as their work one too, but maybe that's not true anymore?

As far as I can recall, I did not use my personal FB account while working at Meta last year. Some tiny detail of it may have been imported somewhere, but I definitely had a totally different account for Workplace.

Contingent workers, that's possible.

FTEs generally require an FB account to start the tech device provisioning process.

i don't think that's true anymore after project meiosis

There was recently a big memo about that. I have completely separate work and personal devices. (This was written on a personal device.)

The only thing that goes in from personal -> work is music or random home office photos.

The only things that go out from work -> personal are public résumé-level performance reviews, tax, and benefits information.

Exceptions on work devices include Github and streaming services, but otherwise not much else should be co-mingled. Everything else stays in its own domain.

(I don't use personal social media accounts so I'm the wrong target audience. Personal FB and Insta accounts are linked to work, and Github if you have one.)

I'm always a little shocked when I see someone who isn't keeping personal and work devices (and accounts) very intentionally separate.

Are the accounts tied together though? Facebook still knows behind the scenes those are related accounts and that bothers me.

They don't have to be anymore (as of fairly recently).

They used to require linking a "personal" FB account as your employee account but recently (as in, like, 2021 or later) have a system that no longer requires this (including FTEs). Note that you could always create a throwaway personal FB account to be your employee account (and this is what I did).

ads is totally enterprise product

I’m not trying to be aggressive, but this is the kind of “just world hypothesis” reasoning you hear from people caught in layoffs.

Sometimes the people doing that kind of reasoning are hit next.

I’m not saying there’s no waste, but this is classic rationalizing.

> cult-of-toxic-positivity

I would love to hear details on a handful of instances of how this manifests. Toxic positivity, I don't know if I am to read that as like burying ones head in the sand, ears plugged singing la la la, ignorance is bliss or what not. What exact behavior would aggregate into a phrase like that? Very fascinating characterization of something interesting I'm sure.

>>> Toxic positivity is the pressure to only display positive emotions, suppressing any negative emotions, feelings, reactions, or experiences. It invalidates human experience and can lead to trauma, isolation, and unhealthy coping mechanisms. https://www.betterup.com/blog/toxic-positivity

Acting like EVERYTHING that ANYONE does is AMAZING with zero accountability and retrospection on if that actually made any difference.

Yes, exactly. It's bad for both mental health and productivity because it denies reality (promotes dishonesty) and puts distance between people with phony, exhausting expectations. Being exceptionally good and real is the real thing. ;)

Life is so short, so why not be legit?

It also gaslights people into believing their negative emotions are their own fault, if everyone else pretends to be positive.

I actually wonder if you're an engineer at Meta because this isn't my experience at all. My whole team (in London) is working hard doing useful engineering work. The evaluation process is such that employees are pressured to deliver. I haven't seen anyone "relaxing".

To me, the problem with Meta is the constant focus on impact. There's little incentive to write robust software. Many tools are half-finished, fragile, poorly documented.

FWIW they did have sales people trying to sell Workplace by FB but it just didn't go so well. Things like Microsoft Teams is so well entrenched (and "free" with the CIOs' Office 365 subscription) that there wasn't much to come for. Even Slack was (/is?) struggling..

The bundling bs sucks. The only reason MS has that many people using Teams is because it’s bundled (free) with the M365. It’s hot garbage.

I hate all those real time chat apps anyways.

Internally, everything and everyone uses Workplace. There is no email except for rare things. It's basically the same thing used by customers.

That's interesting. My team (not MPK) is completely overwhelmed! The entire team is hard working and not much relaxing unfortunately (could use some of that!)

Why doesn't Instagram have a native iPad app?

It seems like an immensely profitable project that would take a relatively small amount of resources.

iPad users likely have a GDP bigger than France, I don't know a single millionaire who doesn't have one... yet their ability to click ads is very restricted by the mini/stretched screen.

How does the culture allow so many non-core projects to get priority over this?

I'm biased (since I work at Slack) but, as a casual user, Workplace seems like an un-loved afterthought.

I don't think Meta is well-configured to make consumer-facing products that "surprise and delight", and Workplace's slightly janky interface is a clear symptom.

And I have to use Workplace everyday. :D

There's jank that can be fixed.

The problem with Slack is it creates an instant, continuous distraction noise barrage worse than email groups. We at least have a cultural norm of not DMing individuals (unless it's a SEV) and seeking info in group posts or maybe group chat.

The problem with Workplace is that it looks much like Facebook, the UI/UX is not conducive to reading long-form posts, it's also a feed of posts organised in hard-to-find groups, there's no focus to guide me through a journey to actually enjoy using the product. Interaction with posts is an afterthought, threaded discussions do not exist, it's just a "dump" of information that seems to work for whatever use-case Meta had internally but was never translated to other B2B needs.

On the other hand, Slack is a simple communication tool where I can asynchronously coordinate with whatever team I need to get in touch with, it's exactly what I had asked for years at companies I worked for in decades prior to Slack: a well made IRC-like interface. It just works, with all its flaws and performance issues, it's still valuable. Workplace on the other hand is not valuable at all, it could disappear from the company stack of tools and I would never bat an eye. And I work for a global company, much smaller in size than Meta but definitely one that you either heard about or use our products.

I’ve been at FB for many years and ~1 year ago moved somewhere where we use a mix of Slack/Google docs/confluence.

Perhaps it’s just because I got used to it, but I do miss Workplace a lot - mostly for allowing me to easily stay on top of what’s interesting to me.

* Workplace made it really easy to be on top of what’s happening in the company, and in what it thinks you’ll be interested in. The algorithmic feed was super helpful.

* I was able to follow along with what’s happening overall in the company and all the interesting to me info just with a few minutes at beginning of each day scrolling my Workplace feed. Now, with Slack / gdocs / emails, I would need to read all updates in hundreds of different Slack channels and figure out myself what’s relevant.

* Google Docs - super hard to find anything, or even know if it exists since everyone applies restrictive sharing options. With Workplace, I would write a post in the relevant group - easily searchable by everyone (and discoverable by being ranked on Workplace).

Same, Workplace is sorely missed.

Slack is good at synchronous communication, it’s a substitute for Workplace Chat.

Where I’m at now there is nothing similar to Workplace, which is great at asynchronous communication. On Slack content scrolls up the page and loses its engagement quickly. Also if someone does make a post that gets super high engagement in a channel I don’t regularly check, nothing will surface that post to me.

We generally have that cultural norm too — I'm as likely to DM someone as I am to sidle up to them in the office.

I never feel like using Slack is stopping me from writing code, but when I'm waiting for a build or what-have-you I can look at channel updates asynchronously.

> The problem with Slack is it creates an instant, continuous distraction noise barrage worse than email groups.

That's why I wrote localslackirc. To make having to be on slack bearable.

Workplace works reasonably well in my experience. It's basically Facebook. What do you find janky about it? My biggest complaints are around some of the weird defaults for video streams.

(My biggest gripe with Slack is how painful it makes replying to specific comments in the original channel. Neither Workplace chat nor Discord have this problem.)

https://www.washingtonpost.com/technology/2023/04/18/meta-la...

> “This will be a difficult time as we say goodbye to friends and colleagues who have contributed so much to Meta,” Goler wrote.

> The company is expected to lay off what will likely be thousands of highly skilled employees — such as engineers and other technical staffers — who help build the company’s products, according to people familiar with the matter.

I wouldn't be so certain

>You don't need any strategic hiring coordinating coordinators for higher-education.

Actually, you do!

Please help out us outsiders. My guesses are: MPK = Menlo Park? FRE = Fremont?

What does your team do exactly? Are you hiring or is there a hiring freeze for all teams?

Hiring freeze everywhere*.

* Supposedly not hiring freeze for senior ICs.

Elon, stop it with the throwaways.

Reply on news.ycombinator.com