Confessions of an Unreal Engine 4 engineering firefighter
allarsblog.com
allarsblog.com
I consider the case, which the author describes, where the consultant actually talks to engineers, a happy day scenario. I was witness to a fast growing startup hiring a well known global consulting firm to help them set up company structure and processes. The startup was unable to do this on its own because of lack of leadership and unwillingness of the owners to empower their own people. The consulting firm sent over a couple of their junior consultants who consulted their 3-ring binder of best practices for the nearest matching industry category and gave their recommendations, which were followed quite strictly by a newly hired junior management. All without talking to a single engineer.
More often than not, employees are considered a necessary means to an end and treated as such, with the least amount of reward and care for which they are willing to work. Their sense of professional honor, responsibility and guilt is often exploited in an attempt to achieve higher output. Does the company's products or services suffer because of this? Of course, but nobody at the top cares as long as there is a metric that tells a good story. And when not, the lowest ranks are the first to be pruned and restructured.
It's not always that leadership ignores line-engineers just because they consider them stupid. Even obvious recommendations get tangled into the web of politics and personal favors. Sometimes the 'obvious solution' is also a play to get more power or influence for someone you're opposed to. That factors into how the advice is taken.
Part of the fiction management sells itself is consultants can cut across those organizational boundaries and find the 'right solution'. Sometimes having an 'impartial' voice echoing what others have already said is enough to convince you it's not just someone trying to poach your turf, it's actually good advice.
Further, our tendencies towards tribalism influence this process. We tend not to think of other employees as part of the same group - they are devs, or ops, or product. That builds an 'us vs. them' mentality that's hard to break. Bringing in an outsider tends to unify the group into us (the company/team) vs. the outsider (the consultant). That psychology helps us to justify bitter medicine like our team getting more work (which will help the company).
I tend to have a negative opinion of consultants, but someone pointed this aspect of group psychology to me in the past and it resonated.
Being a consultant, and knowing multiple others, I feel that not being part of the company "tribes", really allows you to more easily cut across those organizational boundaries you mention.
How many times I've heard of 2 technical divisions complaining about each-other where the main issue is the lack of communication and simply not knowing who to talk to in the other "tribe". Consultants however are part of their own little 'consultant tribe' across company tribe boundaries, who more easily bond together and talk about stuff like that. This means this group is more diverse and has more contacts spread throughout the organization, making them ideal to cut trough the bullshit. You also have the fact that when a company was happy about a consultant, chances are that when they need someone somewhere else, they'll hire the same consultant again, dumping them in a completely different group of people.
Yes this is 100% a social problem, but it's there, and a group of outsiders will address this more efficiently.
That's the "good case" anyway. In actuality, management consultants (the kind you're referring to) are usually brought in so someone can cover their ass and say "look, the smart, expensive people said we should do X!" regardless of whether or not X was a difficult solution to figure out.
I’ve of course heard the “you’re saying the same as what the engineers have been saying for years” at various points, and management might not always listen well enough to engineers, but I think people underestimate the importance of synthesizing their messages to get management to act
As someone who has been on both sides of this, the best advice I have for a manager in that situation is "if you say no, tell them why." As someone in a senior engineering role, to me, it's my responsibility to understand the forces at play with the product I'm developing, so that I can do my job. I accepted a long time ago that engineering is not just about being technically excellent, but also understanding how the product fits into the business. If the answer to engineering recommendations is just a flat "no", then that probably means that there's information that is keeping me from doing my job to my full capacity; I don't just throw out ideas without thinking about them, I take all of the facts that I have and come up with cost-effective solutions.
Now if I can only hunt down those bastards who made Tempest, Qix, and Galaga...
Bear in mind everything that's not included in this analysis: AI, mesh deformation, level logic, player mechanics, enemy mechanics, all the lower level stuff and so many things I haven't even thought of.
Good games? Depending on your outlook, yeah. But there's nothing really...different? About them?
Their anticheat, maybe? It's a bit impressive.
I remember reading an interview with a well known person in the 3D and gaming industry in a gaming magazine well over a decade ago. I can barley recall the details but one thing that stuck out was the interviewer asked why couldn't the game engine be the API giving the developers higher level tools instead of OpenGL or direct 3d, as well as audio, input and others. The answer was that game engines were too focused and not flexible enough to be used as a kitted API.
Fast forward to today and that is exactly what Unreal and Unity have achieved. They built game engines that are flexible enough to be complete tool kits to build other games. Interesting to see how far we have progressed.
There are also trade offs in flexibility that come with using Unity or Unreal. They make it easy to make games that fit within their paradigms but it's relatively harder to make a game that does something radical or experimental with rendering or physics.
I say this as someone who lives in Unity. It's absolutely the right choice in many situations but it's not without tradeoffs and compromises.
Many old style AAA developers see using off the shelf middleware as heresy.
Also triple AAA isn't that old! There's not a generation retiring quite yet :) People who want to still be in the industry are. I've seen the industry get older on average as I've worked in it.
Forms of middleware have been there since the early days of AAA style games - sound, video encoding etc. such as RAD Game Tools
If I recall correctly, Tribes was one of the first ones to have some moderate adoption.
I think the turning point came when the work to write a new engine became greater than the work to use an off the shelf and make it your own. And that only came about as people's expectations for physics, lighting, etc got pushed ever more.
Such as mountains of bloatware/shovelware template games clogging up every single online store and platform.
The solution is better curation and discoverability. Not harsher gate-keepers and horribly difficult development.
So many incredible games wouldn't exist without modern development tools. Just because you were mildly inconvenienced the last you time browsed an app store, don't wish it all away.
It's because there are so many crappy games on Steam that it makes it all but impossible for indie developers with decent games to be discovered. The perception that Steam is a cesspool of con-artists and incompetents depresses the perceived value of any game on the platform not released by a known publisher, meaning any independent game is more likely to fail merely by being associated with Steam to begin with. Also because the last time the video game market was drowning in mediocrity, it crashed.
I mean... when I was in high school I used to doodle Mario levels on my schoolbook jackets because I was just that kind of nerd, and now I live in the glorious future of Unity and Godot and Game Maker and C++ and SDL2 if I want to be a masochist. It's fantastic that anyone can more or less make something that can at least theoretically be called a "game." But that doesn't mean all of it deserves to be on the open market.
>If we want to go back to publishers controlling access, who in their right mind would approve Schenzen IO?
You're right, if it were up to publishers no one would be making anything but MOBAs and first person shooters and maybe Candy Crush clones. Independent game development allows niche and non-profitable genres to thrive. However, again, it's still a problem that the sheer bulk of garbage on these platforms makes it difficult to find that good content without some kind of gatekeeping.
There is crap in all kinds of artistic work, for some reason devs seem to think they should be special and get everything for free.
I disagree.. playing a guitar does make one a musician, and writing does make one a writer. What either is not entitled to is success in the marketplace, but then, no one is, regardless of skill. Once you put your work into the marketplace, the market decides its value, and your value by proxy. But for that to work successfully, the marketplace has to efficiently sort for quality, and one way to enforce quality is having the barriers to entry be sufficiently high. In practice, that doesn't often work because entrants don't want to be sorted efficiently, they want to be sorted to the top to maximize exposure and revenue. It's why SEO exists - to optimize the Googlebot for you and you alone by any means necessary. Unfortunately, Steam also wants to maximize the size of its library and revenue, so it has an incentive to engineer a marketplace that favors quantity over quality.
But even amateurs put in more effort than the developers publishing much of the garbage onto Steam - using the term "developers" loosely. These are incoherent rage games or weekend projects, barely functional tech demos and tutorial games, asset flips, etc. Games that don't even run. If the bar for quality on Steam was merely "amateur" then I would be happy, but it isn't. There seems to be no bar, and that makes it bad for everyone, except maybe Valve.
I had tons of floppies and CDs with shovelware for ZX Spectrum, MS-DOS, Amiga and Windows 3.x, 9x, Linux,....
All before Unreal or Unity ever happened.
The reality is, every single engineer who built the very optimized engines you my have in mind (that power a few specific games at most) have great respect for Unity and Unreal.
Game development is a small world, people know each other across companies, share tech stuff on Twitter and go to the same conferences.
And on the low end you have the Unity asset store run amok.
I'm not sure how you do it, but there is a happy middle of reasonable risk with reasonable competency missing in the market. There are few games made there every year, but they are a paltry few next to the regurgitated annual CoD game or the thousand random shovelwares put on Steam every month.
And that is why nearly 95% FPS games are practically clone of each other.
"If you are the guy asking for help from people below, to your side, and above you, you’re probably an entry-level engineer. If you frequently need support from those above you (excluding C-level and executive support, that my friends are a whole new level of hell), there’s a good chance you are a mid-level engineer. If you find yourself supporting and being supported by those at your side, you are likely a senior or lead level engineer."
Basically: If you're frustrated by your work... probably junior. If you're frustrated by your colleagues... probably mid-level. If you're frustrated by too much management... probably senior. If you're frustrated by too little management... probably lead.
If you're responsible to solve tickets, you're probably junior. If you're responsible for implementing features, mid. If you're responsible for making sure the whole project will work in time for delivery, senior. And if you're responsible for making sure your team can deliver the project on time, lead.
This also suggests what C-levels are supposed to be doing: providing the vision and culture that actually fosters this kind of mutual web of support.
This is the same as being a consultant. You come on site, you listen to everyone and make notes of it, and then you make a presentation and plan for direction.
They pay you way more than their own people, to tell you what your own people are telling you. It's quite crazy.
The parable of the producer ignoring engineering specialities when assigning tickets seemed especially gregarious. Why was that person unable to learn, even after the engineers raised the issue? Why were they still there? Were they friendly with the higher-ups, or what? Why was there no will to improve the process until external help had to be brought on?
It also reminds me of stories I had heard from inside Microsoft in the pre-Nadella era. Tales of power brokers and entrenched lifers who could make things easy or impossible on a whim. What is the nature of the organisations that allow people like these to take root?
It's funny how small scale engineering problems are rooted in code and large scale engineering problems are rooted in people.
But don't mistake it for "erogenous" or "gangrenous"
'It's funny how small scale engineering problems are rooted in code and large scale engineering problems are rooted in people.'
At some point when a company gets large enough the actual product it produces stops being something sold outside the company and instead the product becomes one’s increased status within the organization and that of their immediate manager. Status seekers who contribute less technically but spend all their time producing “good optics” for the people above and selling/exaggerating their achievements get promoted a lot more, even if the good optics were achieved by doing something that makes the product less sellable. Eventually status seekers get promoted into management and then promote upward like people who are also status seekers.
The people who spend all their time doing the heaviest engineering aren’t often spending much time selling their achievements to busy executives who don’t have time to look at their accomplishments at let their work speak for itself. Which is why they tend to stay in the lower levels of the company.
https://blueprintsfromhell.tumblr.com/
at a former job i knew a team of devs who built and maintained a framework to help teams of researchers put together pipelines of simulation (often fluid simulation), animation, rendering, etc. This framework had its own blueprinting functionality.
One of the cute features they had was nested blueprints -- e.g. you could define some blueprint, and abstract it behind an interface as its own component, then re-use the component in another blueprint.
So i guess in a "blueprints from hell" scenario that lets you have "fractal blueprints from hell".
edit:
i'm also reminded of Chris Domas' defcon talk "Psychological Warfare in Reverse Engineering", where he sets out to construct binaries that cause despair when someone tries to reverse engineer them -- in particular he asks the question: "could i make a binary that draws a particular picture when someone views the binary's control flow graph in IDA"
Maybe, strip out most of the game dev stuff and write a small book? That might help. I don't know why, but books work on managers much better than articles. At least I can give or recommend someone a book without making it seem like an insult. "Hey, I've read this book by an engineer who worked on many large game projects. You should check it out."
I'm not qualified to write a book however because this article itself is proof my writing skills need work.
Otherwise I'd be able to be more clear.
The article and writing was very good. The rant sections could have devolved several orders of magnitude further, that wasn't the end of the world IMO.
I got some major general-worldview TILs about engineering from the parts that talked about experience levels.
The bits you said could be their own articles, and/or which you said you could go on about... I hope to read about those in followup posts. :D
Sadly this was still all par-for-the-course back when I was in the industry and all my old contacts still affirm that this is the case today across most studios. Easier to burn people, ship something and then re-hire/train a new batch.
It's a shame the incentives of development aren't aligned with reasonable employment, the industry would be a lot better for it.
Wow! That sounds terrible.
> So, here's how to stamp out creativity in the rest of the organization and get a bit of respect going.
> One: Allow subordinates no humour, it threatens your self-importance and especially your omniscience. Treat all humour as frivolous or subversive.
> Because subversive is, of course, what humour will be in your setup, as it's the only way that people can express their opposition, since (if they express it openly) you're down on them like a ton of bricks.
> So let's get this clear: blame humour for the resistance that your way of working creates. Then you don't have to blame your way of working. This is important. And I mean that solemnly. Your dignity is no laughing matter.
> Second: keeping ourselves feeling irreplaceable involves cutting everybody else down to size, so don't miss an opportunity to undermine your employees' confidence.
> A perfect opportunity comes when you're reviewing work that they've done. Use your authority to zero in immediately on all the things you can find wrong. Never never balance the negatives with positives, only criticize, just as your school teachers did.
> Always remember: praise makes people uppity.
> Third: Demand that people should always be actively doing things. If you catch anyone pondering, accuse them of laziness and/or indecision. This is to starve employees of thinking time because that leads to creativity and insurrection. So demand urgency at all times, use lots of fighting talk and war analogies, and establish a permanent atmosphere of stress, of breathless anxiety, and crisis.
> In a phrase: keep that mode closed.
> In this way we no-nonsense types can be sure that the tiny, tiny, microscopic quantity of creativity in our organization will all be ours!
> But! Let your vigilance slip for one moment, and you could find yourself surrounded by happy, enthusiastic, and creative people whom you might never be able to completely control ever again!
> So be careful.
https://www.youtube.com/watch?v=Pb5oIIPO62g
https://www.brainpickings.org/2012/04/12/john-cleese-on-crea...
Second paragraph: "If a game company in Los Angeles is having an Unreal Engine 4 problem that no one can solve, I usually end up getting that call. I am writing this article to explain why I get that call, how one would prevent the need for that call, and what I normally do after getting that call."
So while a lot of these things are pretty useful strategies for any large project (we had asset naming conventions for parts of the Flash cartoons I was working on around the turn of the century), they are all common problems he's encountered in his career of being That Guy You Call When Your Big Unreal Project Is Broken.
I still read it as him trying to suggest what better practices you can adopt specifically pertaining to UE4 that would "prevent the need for that call"
“responsibilities than they can handle, and development suffers from this person being fatigued, or worse, leaving altogether. Neither the engineer nor team can grow because of this, as new features or changes get increasingly more resource-demanding to implement”
This is said about mid-level engineers who burn out, but I think it happens to senior level engineers as well.
I learned a very different things, but thank you nonetheless: there are companies that care about these things! I just got a full-time job that is basically like this, a hell on Earth (I've been freelancing before). So the lesson I learned is, I should be looking for a new job :)
My coworker (we are 2 in the team) is already jumping ship even when he joined later than me.
My question now is: is there any company mid-big size not guilty of those sins? Does it even exist or it is an utopia wished but not achievable? (just asking)
This sort of thing drives me insane (or at least used to; my current colleagues, from junior to senior, are all actually pretty good). This is something that everyone knows, just by looking at it, is a bad thing. Everyone who saw this must have known it wasn't a great idea, even if they didn't know why. Even without being able to elucidate why, everyone (should) get the feeling that having these things in the repository is asking for trouble; I'd expect a junior to have at least a bad feeling about it, and a senior to be able to specify what kind of problems it could cause. I don't know about the exact circumstances, of course, but it's even surprising that nobody found this problem while they were investigating. I've certainly gone as far as "are we definitely using the exact same binary files on the different systems?" when chasing odd problems that only manifest on some installations and not others, so what went so wrong that nobody got that far in their debugging?
So unless the company in question had by chance a room full of people with this particular massive blind spot, people saw this happening and they knew it was a bad idea and they didn't do anything about it. Culture failure. I don't know why. Maybe they were afraid to mention bad things because pointing out possible problems is seen as not being a team player. Maybe they were just so sick of it all they didn't care anymore. Maybe there were a couple of forcefully strong people in charge of the repository who saw any question as a personal attack and people were just sick of dealing with the egotistical prima donnas. Maybe the process for doing anything about it was laughable painful and went through someone who would shoot it down on the grounds that we're so busy firefighting things that are broken and you want to waste time tinkering with something that looks fine to me?
If Allar is reading, could you say a few words about this? How many times do you get called in to fix something that could have been averted with a decent programming culture, even if the specific technical chops weren't there? We've all seen teams with good culture deliver software that they really didn't know how to build when they started through good culture, and we've all seen teams of people who all know what they're doing deliver a flaming wreck because the culture was a disaster. How often are you called in to deal with something that the existing team really couldn't have foreseen without some particular piece of specific knowledge or experience, and how often is it something that they could have dealt with months ago if they had just been allowed and guided to work sensibly and professionally?
20% of the time I work on novel unforseeable issues. R&D and future tech stuff.
80% of the time its culture, professionalism, and generally "are you fucking serious" issues.
I've dealt with literally every situation you've described however. Just because this particular reason was that there were no engineers, you could swap in any other situation and still have a true story.