Accidentally making a language, for an engine, for a game
verdagon.dev
verdagon.dev
One time, about 8 years ago, I backed a game called Nowhere[1] by a very talented programmer. The original premise was an alien life simulator.
Well, it's been eight years, and development is still going strong!
The developer is currently working on the String implementation for the programming language he invented[2], which he's using to write the other programming language he invented[3], which is eventually going to be used to write the game.
[1] https://duangle.com/nowhere
At what point do you stop believing that?
I mean, never. But sometimes people just need to make stuff, and that's super cool. They're doing no harm, let them enjoy their fantasy.
I have been in a couple of communities involved with developers as they built games and it was almost disappointing when the game was released because the 'journey' was finished and so time to disband.
It can certainly be a curse though if the developer isn't or stops enjoying it.
His main YouTube channel: https://www.youtube.com/c/RandallThomas/videos
His secondary YouTube channel: https://www.youtube.com/c/RandytheSequel/videos
Of course, he still has nothing close to an actual game and the progress he's made in a year is visually similar to what i threw together in a month in Unity for a freelance client once. Though admittedly, he's probably learnt a bit more about how game engines work by creating his own and it's been quite the educational experience, as well as something that actually has gained him a nice following of people who support him towards more videos.
But that's still not very conductive to shipping finished games. It's very much the same as people whose games tend to balloon in scope, another thing that's as damaging to finishing a product in a reasonable amount of time as writing your own engine is.
I do suspect that not everyone has the utilitarian approach to game development that i or other people like me have: who just pick Godot/Unity/Unreal for every single project of theirs without ever caring much about the inner workings of everything. Then again, i'm the kind of person for whom, for example, adding graphical effects to their game consists of utilizing pre-made shaders instead of learning to use the shader language properly and all that.
Not that there's anything wrong with that, but those are wildly different mindsets.
I’ve decided to shelve Arcane (a video game I’ve been “working on” for 7 years) so I can practice shipping smaller games first
I’ve been sending out a personal changelog every Sunday
I’ve started using Unity so I can hone my design & prototyping abilities
I scrapped my old custom website with this WordPress one
I’ve transitioned to having writing (this site) be my main communication medium since I’m not the biggest fan of chasing the YouTube algorithmNonetheless, i hope that his further pursuits are more successful or at least fulfilling for himself!
Who knows, maybe he'll be able to create interesting content about Unity some time in the future, like Sebastian Lague and others make: https://www.youtube.com/c/SebastianLague/videos (if he ever feels like doing more YouTube content)
A following which has recently turned against him after they funded 6k a month on his patreon, and he flatout told them that he would stop communicating, doing YouTube videos and he would also stop working on the game they were giving funding for.
But as of recently I'm now doing game development again, but this time using Unity. I've learned that game development is really hard, geometrically more so than a random business application or tool, and you have to use all/any tools that are available to you.
I loved his (demoscene) demo Masagin[1] way back when, and have the SVGs of the geometric shapes somewhere, with a vague idea of bleaching them onto t-shirts (speaking of old projects that never come to fruition...)
Frankly, I can't say I'm surprised that's where he's at with Nowhere. Well, I'm glad he's having fun.
I'm glad he's having fun, too :)
People get into game development because they want to have fun programming. Getting projects done involves a lot of things that are not fun, so it usually goes nowhere because the incentives are misaligned.
I've noticed this in many hobby professions - eg. hobby woodworkers spending more time on creating workbenches/tools/jigs/sleds than actually building some projects, hobby guitar players spending more time on gear than playing, etc. and I'm 100% guilty of it as well.
The only thing to keep in mind is that writing software for business (e.g. SaaS) requires the exact opposite approach. You go lean, write what other people want, and go for an MVP first.
I sort of did this once because I wanted both PDF (via LaTeX) and HTML output.
As a hobby woodworker, I was saved from this by circumstances: I couldn't build a workbench in the old shed, I couldn't build a new shed at the time, but I needed a new nighstand ASAP because my son needed a larger bed, so I had to make it on an old kitchen table in the old shed.
But the good thing is it also made me get used to the whole process of refining a 3D print and after a year I barely started to print functional&useful things.
So yeah, I don't see a problem with this approach :)
That will involve sorting, carding/combing, spinning, weaving/knitting, and all sorts of other stuff that doesn’t involve razor blades and yaks.
I guess the trick is to fall in love with a species of yak that everyone else needs but wants to avoid, so they'll pay you to shave your favorite yaks.
Which is to say: sometimes, if you build a low-level library/infrastructure service and release it, someone else will take advantage of it to write the high-level thing you were originally aiming to create.
I'm speaking from experience — my company's DBaaS is built on the premise of just building a really good database domain-model to hold a certain type of large, public-available datasets, for efficient, flexible querying of them; loading those datasets into it; and then giving people access to it (through SQL or various use-case-shaped APIs.) It's a really thorough yak-shave of the DB+ETL layer of what was originally planned to be a higher-level B2C product.
But it turns out I have a pretty unique view of what should be considered "fun", because my yak-shave was everyone else's schlep. Everyone turned out to be dying to get their hands on this thing, because they wanted to spend all their time on a product, rather than getting good data to feed their product. Once we realized that, we stopped trying to build a B2C product at all, and just started selling the yak wool directly.
Totally agree. I think the only real issue here is expectations. If the project mentioned was crowdfunded specifically and only with the goal to deliver a game, then I think people would be right to be a little bit irked that the developer is instead tinkering with their own programming language rather than the game, after eight full years. However if the project was funded with the understanding that the developer was just going to explore and share the process of game development and everyone is on board with them going off-piste to develop something interesting, then I guess that's fine.
That said, the amount of money in question is $67k spread over 1660 backers (~$40 each) so it's not like this person has grifted millions then done a rug-pull :)
As a solo bootstrapper founder it's really hard to do development and business / marketing in the early stages. Focusing purely on development is preferable when time limited, however, I think it's sometimes helpful to stop development and take a break to spend on business / marketing - i.e. some education, or preparing marketing content for the future. This tends to put in perspective how much time sometimes is wasted on meaningless things and gives clarity on what is really important - shipping a quality product.
Well, it's been eight years, and development is still going strong!
The developer is currently working on the String implementation for the programming language he invented[2], which he's using to write the other programming language he invented[3], which is eventually going to be used to write the game.
The day they decide to get into that is the day they stop being a satelite company and become a rocket company instead. The person put in much more eloquent words so wish i could find that comment but its sounds similar to that.
> 39. Any exploration program which "just happens" to include a new launch vehicle is, de facto, a launch vehicle program.
> 39. (alternate formulation) The three keys to keeping a new human space program affordable and on schedule:
> 1) No new launch vehicles.
> 2) No new launch vehicles.
> 3) Whatever you do, don't develop any new launch vehicles.
This one really made me laugh :D
> At the start of any design effort, the person who most wants to be team leader is least likely to be capable of it.
What an excellent link, thanks for sharing. Definitely bookmarking this.
I'm having a lot of the same struggles working on my search engine. What I want to do with files is often somewhere between what the operating system provides (too low level), and what a DBMS provides (too high level). So I'm having to build all these weird bespoke disk-based data structures myself. It's clear these things can be done and there is a lot to gain from doing it, but the language support for non-trivial disk based data structures is a bigger pain in the ass than sitting on a burning cactus covered in tabasco sauce.
The vast majority of games don't need to be low level for performance. Practically nobody outside of massive AAA studios is writing architecture specific assembly for modern PCs or consoles (and even in those studios, there are very few people doing so). The majority of gameplay programmers are writing bog standard C++ that would be just as performant in C#, or in many cases a lot of the logic is written in a higher level scripting language that is orders of magnitude slower than the native C++, or even just writing all of the game code in something like C#.
> Really does invite the notion that there ought to be a better way, you get a sort of itch you can't quite reach to scratch.
Oh for sure, and there often are better ways to solve these problems. In my experience, one of the most common reasons for not doing things better is because the codebase is based on an engine from 20+ years ago when they _did_ have to avoid interfaces and virtual function calls for game performance, and you have "modern" systems that are built on top of those abstractions (in the same way that lots of modern networking libraries still leak details about concerns from the 80s/90s)
You're comparing games rather than engines here. There is no reason at all that you can't write your simulation logic in vanilla bare bones C++ as factorio has done, and still get the benefit of using an existing engine. You get all the neat (and hard) things like serialisation, multiplatform support, networking, asset management, patching, and a bunch of other things. Sure, factorio may not need a 3d renderer, but it definitely needs and uses the rest of what I listed above.
To be fair, factorio is a pretty good example of when the existing options offer limited value (lock step networking, _very_ simplistic rendering, very little in the vein of "gameplay" features), but it's one of very very few. For every factorio, there's 100 2d side scrollin platforms that are written in a custom C++ engine rather than using unity.
> adding a level editor, asset loading, character animations, scalable UI, collision detection, networking, state management, serialization,gamepad support, multiplatform support, store/platform integrations
None of these are nearly as hard as cult-of-always-use-an-engine makes them out to be. And there's value in having complete control over your product and understanding how every part of it works. You don't have to wait for Unity to support a new platform, or fix a bug that's blocking you, or implement a feature you want, you just do that yourself.
I'm not saying it's the wrong choice to use a bespoke engine, indeed many games that are made with bespoke engines would not have been made otherwise and I'm grateful that such things exist and enable those games to be made, I'm just saying it's not necessarily a bad idea to make your own engine. Hell, Jonathan Blow is a successful game developer who made his own engines for Braid and The Witness, and now he's gone as far as making his own language and compiler to make games with going forward because he feels C++'s eccentricities get in the way too much.
I never said they were _hard_, but they take time. Time that can be spent on your game.
> You don't have to wait for Unity to support a new platform, or fix a bug that's blocking you, or implement a feature you want, you just do that yourself.
Every system has bugs. You're always going to have to make tradeoffs when building projects, and building an entire framework from scratch to avoid those bugs is throwing the baby out with the bathwater. You're trading the possibility of at some point in the future being blocked from developing a very specific part of your game for the guarantee of spending time up front yak shaving.
> Hell, Jonathan Blow is a successful game developer who made his own engines for Braid and The Witness, and now he's gone as far as making his own language and compiler to make games with going forward because he feels C++'s eccentricities get in the way too much.
Blow has shipped two games in 20 years, and one of those (braid) predates _any_ of the existing engines being free and easily accessible. Another of the "build it from the ground up" camp is Casey Muratori, who started handmade hero almost 8 years ago, and is nowhere even close to a game. The post we're commenting on here is entitled "Accidentally Making a Language, for an Engine, for a Game". If your take away from those things is that "you'll spend more time fighting the engine" then I don't really know what else to say.
You say that like it is a bad thing. By (nearly) all accounts Braid and The Witness are lovingly and expertly crafted, unconventional, and brilliant games. He's not making the annual Madden or Call of Battlefield here.
> Casey Muratori, who started handmade hero almost 8 years ago, and is nowhere even close to a game
Be fair to Casey, Handmade Hero is primarily an educational project and he only devotes a few hours to it each week. Though admittedly he has allowed the scope to creep quite a bit from the original goals.
> The post we're commenting on here is entitled "Accidentally Making a Language, for an Engine, for a Game". If your take away from those things is that "you'll spend more time fighting the engine" then I don't really know what else to say.
My point is more that I'm kind of sick of people being so down on the concept of building your own engine even for relatively simple 2D games. Yes, I agree that it takes time, but so does learning the ins and outs of an existing engine and the time you invest in the latter nets you less generalizable skills than the former and leaves you with less control over the end product. It is a tradeoff, and like all tradeoffs there are good reasons to go one way or another.
If you do end up in a situations where you have more objects than is suitable for even these systems to simulate you don't need to throw the baby out with the bath water. Create your own managers running whatever simulation you wish without tying the units directly to the most obvious game engine entities, with whatever scheduling you need to make the updates work with the compute budget you have. Use the engine entities to represent whatever higher level object makes the code manageable and easy to work with.
Then you can surely leverage the game engine for input handling, rendering, and a lot of your logic, just writing your own code for whatever challenges are unique to your game and where the standard approach for that engine is unsuitable.
For the level of performance the Factorio devs want it makes sense they'd opt for maximum control over everything rather than fighting someone else's implementations.
I wonder if there's a DBMS out there somewhere that already does what you need? Seems like by the time you got to a scale where you would need custom stuff, you'd have money to redo it(Unless it's a foss thing meant to be big but not commercial, or to run on a server you're paying for yourself)
Existing engines are great as long as you want your game to look and behave like every other game written in the same engine. It's gotten to a point where a careful observer can tell which engine a game is written in just by looking at it and feeling how much input lag and framerate variance it has.
If you can improve culling before rendering and physics, minimize the surroundings streamed in to your immediate vicinity, and optimize some of the heaviest materials and post effects, that goes a really long way.
Of course, there's an endless list of caveats depending on what your game specifically is, but even in an open world game where things happen off screen the above applies. You just have very simply LODs/imposters for faraway visuals and simplified logic running at lower frame rate for characters in other parts of the world.
With trading platforms, they last for years, decades maybe. They're created with some ideas around how to manage performance and to support evolving requirements. Developers know this thing is going to be around for a long time, so its treated as such. Years down the line, you start creating a new platform and look to do a long migration to keep clients happy because you can't just turn off their favourite functionality.
Modern AAA FPS games seem to be the complete opposite. Reinvent significant amounts every release. Dump the old game as the new one is released. Much seems to be from scratch. The BF2042 scoreboard issue seems like it should be almost off the shelf. There also seems to be this big shift towards short release cycles which pushes even more churn and reinventing the wheel. Look and feel must be updated to keep things "fresh". Although most of the popular games on Steam[1] are older games that have been around for years.
So many places to shave a yak, I'm surprised games get shipped at all.
Of course take all of this comment as an outsider who just yearns for the old days of cool mods and custom servers.
I think this should be qualified. The audience and creative workers demand new assets for new titles, but gameplay concepts and codebase seem to move at a slower pace, though live services seem to be distinct systems and not expected to live long, of course.
On one extreme, you have EA's sports titles, where new releases have just incremental updates.
What is the architecture pattern of a trading platform?
I am looking to build a system that is able to: - receive 1000s incoming streams of data - save the data - make data available to live subscribers
The closest analogous system I can think of is bond/stock/commodity/etc price subscriptions for traders.
I feel like this many in - many out data stream architecture should be solved by now and I rather not start from scratch in architecture and technology choices.
I can’t find the right phrase to even google to get started.
The trading platforms I've worked on tend to order things into a single stream that you can act upon. This helps with testing, race conditions, auditability. You will want to be able to replay an exact series of events to recreate conditions.
LMAX Disruptor[1] and Aeron[2] are two open source examples of something widely used, either using the libraries themselves or as concepts.
Some things that trading systems (generally?) need which you may not, which might simplify your architecture:
- Every message must be delivered and in order. UDP is usually used over TCP, as the platform will likely want more control over handling missed messages. - It will be common to have read/write applications that need to add a message as a response. Writing is difficult as you need to be quick, otherwise a subsequent message from another application might invalidate your message.
Do subscribers need older data? How do dropped packets impact the system? Can they just be forgotten? What latency requirements do you have? Do subscribers also write to the same stream?
- [1] https://lmax-exchange.github.io/disruptor/ - [2] https://github.com/real-logic/aeron
For franchise games there will be incremental improvements to engine systems; some to support new gameplay features, some to help with development tooling/pain-points, and some to improve visual fidelity. The trick is choosing the correct number of upgrades/refactors that fit the release schedule while remaining relevant and not dooming your next franchise release with insurmountable feature debt.
In my experience the biggest engine changes come with new hardware platform support, typically a console generation jump. More power requires a toolchain that can handle more complex assets (and more of them), new graphical features and expectations, different optimizations, new API, different storage types or capabilities etc. Bigger franchises may use a new console generation as an opportunity to gut an existing engine and rework to the strengths of the new hardware generation (and remove the support for older hardware, such as removing 32bit pointer support, DirectX9, Windows7 etc).
Historically where game engines get in trouble is when a codebase is written to one franchise and then forced on to a team making a fundamentally different game with fundamentally different requirements (open world vs compact maps, physics heavy vs platformer, single player vs MMO). At the executive level it makes total sense, they spent x million$ on a game engine and are told by everyone how great an investment it was... why not spin up a new game using that amazing (and paid for) technology?
The flip-side has also sunk numerous game engine efforts, which are often buried without ever becoming public. Doing everything all at once is impossible, you can either do many things generically but with restricted performance and capabilities. Or you can do 'everything' required by a specific game and slowly introduce features that support other games and make things more generic.
Actually I'm not sure it's really a mistake, it's just that doing PL development is so much fun that there's no reason to go back to the original project, if original the impetus for it was to have fun.
Hi, this is the yak shaving developer in question speaking. I was worried we'd get grilled hard over our slow progress, but I'm relieved to read that most of you understand how perilous and long-winded gamedev can be. Our backers are also very patient with us, and I don't want to destroy this relationship, but keep being as open and forthright as I can with our progress.
I don't mind the jokes (after all, yes, the whole undertaking is a bit ridiculous), but just want to make sure the facts are understood as they really are:
* Yes, the premise of Nowhere continues to be that it's an alien life simulator (as in: simulating the everyday life of an alien).
* The game is being developed by not just one, but two people: My wife Sylvia Ritter[1] is responsible for the concept art that drives the procedural design (she has given me a lot of work), and I am responsible for programming and direction. We both have a hand in the game design.
* Yes, development is still going strong, and will continue to do so, until the game is done or I keel over, whichever comes first ;-)
* Yes, I am absolutely yak shaving, and there's definitely some sunken cost thing going on here, but there is still no project I'd rather work on than this one. When we launched the crowdfunding, I had a hunch that this would end up taking us 10 years. Now we are on the far side of it and I had to start cutting features and workflow improvement ideas in order to have a chance at making it in time.
* Yes, I developed a programming language for the game, because the art is procedurally driven, and that requires both fast turnaround in prototyping (close to Scheme or Python) while providing C level performance at the same time. There simply was at the time that I started, and still is, no adequate solution available. Originally, I didn't want to do it, but we rationalized that innovation of technique requires innovation of tooling, and hence worth the effort, provided we'd open source everything we made in pursuit of our goal.
* No, Scopes' string implementation is complete for several years already, the recent commits are all touch-ups, augmentations and small fixes for the userbase that has grown around the language, which were quick to do and cost me no significant time. We have in fact other, much larger, support problems that I can not adequately cover because the game is main priority, and the language exists to service the game.
* Yes, I developed a pure functional reactive language on top of Scopes for our game engine that I wrote a few examples with, in the intent of finally merging the CPU and GPU models so we save 90% of pointless and repetitive boilerplate code for CPU/GPU resource management. It's a fantastic idea that is going to go places, but after realizing how much more I'd have to write to get it all the way to its final, visual programming oriented form, I aborted the prototype and focused back on the game.
* What else have I spent my time on? I spent most of those years writing a bunch of prototype shadertoys to explore possible technology used for the game and also teach mathematical concepts to other developers, and they're all released here[2]
* What's going on right now? After streamlining package management[3] for both Scopes and Nowhere, I am presently working on our sculptable terrain engine. There is a by now somewhat outdated video demo[4] of one of its earliest incarnations. The LOD stitching has been fixed, and we have occlusion culling now, but I still have to rewrite parts of it to get a rock solid sub 10ms per frame performance. It would have been more fun to do all that with FRP instead, but tooling is never quite where you want it.
Thanks for reading all that,
Leonard
What actually happens when scope explodes is that a coherent view of the project has been lost. This is not problematic in a creative sense, just in a "finished product" sense: every time you introduce a contradiction into the work you have to either eliminate it(which creates a negative attachment response, and therefore really requires project managers to step in and cut off some heads) or you work very hard to create some kind of technical rationale, e.g. "we'll do both ideas, so now we have a new type of asset and the entire game must be populated with it and the features will be a little more complicated by it". Which you can proceed to start doing without difficulty, and only feel the downside of later.
Game devs are particularly susceptible to this because games sit in an intersection of dynamic/synthetic/interactive that allows infinite numbers of assets and features to be added, but at a gradually increasing cost, even if you drop fidelity(see every roguelike that has been in development for more than a few years.) And it lets the dev sit in a space of perpetual escape from coherence, because "it'll be great as soon as I add this next thing - after all, nobody has ever done this before". It's a little hype cycle that can be reinflated over and over.
But if you paint a picture, it's one-and-done: there's only so much room on the canvas, so you have to deal with your folly immediately to finish. You can, of course, go the route of burning it and starting over, making the same mistake repeatedly, but this provides much less of an illusion of progress than hitting the compile button on your ever-growing codebase. And you can do a lot of preparatory studies and meander without committing to the image you're making, but the act of doing the studies still propels you into a space where you can hurry up and finish whenever you want.
So the usual advice given to game devs to manage scope is to introduce a technical restriction that "limits the canvas," but if you're technically inclined I think the proper advice would actually be to limit dynamism and make more static works with simple design scope so that more of your technology can be one-off, and not required to be integrated into the large, fully-automated framing of a game engine. E.g. a magnificent rendering system that creates static images for a visual novel. "Simulator" type projects are mostly eliminated by this except where they can take a direct reference, like a vehicle sim. All of my least coherent designs started embracing the simulation concept - and doing that was itself a way of getting away from a clear statement of belief within the game, of trying to accommodate multiple sets of beliefs without directly engaging them.
Not that anyone is going to listen, of course. Sometimes one has to lose a few years of living to this stuff ;)
I also don't want to have to use slow and clunky UIs like Unity, I just want to spend the majority of my time writing code in text files, and making assets in other tools like Tiled or photoshop or whatever.
It really surprises me how few engines there are in that category. Only ones I've found were: PhaserJS and HaxeFlixel, and unfortunately no 3D ones.
My theory why: good web frameworks consider developer experience to be of paramount importance, and invest heavily into examples, documentation, and API improvements. Unreal and Unity by comparison are unpleasant to work in. The UI is clunky, the examples cap out after a certain point of complexity, and community input is almost nothing when compared to the Django, Express, Rails, or language-specific ecosystems.
Anyway, I can't say I've ever made it far enough down the rabbit hole to want to make my own language... but I know for me, the desire to make my own engine* has punctuated every attempt over the years to get better at both Unity and Unreal
*Acknowledgment that I might be partially disproving my own point, as I've tried to build both my own web framework, AND my own game engine... but I spent a lot more time on the game engine (https://github.com/RobertTownley/gamehook)
My reason for making my own was partly that it felt like less work to write what I needed than to learn the intricacies of an existing engine. Also, back then the big engines didn't have as generous licenses as now.
As for Unity and Unreal: I found them both very easy to get into and get productive with.
In the same way, every React app eventually starts to look and act like the Reddit redesign, but many web developers consider that more of a benefit than a drawback.
Conversely users of a web app generally don’t want to invest any time in understanding its underlying dynamic and instead want something that works in a familiar way. So we optimise for that.
Once you've bought into the whole ecosystem, you get an "engine" that wants your app to look and work a certain way just like how Unreal wants your game to look and work a certain way. You can customize it however you want, but every decision you don't make is made for you by the engine.
And even when you are making the decisions, the engine is always there, nudging you in a certain direction by making some things easier than others.
I assume this is also true for jquery/angular/vue and other library tutorials.
Maybe just for fun they should use alternative frameworks!
Then on the job, well unless you are a designery company it will be devs who do design and will happily delegate that thinking to a framework. Again not a react thing but a general dev thing.
I have seen this too in desktop apps of 90s. Just use MFC or some popular toolkit.
For example; animations responding to user interactions are easier to do in frameworks with an OOP approach. It is generally more "different" in a declarative way. So most React apps simply don't have those. There are lots of small things like this that over time make React apps feel Reacty.
Not every unreal engine game starts to look and behave like gears of war. If you drop a bunch of asset packs from the store it's going to look like every other asset flip out there, but so will your game engine if you use the same assets.
> , I like to write my own engines to get the end-user experience I want.
My day job is working in unreal engine, and in the last 7 years of using it I can only think of one scenario where the engine was the limiting factor in the user experience I wanted, and not something I could easily work around. If you think you're going to be limited in your end user experience in unity or unreal you probably need to reconsider how much you know about those engines.
However, I will say that I've constantly (during 10+ years of working full time with various game engines) come across cases where I need to extend or modify the engines I work with to either fix bugs or add missing features.
To take a fairly recent Unity example I worked on a game using the Universal Render Pipeline (URP) and found myself having to implement some things I just couldn't understand were missing. For instance, URP supports a depth prepass for opaque objects but still uses an empty depth buffer on opaque rendering instead of utilising the one generated during the prepass.
Just binding the already generated one instead allows you to get a lot of free depth culling and in my mind is one of the main reason to use a depth pass. But in URP it was apparently used exclusively to feed depth information to shaders.
Agreed wholeheartedly, and this is part of game development. Sometimes that comes in the shape of adding features that are straight up unfinished, other times it comes in bugfixes/workarounds. I hope my original message didn't come across as "there is no work involved in using a preexisting engine!"
> To take a fairly recent Unity example I worked on a game using the Universal Render Pipeline (URP) <...> still uses an empty depth buffer on opaque rendering instead of utilising the one generated during the prepass.
Presumably implementing that was _far_ less work than writing a URP yourself, and that's before you take into account the benefit of the art/design pipelines being able to use the render pipeline for the X months it would take you to write one yourself.
That example doesn't effect the "user experience" of the game either (which is what the GP comment claimed they wrote their own engines for), but that's not to say it's not worth doing!
> I hope my original message didn't come across as "there is no work involved in using a preexisting engine!"
Absolutely not. I've come across the argument about making a game for the end user experience before and it always strikes me as a case of not understanding the capabilities of your tools. I just wished, more for others who come across this than people with our experience, to add that some modification of the tools is to be expected.
> That example doesn't effect the "user experience" of the game either
Let me finish by saying that it's true, unless you count your game having stable frame rate on weaker platforms as part of the user experience. ;)
While this is true, it's also not a "given" from a custom engine and needs to be planned for. It's a big ask for a team to reimplement their renderer as a forward renderer, but it's a checkbox in UE4 (plus reimplementing all the materials etc).
But as a practical matter, using Unreal turns every decision you make from "what behavior do I want" into "what behavior do I want and is it worth fighting Unreal on this", so Unreal games are a lot more Unreal-y than they otherwise would be. (This is especially true on PC, where things like "how are the game assets organized on disk" are visible to end-users.)
Particularly on the 2D hobbyist stuff that I do, writing my own engine is less work than becoming an expert in an engine I don't really like so I can change most of it.
I don't agree with this at all - Unreal gives you defaults that can be easily replaced. The decision is "do I use what unreal gives me or do I write my own" for most systems, compared to "do I write my own or do without" if you're starting from scratch.
> so theoretically if you use Unreal your game could be any legal C++ program.
That's a bit reductionist, and not really fair. All of the "behavioural" parts of the engine are exposed in very customisable ways. As an example if you're not happy with the collision detection behaviour/triggers, they are designed to be modified and changed around. If you're not happy with character movement, you provide your own character movement definitions.
> (This is especially true on PC, where things like "how are the game assets organized on disk" are visible to end-users
If your definition of end user experience of a game is file layout on disk, then so be it. Knowing that something is made with unreal engine doesn't immediately turn it into another copycat unreal engine project. Besides looking at the disk layout, you could also just see the splash screen that you're legally required to use when licensing the engine. Also, you have source code to the engine, to the automation process, and the pak tools. If you want a different layout on disk, go ahead and change it.
> Particularly on the 2D hobbyist stuff that I do
If you want to do hobbyist engine development work, _that's_ a great reason to write game engines. Not "liking" an engine and wanting things done differently isn't the same as wanting something that's incompatible with a game engine's design and architecture. If you want a lock step multiplayer game with rollback then sure, you're probably not going to find it. But if you want "less floaty" character physics, or a different camera perspective, a different startup flow/implemention, you can _definitely make that within the bounds of Unity and Unreal
> writing my own engine is less work than becoming an expert in an engine I don't really like so I can change most of it.
You definitely don't _need_ to be an expert in an engine to use it, any more than you need to be an expert in python to start writing some scripts. Also, how can you know how much of the engine you need to throw out before you actually know how to use it?
There's also the issue that most engines are built to be good at a specific game type and may make trade-offs in terms of how assets are managed and scene complexity. An open-world has much more different requirements than an on-rails shooter. For instance I had to do some pretty heavy retrofitting of one of those engines to handle LoD/geometry streaming and a bunch of other optimizations to make them work well with a large sight-distance game.
Not when you start. While UE5 is no doubt orders of magnitude more complex than any web framework, no one sets out to write UE5. In fact most people start writing their own game engine because they want something a lot less complex and more tailored than what is available.
If you have the right background knocking out a very simple game engine isn't hard. Many people do it as part of a university course. The problem is that your 'simple' game engine ends up being not that simple very quickly once you start adding features.
Making a toy browser is something you can do as well.
Even if nothing goes anywhere, it'll improve your craft.
So that's what I've been doing and it's been a lot of fun. A lot of the time I'm pulling concepts from HTTP but what's great is that I can be very picky. Maybe I'll end up just rewriting HTTP but at least I'll understand deeply why all of those decisions were made.
I can practically guarantee that any given AAA game is less complicated than the horrible morass of complicated system requirements a multinational company can invent for their internal systems.
Mostly because it’s designed to approximate reality, which is complex.
Ridiculous statement. Making a polished AAA game is orders of magnitudes harder than making a React ToDo Clone. Making a Tetris or Snake clone is orders of magnitude easier than making Gmail. With engines like Unity you go from never having written a game to a shitty 3D fps game in a weekend.
It could be the connection to the physical world. There is something satisfying about working with geometry and differential equations.
The realtime constraints force you to care about performance, which many people find a fun challenge.
Your output is the frame buffer instead of the DOM. It's more of a blank canvas.
Most web development is utilitarian, whereas games are more artistic.
I am not sure, but I would love to be a game engine developer in another life. I have zero interest in web framework development.
I dabbled in games development, but I've met the same problem; videogames development is actually much more than programming (the engine), and the creative part, which is the primary one, may not appeal people who are technically-minded.
This phenomenon is very visible in Rust; there are several game engines, but very few production quality games. The most famous game engine, Bevy, is very popular, but after approximately two years of development is still an alpha (and a very changing one); contrast this with Fyrox, which is much more stable, but is considerably less popular (essentially, unused).
An example of this would be the frostbite engine, of which multiple devs at EA reported that it was an absolute nightmare to work with. Turns out, if you need developers to work on the engine at the same time the game is developed, you just lose manpower to actually engineer your game. [1]
[1]: https://www.usgamer.net/articles/ea-frostbite-engine-history...
It's used to create fancy new features and show them off in tiny toy projects. Then it's handed off to developers to make actual games and the issues begin.
The biggest issue with Frostbite was always that its developers treated it like an ever revolving set of cool new projects. There was always some fancy new system meant to replace the old one in just a few years and never any fixes for the shipped and still broken systems.
Everybody made their own framework before some of those became popular. And people are constantly producing new frameworks. Because it always will be more entertaining and meaningful for someone to build a framework/engine that feels good than yet another app/game.
Not to mention many compiled to js languages made by and for webdevelopers.
For anyone interested in how that code would get zero memory safety overhead without the classic borrow checker, a big part of it is because of the "region borrow checker" [0]. There are some other factors (iso regions, hybrid-generational-memory, etc.) but region borrow checking is the big one.
> Note that hybrid-generational-memory is not implemented yet, it's still just a design.
I'd not heard of vale, but after reading around a little looks very interesting. Nice job!
I've writing a Rust based canvas engine, with content via my own IDE, the runtime is fed data via WebSocket using my own API protocol, web proxy routing through a streaming and distributed load balancer using a new protocol, landing on my back-end platform running my own language, writing deltas to my own data store. https://www.adama-platform.com/
Why? For board game glory!
> I mean, we are running Access in Wine in X11 on Linux in an isolated user account on our server slice that revision controls your Access database in git, and we're displaying it using VNC in your web browser in flash. People can't possibly want that. But they need it.
https://apenwarr.ca/log/20120326
(Not mine, but seems relevant)
With regards to C#/.NET 6, it is now easy to:
- Have zero-allocation code through the whole stack, making garbage collection zero/near-zero. [Span<T>, Memory<T>]
- Distribute platform specific all-in-one binaries with no unbundling/uncompression or need to install any runtime. [dotnet publish -r win-x64 -c Release -p:PublishSingleFile=true -p:PublishTrimmed=true]
- For hot-path optimisation (many available profiling tools), dive into new high-performance APIs such as CPU vector intrinsics, or native memory allocation (aligned or unaligned). [System.Runtime.Intrinsics.X86, System.Runtime.InteropServices.NativeMemory]
- For faster startup times: AOT compilation. (Not necessarily faster at runtime in general; JIT has advantages there with being able to detect hot paths)
These are just the features I've used, there are doubtless many more. In the future .NET 7, it will be possible to disable runtime marshalling so interop calls to C DLLs have no overhead (along with a compile-time analyzer to throw errors if the types you're using are not compatible, aka 'blittable').
Credit to Vale - it appears to be addressing all these things so there's certainly an interesting future ahead.
Most people have no way to guess of the new ideas will be influential, but we do know we aren't interested in using niche tools regardless of what they are.
It could be an amazing leap forward for computer science, or a slightly interesting toy, and either way it's above most people's heads.
This language is great for the author as learning process, and that is about it.
But not like creating a typesetting system to use to write your book.
It seems like it is shaping up to a pretty nice language. But writing the compiler in itself is very 20th-century. Just add a parser to LLVM or Gcc and call it good.
But don't make the mistake C, C++, and Rust did, using a prefix dereference operator. Pascal got that one right.
BTW: "get" in a pure function name is code smell. More generally, transitive verbs in pure function names are code smell.
This is only a mistake when the deference is frequently followed by postfix operators, in most cases field and method accesses. Unlike C or C++, Rust does auto-dereference that essentially eliminates such situations and thus a prefix operator doesn't do much harm.
> It's like I've been driving a Honda Civic in city traffic, and now I'm in a BMW cruising down the highway.
There's a certain satisfaction in doing my own thing, even if it's crappy. Like Gilfoyle's electric car in "Silicon Valley".
Though, I do want to be clear for readers, Seamless Concurrency [0] isn't fully implemented yet, we've only just started to play with its first building block, the region borrow checker. It's promising, but I don't want to get anyone's hopes up too early.
[0] https://verdagon.dev/blog/seamless-fearless-structured-concu...
Reminds me of the time the FoundationDB team wrote a compiler to write a simulator to test their database: https://www.youtube.com/watch?v=4fFDFbi3toc
Hasn't gone quite as far as making our own language, we did it all in c++. For us we know doing things this way is a vastly more difficult approach, but nonetheless we plough on. Yesterday I was learning how bezier curves work so we can animate wind effects. The list of interesting maths and physics problems to solve are endless but oh so satisfying, and ultimately its a hobby not something we sell for money.
I first wanted to make it in C++ and opengl, and it did not look like a difficult thing to do, but godot is good enough for a lot of things, it's small, and it allows me to do multiplatform, which is such a big bonus, since C++ is never easy to use on several platforms.
I have no idea if I will be able to implement network prediction with godot.
To me, the biggest hurdle with game development is always assets. I'm a terrible artist/musician.
I plan to make assets using procedural generation of terrain, building indoors and streets.
In paid $work, there is a) an imposed direction and usually a schedule from a customer b) often a team of people to distribute work to, and to help keep one disciplined.
Example: I'm writing a VST synthesizer, virtual instrument, as a personal project in my time off between jobs. Maybe I'll make it commercial someday, not sure. I became increasingly frustrated with the half-assed portable GUI toolkit bundled with the VST3 SDK ("VSTGUI"). There's a complete lack of decent cross platform UI options that will work in this context (QT won't do it here, for reasons.)
I realized that there's already a cross platform UI option with millions of man hours put into it: the web. So I have spent the last three or four weeks writing a bridge to embed Edge-Chromium, WebKit, etc. Have that working on Win32, at least (still have to do the work for Linux & Mac, but ok).
Then on to actually making the UI in this platform. Now I'm learning npm, webpack, TypeScript.
Need a nice little HTML/CSS knob for my UI. Most are bundled in a giant framework I don't want. Oh, here's one in pure JS. But, hm. I think I'll refactor this into Typescript. Now it's forked, just for me.
Ok, well, yes I have that mostly working. Some warts, but basically is very similar to the "native" (VSTGUI) UI I had before. But now I want to write a graphical envelope editor. Off to learn HTML Canvas drawing.
Many parts of this whole process could themselves be spun off into separate projects, that I could spend years maintaining, on their own.
And so it goes. The Yak is pretty shaved at this point.
Also, how’s the compile time? If it’s too long won’t it drive you crazy?
The other obvious one is Knuth - inventing TeX and Metafont and Computer Modern typefaces so he can write his books the way he wants to. Another shining star.
Why is there a need for a "parallel" keyword? Can't the compiler decide for you whether to parallelize a loop for you?
If the decision to parallelize depends on the data, couldn't the compiler generate code for both implementations and select the right one at run time?
Or maybe the programmer could describe in code what they expect the data will look like, and the compiler will take a decision based on those annotations (and other signals, such as the HW the program has access to, what the programmer wants to optimize for (memory? latency?)).
And at least from my POV, is where it ACTUALLY need it, not just is fun.
Oh, and I had to write my own FFI library to bind into the MRI.
Oh, and the language I'm using has an SDL2 library available, but it was incomplete the last time I checked.
Fun!
> So, having learned that lesson thoroughly, I then made the same mistake again!
So true ! I don't know if it is written with a bit of sarcasm, but this one made me laugh !
'cos in the end, it's about doing what one loves. So if you think you like to make games but have fun writing the perfect game engin instead, it's not less fun !
It seems the language is still in very early stages. Not production ready.
First of all, it is a matter of which implementation we are talking about.
Second, all major implementations wouldn't have had any issue dealing with the demos shown on the blog.
I guess it was a good learning process about compilers for the author.
As for the GC, it can be turned on in critical code paths, C# supports manual and stack memory allocation.
Profilers exist, many console games have been shipped in C#, besides Unity, there are Xenko, FNA and MonoGame custom builds for game consoles.
Compare the demo on the blog with,
https://store.steampowered.com/app/236090/Dust_An_Elysian_Ta...
https://store.steampowered.com/app/985890/Streets_of_Rage_4
https://store.steampowered.com/app/107100/Bastion
https://www.pcgames.de/Arena-Wars-Spiel-18141/Tests/Arena-Wa...
All of them done in regular C#, not even taking into account Unity and its DOTS/Burst C# subset.
My point was that some languages have heavier runtimes than others (out of the box, at least).
And yes, C# is very popular for gameplay code, but I'm talking about the parts of an engine that typically use systems programming languages. Even then, you can shoehorn C# into that role with some tweaking and AoT compilation (which is a testament to the C# ecosystem's versatility). But I was just explaining why I think the word "slow" was used regarding the C# language in general, even though it's not perfectly technically correct.
https://www.jagregory.com/abrash-black-book
https://www.amazon.com/PC-Intern-Encyclopedia-Programming-De...
https://www.amazon.com/Black-Art-Game-Programming-High-Speed...
Very little regular C and C++ code, plenty of inline Assembly.
There are some conveniences that come with the technology and not every code path on the game engine is screaming for perfomance.
If you are on the path for the ultimate performance like some console titles, the C and C++ code on those engines will be very alien to most developers as well, yet you don't seem many discussing that.
Just on the subject, here is a talk I saw recently on the theme,
"Beyond the Remake of 'Shadow of the Colossus': A Technical Perspective"
https://www.youtube.com/watch?v=fcBZEZWGYek
However not every single game out there is trying to fit Shadow of the Colossus into a PS 2 or XBox 360.
More than any profession
Buying stuff on the other hand is very easy.
Just like running is(I assume, having never seriously done any sports) way harder. Buying more crap and then not even using it is easy.
Everyone always wants their fast instant gratification hits, regardless of hobby.
There's always someone who thinks: "I could make this better". Some of them even try to. Emphasis on try :)
Software developers are in the fairly unique position where most of our tools are software defined. We are experts in the discipline required to improve our own day to day work lives.
How many other professions have this liberty? Woodworking does to an extent, carpenters regularly make their own jigs and clamps but rarely venture into the realm of sharpened tools.
Contrast this with anyone working in the medical or retail(?) industries. People working in these fields are close to powerless to improve the tools they use every day.
Embrace the tool improvement life, there’s still a long way to go before software tooling is “finished”
I've never seen a custom build system or VCS or personal/project specific tooling I'd want to use.
Seems like most of it happens just cause devs enjoy it.
Making your own clamps seems very much the same.
The other reason is there's a real cult of assuming that minimal is always best, and devs resent using any tool that has more features than they absolutely need.
They also resent any more opinionated structure than they need, even if it makes things safer and easier, because they seem to really enjoy blank canvas creativity.
Plus its just very easy to work on a tool, I don't need to go buy a bunch of materials or anything
So, I'm wondering if and when someone will write an operating system in it.