Hacker News Reader

Comment by russellbeattie | Hacker News Reader

Whenever I think, "Thank goodness the limited memory days are behind us", it pops up again and again. Sure, you can buy a new iMac Pro with 128GB of RAM(!!) and smartphones regularly have 8GB available, but the increasingly popular IOT devices and smart consumer hardware (like streaming media boxes, etc.) try to limit the BOM cost and thus limit memory as much as possible. Tiny memory leaks become an issue, or random crashes from wonky media codec implementations, etc.

I think the skills (and hacks) that used to be useful only to game developers and OEMs are now going to be needed by a much wider audience of devs.

View on HN

> Whenever I think, "Thank goodness the limited memory days are behind us", it pops up again and again. Sure, you can buy a new iMac Pro with 128GB of RAM(!!) and smartphones regularly have 8GB available

I think you're underestimating the accretion of software bloat. You remember the days when you did exactly the same things, exactly as fast, with 1GB machines? 512MB machines?

As long as devs don't give a fuck (er, they make "professional decision" of optimizing dev time (over product quality)), the days of limited memory are not going to be behind us.

I always think about Windows Vista when I hear "unlimited memory". Somebody looked at expanding memory and processing, and decided "let's have an animated 3D background for the OS, that'll be good". Memory is always going to be pushed to its limits unless devs and users alike go out of their way to keep it intact.

What's the point of having more memory if not for new features?

In general, nothing.

For example, Chrome eats pretty much all free memory keeping tabs loaded, but surrenders it gracefully when needed. That's a great use of memory.

But for an OS, and specifically for a feature that doesn't devolve unless the user goes and manually changes it, it's obnoxious. The OS is inherently a support layer for the things the user opens by choice; I'd argue any expansion of its resource footprint ought to have a clear justification.

In an Operating System, that ‘more memory’ could be used for applications instead of bloat.

Sorry, I don't trust myself or you to exercise restraint. The ONLY thing that keeps memory usage in check is physics.

Yes, man, and kids don't understand or perceive software bloat at all ;(

I remember coding while listening to music and reading docs somewhere using a P166MXX with 32mb of ram, and that ran quite snappily using debian and fluxbox.

I now code while listening to music and reading docs somewhere using an i7 with 6GB of ram and sometimes I'm swapping because I left Google Chrome opened for too long :(

It feels like that dev time cost is instead being crowdfunded by energy, disk, CPU and RAM that every user has to pay with and eventually that cost is paid by Earth itself. An app might be cheap or free but in the long time cheaper for the planet and each user would be an app costing a dollar or two more per install. I was using a 2011 laptop with only an HDD, 2 gigs of RAM and no discrete GPU until it actually went and finally broke last year so it irks me especially when I see this handwaving of "computers are cheap", especially from Westerner millenials or developers from SF. The fact that well off people who change machines every few years can even dare to call less well off people with older and shittier hardware "entitled" for wanting performant snappy software just like they got 5-10 years ago when they bought their machine is baffling. Not everyone needs a crazy and new machine, writers, reviewers, sales people, admins, etc. Case in point - G.R.R. Martin uses a DOS machine to write - https://www.youtube.com/watch?v=X5REM-3nWHg .

I.e. Slack and Atom got absolutely lambasted for performance, sluggishness and resource use (while VS Code was applauded, so it's clearly not an Electron specific thing) despite being made by companies valued in billions and based in the most expensive region of the world, one of them even being a paid product.

Or a game with pixel art (I do like it and I understand that particular indie dev optimizing for time with such a niche product so I don't want to name names here) graphic and gameplay only as deep as some better Flash ones from mid 2000s requires as its minimal system requirements several GBs of RAM (for comparison, Doom 3 recommended, not even minimal, was 512 MB in 2003) and disk space, etc.

Or when a graphically simple 2D game requires a 64 bit OS (despite using no 64 bit features seemingly), a non-integrated GPU (and not because of some lack of OpenGL features but due to poor optimization) and runs at 30 FPS on an integrated Intel that has 0 problems with Mincraft with really far draw distance. And it attempts to load hundreds of files (all of the game assets for an entire 4-10 hour long VN) at boot, taking 30 seconds on an HDD. And they could be loaded incrementally (loading what is needed right now only and everything else in the background, even dumbly and fully into RAM as it does now) or packed into SQLite or a ZIP to avoid so much FS access, but no - hundreds of files are being opened at game boot and there are tons of XML assets with 0 compression or minization. But instead the solution to performance woes (in gaming especially but through things like Electron it's seeping into main stream) is apparently to "git gud", "stop being a poor pleb" and getting a new GPU (apparently GTX 950 M is a potato level GPU now and only an idiot would play games on it in 2017) or an SSD so that the developer doesn't have to bother to do the tiniest of optimization.

That 2D game loading all assets, wanting a 64 bit CPU and non-integrated GPU, all for no good reasons, was Tokyo Dark by the way and due to the way the developer carry themselves I have 0 problem name dropping them, I made an entire video about that game, the disk and GPU part is at 15:15 : https://www.youtube.com/watch?v=sCXwgPJGLIE

It feels like what was done with Crash Bandicoot is some interstellar death star level technology in comparison to what some developers do, not even bothering to pack files to reduce FS chatter or load smartly or compress textual assets, they probably had it developed on an SSD, it loaded fast enough for them, it's done and prime for shipping, duh! Just gotta write a hype text about how extensively we tested it and how much effort we put in making it!

I realize I sound like an ass that's ranting and I am writing too lengthy (I did think about writing articles instead of lengthy HN comments like this one so if someone is interested feel free in hitting me up) but some of the stuff just blows my mind in ways I didn't know existed.

It's not even optimization for dev time like Python could feasibly be but sometimes outright waste or lack of basic care, i.e. Slack was apparently launching a full blown browser per organization until recently (or something like that), completely needlessly, now that part is out. At the same time they had this crazy involved (and cute, because it's 2017 and things must be cute) error page: https://slack.com/asdsad , or that semi-notorious reply article from a guy using unix CLI instead of hip BigData(tm) tools to analyze relatively small amount of data (yes, the guy is rubbing it a bit in too badly when he brings out mawk): https://aadrake.com/command-line-tools-can-be-235x-faster-th...

That lack of care is evident in other areas too, i.e. in security it manifests as these SQL injections, IoT botnets, outdated software pwns and plaintext/unsalted+sha1 password debacles. Afterwards it gets justified by "state attack, China or Russia probably" or handwaved like "we store passwords in plaintext to send them to user via email when he forgets them" (an actual explanation I read once..) or "we innovated so fast to deliver SUPERB customer experience that we didn't focus on security" (while 'security' in that case would amount to closing an admin port on an IoT appliance for example..). In general software we get also stuff like that TP-Link repeater (recently on HN) that needlessly queries NTP every 5 seconds, squandering hundreds of megs of transfer per month and basically DDoSing these NTP servers.

It's like this entire mentality that good stuff is too hard or too complicated or too expensive to do (like that Chess guy and his "clever multi-threaded application") while Pareto is very much in effect and even as little as not opening a hundred files at once at game boot or reading the dense man/info pages and thinking for 20 minutes about the problem at hand or back of the napkin math could make a big difference. 10 or 20 minutes or hours of dev time per year is not a big enough reason to squander resources so badly. There is an expression in Polish that seems really apt for developers who "optimize" their time to that degree: korona ci z głowy nie spadnie (the crown won't fall off your head, basically meaning something along the lines that exerting a little effort towards something isn't too much to be reasonably asked/expected of you).

I recall a similar event when someone wanted to stress test something on a webserver and had a few million long file with URLs in it, he did while read line curl $line in bash, it brought his local machine to its knees, probably due to this rapid process creation and destruction. I gave him an xargs with -P and -n to launch a single curl per each 100 URLs instead and it ran no problem and this time the webserver we were testing was on its knees on my much weaker laptop (weakest in the company actually, since I wasn't a programmer and didn't need a strong one), as intended. I'm actually guilty of overengineering myself, since my first try was a Python 3 + requests + grequests script, and only when weeks after I forgot where I put the script and didn't want to rewrite it I ran that xargs version (very Taco Bell eqsue solution actually - https://news.ycombinator.com/item?id=10829512 ).. And that's an anecdote but it feels like people (actual 'professionals' making a paid product and working in $billion+ corps) ship stuff as bad as the original 1 curl per URL script as if it's not a big deal and then it gets justified with some handwaving, "focus on features and not performance and security", "no one is gonna hack a toaster for anything", "computers are fast and cheap", "optimizing for dev time", etc.

It's a typical high volume low margin situation, like Steve Jobs once said during original Mac building that improving a load time by even a few seconds saves lives of people because so many people will use the Mac so often that it will add to a few lifetimes.

In overall I mostly agree with you. However I doubt effective programming will only add 1-2$ per app in development costs. For better code, you need better and more programmers and more time and money. And excellent programmers don't grow on trees. There is limited amount of them, so they're really hard to get (event if you have money).

If you're company owner, which path will go? 1. Adding features less frequently, costly development, more people needed, but highly efficient code. 2. Frequent feature updates, cheaper development, less people needed, but shitty code base.

Even if you're brave enough to go for 2, there always will be competitor with 1. attitude, that will crush you into oblivion.

In case of game development, there is Duke Nukem Forever example. They tried to perfect it, changed game engine twice, but release took them so long, game looked dated anyway.

How much time and cost do most of the things I listed add? I mean really.

Building a 32 bit exe of a game that uses no 64 bit features, packing assets up to avoid FS chatter, loading lazily, closing up ports on an IoT appliance, not abusing NTP like TP-Link does, not pasting raw user input into an SQL query, having a dedicated security team that 24/7 monitors all tech deployed in the company for outdated versions of software?

These things are absolutely basic and most are one time efforts and others completely achievable. None of them require any degree of excellence. This is not about excellent code, this is technology 101. There are trade offs to be made like IDEs in Java vs. native ones on look and feel, features, start up speed, snappiness, etc. but there is no trade off in a situation where a program does less stuff, does it in a worse way and does it slower and taking more resources.

Look at amounts of money Equifax operates with and how touchy the information they handle is and try to tell me again with a straight face that what they did skimping on security and running outdated software was all okay because if they did better they'd be crushed by costs and competition into oblivion. And now there are already articles pointing at China with evidence as flimsy as "Chinese security blog reported the vulnerability day after it was patched by Apache and a week later Equifax got hacked".

Or explain to me what and why is TP-Link doing with it's repeaters querying NTP every 5 seconds (which actually takes more development effort to do than not doing anything would).

Or the recent failures of Apple, like password being stored in the hint field, that got deployed despite their (supposedly) stellar QA and polish that justifies the high price of their products.

This fail talk all reminds me of yet another crazy negligent story. There (and still is) an online shop in Poland that once was doing some "adjustments" on a world facing machine (that was supposedly not available from the internet due to high traffic causing the hosting provider to take it offline... I don't get it, the language and concept described is murky). Someone accidentally removed index.php (by renaming it to inedx.php), the web server had file listing enabled so what was shown was the webroot file listing and there was a textual backup of entire DB in it that had in it real names, phone numbers, delivery addresses, plaintext passwords and email addresses in it, it was of course accessible to the web server so all that separated you from data of 65 thousand people was a single click... The company of course bullshitted and gave 20% sale to everyone affected after lying for 4 days and saying they have "experts working on it"... They are also quoted as saying that "users agree that all their data is public when they sign up" (about real names, phone numbers, addresses, etc. despite the fact their terms and conditions said that all data is used only for order processing and never made available to anyone..) but it's murky and might have been a hoax. I'm not aware of anyone going to jail over this and the shop is evidently still open for business. Here's an article (I do not have an English one) if you're interested: https://niebezpiecznik.pl/post/kupiles-papierosa-przez-inter...

Tell me that stories like these are not absolutely surreal and that you'd never do as badly personally (I mean really - all it takes would be to try visit the website you just edited to see if it's okay and notice the file listing, lack of index.php, etc.). I'd not believe such a multi-layered fail story (file listing on, removing index.php, plaintext passwords, DB dump in web root and accessible, they way they didn't do responsible disclosure, etc.) if someone told me, it's too outlandish but it's also - evidently - true.

A university teacher would have crushed me into oblivion if for homework I submitted a web app vulnerable to SQL injections because "no one will guess to do that and it's illegal anyway" and that stored plaintext passwords as a "reminder feature". But I would just not submit something as bad in the first place, and as you can see I am not coy and can stand my ground if I think something right. But in real world both happen and then people scream China.

Even just recently someone had a laugh here in the comments under Mirai story about how it was considered (as always..) to have to been China, Russia, North Korea, etc. and then it turned out to just be few really smart Minecraft kids plus millions of devices with Swiss cheese security out in the world.

Duke Nukem Forever is a very special case of development hell, it doesn't exonerate games that don't even care. I have played games on my old laptop with no real GPU, including Unity3D ones, it's not the tool, it's how it's used. Today I can't play a 2D VN I paid for on an integrated Intel GPU and that's somehow okay.

I've already spend too much time replying to you and the "hurr durr we cna't all use cppluspluz!" gentleman/madam below. I won't be reading any more replies here, if I didn't convince you then nothing will (short of getting burned yourself by some company leaking your data in a dumb way - hopefully not).

That's a trade off. Have you ever estimated a game development budget? Sure, we can write our own engine in C++ and be running smoothly on 7-year old machines - which, theoretically, can bring some additional sales. But we can also use the money that C++ engine developers cost to hire much cheaper mid-level devs with a typical industry engine (Unity/Unreal), invest that time in additional polish/iterations and get a much better ROI.

And quite more often, it's a choice between doing a game using a modern engine or not doing the project at all.

No. There is no trade off. This game is a VN, this is a very simple genre, I could reimplement the engine easily in C++ and Lua and I do not consider myself a superstar. And I do not need to be Gordon Ramsey to be able to tell I've been given a plate of shit.

The C++ part is a complete strawman, nowhere did I say that everyone must now use C++ only. Unity, Unreal, Python, Electron, et al are not the problem here, the problem is bad practices and laziness. I ran simple and free 3D Unity games like The Very Organized Thief on that years old integrated GPU, today I can't run a 2D VN I actually paid for and that does nothing graphically stunning on a much newer and more powerful integrated GPU. Because reasons. Loading assets of the entire game all at once in a blocking way and storing them in loose files instead of packing and compressing them is not generating ROI or saving time - it's plain dumb and lazy. This is not okay in my book and if it is in yours then you are part of the problem. If you opened a hundred files one a time in C++ the result would exactly be the same and a fix is to actually test it and improve it, not decide it's good enough on your SSD and slap a '1.0 Gold' label onto that build.

It's absolutely not polished either. If you have watched the video you'd know about how "voice acting" amounts to c.a. 70 seconds of gasps and hellos, how translations support and control support has been cut and there is 0 ETA on them, how the developer ignored my questions about these issues, how new game plus is semi-broken, how lacking the VN features are (not even a dialogue history) and how clicking too fast can break the dialogue system (in a VN, for crying out loud, a genre that you do nothing in but click through dialogues). There are also a few writing mistakes that slipped through.

The game went through Square Enix QA (via their indie program) for months like that and took 2 years and about 200 000 euros to develop. There is still no promised Mac build despite them using a "portable, no code, HTML5" engine. I will not buy any excuses at this point, especially after being radio silenced while the developers take time to make cute replies to positive Steam reviews, and I will advise everyone to stay away from that borderline dysfunctional developer.

I have made plenty of sacrifices in the past, running really heavy IDEs on even my old laptop because features they provided me with justified the costs and the lag I experienced compared to vim, Notepad++, etc. When I pay for a product (or even get one for free) and it does less, does it worse and also uses more resources - that's not justifiable, that's bullshit and needs to be called out and stop. Clearly not everyone is cut out for tech related works and between all open and free tools the doors to tech world are open wider than they ever were, plenty wide to require at least a bit of decorum from everyone who gets in, not require users with 3-7 year old machines that are in completely working order to throw them out.

I can also forgive actual indie developers (that do not have a pile of cash and a corporate supporter taking care of promotion and "QA") a lot but there is absolutely 0 excuses for these guys or for Sony to get SQL injected, Equifax to handle people data in the way they did, TP-Link to misuse public NTP servers, IoT devices and drones to have their ports wide open, etc. zero (greed, sloths and other cardinal sins are not valid reasons).

I think the problem is two fold. Its this ideal that you should be working on "hard problems". Which for developers, means working on things one step above their current competency. Well that is great if you're doing research, but if you're shipping something or doing anything on production, you want to hire someone for whom this problem is easy, not hard. You don't want the wild-eyed fresh graduate with 'crazy' ideas, you want the old grizzled veteran for whom this sort of stuff is old-hat and boring because they've done it a million times. The first solution you come up with to any problem is never going to be the best solution. Its only when you've solved the same problem a few times that you will get better at solving it.

The second problem that I see is that of 'free' speedups. If you get a free speedup from hardware tech (like SSDs), you're thought is never going to be how can I match this speedup with my own code optimization, it means your production time is now cut in half or you can go focus on other things. Its only when you're forced to come in under a certain performance budget that people bother to optimize. As it is, this only seems to happen in fixed-hardware situations like console games/embedded systems, etc.

My point is about game development in general, not one exact game. The fact that in general, you make a trade-off between performance and features doesn't excuse the fact that there are actually bad games made by bad developers out there - in fact, I usually work in legacy codebases and have seen many examples first-hand.

Thanks for so many good examples and generally a decent writeup. You say you've considered writing articles; I'd say it's a good idea.

Favouriting this comment for reference the next time I have to bring this topic up here.

I might. I'll inform you if I do but I generally try to avoid controversial topics like these.

> Case in point - G.R.R. Martin uses a DOS machine to write

George RR Martin used a DOS machine to write. I'm pretty sure GRRM does not actually write anymore.

To add on to this, I am currently working on a project where the only source of persistent storage is [0], which offers 64B of general purpose SRAM. It is a $0.70 low power clock with built in support for battery backup.

Our main processor has more memory, but that goes away when we loose main power. Still, 64B should be enough for everyone.

[0] http://www.mouser.com/ds/2/268/20005010F-737592.pdf

I love tiny things like that, there's already so much you can do with it.

I wrote a thing way back when, probably a TI chip or something, which had like 300ish? bytes of memory. It got commands from a serial in (which was a bluetooth device, commands were sent from a java application on a laptop), the commands were a simple self-made thing which was a command (two numbers) and an argument (like, "set speed to 10"). That then controlled a PWM for the engine and things like that. RC boat with a few hundred bytes of memory.

I'm reinventing all that now with using raspberry pi's and arduinos and such. I just got an EPC32, which is a $5 device that should have wifi and bluetooth and such built in already.

You'll still get an increase in the number of things you can have loaded into RAM at once that's proportional to the reduction. That'll give you far more leeway in when trying to load resources on the fly, which most games still struggle with. So reducing memory usage is definitely something you want to do regardless of how much you have.

Also, if it's a PC game, you're looking at memory limits far below what the HW may offer - at least if you want it to run without annoyances. Windows limits the amount of memory it'll allow to be committed to the DRAM size + (current) pagefile size (may only be 4-8GB on SSDs); note how that doesn't include GPU memory - and allocating that counts as commit due to how residency works. Then you're looking at the OS eating 2-3GB of that, plus 1+GB if the user has a browser opened, plus anything slowly leaking memory. And a LOT of recent Windows applications leak memory or actually use insane amounts (including, amazingly, a built-in service on Win10) - if they never access it again it'll eventually get evicted from the working set, which means it only counts towards committed memory; task manager doesn't display per-process committed memory by default. That's only a fraction of all the problems.

I still use things like struct packing on a regular basis to reduce bloat. Just because systems have absurd amounts of ram doesn't mean that Chrome won't gobble all of it. If I can get my application's memory down to just a few pages (4k chunks on x86) then I'm much easier to page in after they switch back from Chrome. I don't have hard targets but I do still make an effort here because it's worth it not to get bugs about my applications being slow because of other bad actors.

Reply on news.ycombinator.com