Ask HN: What's the largest amount of bad code you have ever seen work?
I think I've broken my own record with this one ~2500 lines of incoherent JavaScript/C#. Works though.
It is close to 25 million lines of C code.
What an unimaginable horror! You can't change a single line of code in the product without breaking 1000s of existing tests. Generations of programmers have worked on that code under difficult deadlines and filled the code with all kinds of crap.
Very complex pieces of logic, memory management, context switching, etc. are all held together with thousands of flags. The whole code is ridden with mysterious macros that one cannot decipher without picking a notebook and expanding relevant pats of the macros by hand. It can take a day to two days to really understand what a macro does.
Sometimes one needs to understand the values and the effects of 20 different flag to predict how the code would behave in different situations. Sometimes 100s too! I am not exaggerating.
The only reason why this product is still surviving and still works is due to literally millions of tests!
Here is how the life of an Oracle Database developer is:
- Start working on a new bug.
- Spend two weeks trying to understand the 20 different flags that interact in mysterious ways to cause this bag.
- Add one more flag to handle the new special scenario. Add a few more lines of code that checks this flag and works around the problematic situation and avoids the bug.
- Submit the changes to a test farm consisting of about 100 to 200 servers that would compile the code, build a new Oracle DB, and run the millions of tests in a distributed fashion.
- Go home. Come the next day and work on something else. The tests can take 20 hours to 30 hours to complete.
- Go home. Come the next day and check your farm test results. On a good day, there would be about 100 failing tests. On a bad day, there would be about 1000 failing tests. Pick some of these tests randomly and try to understand what went wrong with your assumptions. Maybe there are some 10 more flags to consider to truly understand the nature of the bug.
- Add a few more flags in an attempt to fix the issue. Submit the changes again for testing. Wait another 20 to 30 hours.
- Rinse and repeat for another two weeks until you get the mysterious incantation of the combination of flags right.
- Finally one fine day you would succeed with 0 tests failing.
- Add a hundred more tests for your new change to ensure that the next developer who has the misfortune of touching this new piece of code never ends up breaking your fix.
- Submit the work for one final round of testing. Then submit it for review. The review itself may take another 2 weeks to 2 months. So now move on to the next bug to work on.
- After 2 weeks to 2 months, when everything is complete, the code would be finally merged into the main branch.
The above is a non-exaggerated description of the life of a programmer in Oracle fixing a bug. Now imagine what horror it is going to be to develop a new feature. It takes 6 months to a year (sometimes two years!) to develop a single small feature (say something like adding a new mode of authentication like support for AD authentication).
The fact that this product even works is nothing short of a miracle!
I don't work for Oracle anymore. Will never work for Oracle again!
I think it's important to point this out, because one of the biggest mistakes I'm seeing developers do these days is relying too much on unit tests (especially on "behavior" tests using mocks) and not trying to catch problems at a higher level using higher level tests. Then the code gets deployed and - surprise surprise - all kinds of unforeseen errors come out.
Unit tests are useful, but they are, by definition, very limited in scope.
"... is an indication that the software design isn't (very) decoupled ".
You can be modular without being properly decoupled from the other modules.
We ended up emulating the hardware, run all the software on the emulated hardware, and deploy integration tests to a thousand nodes on AWS for a few minutes it takes to test each combination. Tests finish quickly and it has been a while since we shipped something with a bug in it.
But there’s a catch: we have to unit test the test infrastructure against real hardware - I believe it’d be called test validation. Thus all the individual emulators and cosimulation setups have to have equivalent physical test benches, automated so that no human interaction is needed to compare emulated output to real output. In more than a few cases, we need cycle-accurate behavior.
The test harness unit (validation ) test has to, for example, spin up a logic analyzer and a few MSO oscilloscopes, reinitialize the test bench – e.g. load the 3rd party firmware we test against, then get it all synchronized and run the validation scenarios. Oh, and the firmware of the instrumentation is also a part of the setup: we found bugs in T&M equipment firmware/software that would break our tests. We regression test that stuff, too.
All in all, a full test suite, run sequentially, takes about 40,000 hours, and that’s when being very careful about orthogonalizing the tests so that there’s a good balance between integration aspects and unit aspects.
I totally dig why Oracle has to do something like this, but on the other hand, we have a code base that’s not very brittle, but the integration aspects make it mostly impossible to reason about what could possibly break - so we either test, or get the support people overwhelmed with angry calls. Had we had brittle code on top of it, we’d have been doomed.
Yes, they were not unit tests. There was no culture of unit tests in the Oracle Database development team. A few people called it "unit tests" but they either said it loosely or they were mistaken.
Unit test would not have been effective because every area of the code was deeply entangled with everything else. They did have the concept of layered code (like a virtual operting system layer at the bottom, a memory management layer on top of that, a querying engine on top of that, and so on) but over the years, people violated layers and wrote code that called an upper layer from lower layer leading to a big spaghetti mess. A change in one module could cause a very unrelated module to fail in mysterious ways.
Every test was almost always an integration test. Every test case restarted the database, connected to the database, created tables in it, inserted test data into it, ran queries, and compared the results to ensure that the observed results match the expected results. They tried to exercise every function and every branch condition in this manner with different test cases. The code coverage was remarkable though. Some areas of the code had more than 95% test coverage while some other areas had 80% or so coverage. But the work was not fun.
FWIW, I know people who work on SQL processing (big data Hive/Spark, not RDMBS), and a recurrent issue is that an optimisation which benefits most people turns out to be pathologically bad for some queries for some users. Usually those with multiple tables with 8192 columns and some join which takes 4h at the best of times, now takes 6h and so the overnight reports aren't ready in time. And locks held in the process are now blocking some other app which really matters to the businesses existence. These are trouble because they still "work" in the pure 'same outputs as before', it's just the side effects can be be so disruptive.
Integration tests test those things you're going to get paid for... features & use-cases.
Having a huge library of unit tests freezes your design and hampers your ability to modify in the future.
In my experience it's far easier to introduce testing by focusing on unit testing complicated, stateless business logic. The setup is less complex, the feedback cycle is quick, and the value is apparent ("oh gosh, now I understand all these edge cases and can change this complicated code with more confidence"). I think it also leads to better code at the class/module/function level.
In my experience once a test (of any kind) saves a developer from a regression, they become far more amenable to writing more tests.
That said I think starting with integration tests might be a good area of growth for me.
i.e. Test business logic edge-cases, don't test a linked-list implementation... that's just locking your design in.
In my day to day programming, when I neglect writing tests, the regret is always about those that are on the side of integration testing. I'm okay with not testing individual functions or individual modules even. But the stuff that 'integrates' various modules/concerns is almost always worth writing tests for.
I post this all the time, it's like free upvotes :)
This, and stuff like your story, are why I don't trust people who promote test-driven development as the best way to write clean APIs.
I totally agree, but I met several programmers who think the opposite.
I've experienced myself how the code quality of proper TDD code can be amazing. However it needs someone to still actually care about what they're doing. So it doesn't help with idiots.
The response to every analogy ever made on the internet. Can we stop using them yet?
But at least he wasn't using analogies, huh?
http://comicsalliance.com/scott-adams-plannedchaos-sockpuppe...
>Dilbert creator Scott Adams came to our attention last month for the first time since the mid to late '90s when a blog post surfaced where he said, among other things, that women are "treated differently by society for exactly the same reason that children and the mentally handicapped are treated differently. It's just easier this way for everyone."
>Now, he's managed to provoke yet another internet maelstorm of derision by popping up on message boards to harangue his critics and defend himself. That's not news in and of itself, but what really makes it special is how he's doing it: by leaving comments on Metafilter and Reddit under the pseudonym PlannedChaos where he speaks about himself in the third person and attacks his critics while pretending that he is not Scott Adams, but rather just a big, big fan of the cartoonist.
http://comicsalliance.com/scott-adam-sexist-mens-rights/
>Dilbert's creator Scott Adams Compares Women Asking for Equal Pay to Children Demanding Candy
Hmm, that sounds an awful lot like another analogy to me, actually... Oops!
So maybe Scott Adams isn't the most un-hypocritical guy to quote about the problems of analogies.
The other commenters made me think of the kids game Operation. https://en.wikipedia.org/wiki/Operation_(game)
How about shock collar programming? Or electric fence programming? Or block stacking (Jenga) programming.
Good times.
test passes boss!
There should be integration tests along with some property based tests and fuzzy tests. Usually catches a lot of things.Invest in monitoring and alerts too.
TDD is like relying on debugger to solve your problem. Is debugger a good tool? yes,it is a great tool. But using it as an excuse to avoid understanding what happens under the hood is plain wrong.
The problem lies in industry where software engineering is not given any value but whoteboarding and solving puzzles is.
Software engineering is a craft honed over years of making mistakes and learning from them. You want code asap, kick experience engineers get codemonkeys in and get a MVP.
Quality is not clever algorithm, but clear conscise logic. Code should follow the logic, not the other way around.
Clear > clever.
- reproduce bug and verify your bugfix in matter of ms with proper unit test
- understand what code does
- change and refactor code whenever you want
You can tell from what is written that they are not following TDD. Redesign that codebase in an easy and clean to test design would require an exponential effort and time compared to have it done step by step, but it would be worth it
So let's look at a simplified example.
https://bitbucket.org/iopq/fizzbuzz-in-rust
My tests are in the test folder. They are actually superfluous since integration tests test for the same thing.
I cannot break up the program in a way that would unit test a smaller piece of it in more detail. They only tests I can add would be to test the command line driver
This is especially if your "integration tests" are testing the same component, and not actually integrating with numerous other components being developed by different teams - or, if the system is so small it can run on a single workstation.
Working in teams on larger systems, the situation is different. Part of the point of unit tests is the "shift left" which allows problems to be discovered early, ideally before code leaves a developer's machine. It reduces the time until bugs are discovered significantly, and reduces the impact of one dev's bugs on other devs on the team.
Plus the tests never break on their own because they're modular, and each time you run a test that was obviously going to pass, you've wasted your time.
As long as you have code coverage, better to have lots of asserts and real-world integration tests.
If you unit test properly you are unit testing the business logic, that you have to properly divide and write in a modular fashion. If you want to test a more complex scenario, just add initial conditions or behaviors. If you can't do that or don't know how to do that, then you don't know what your code is doing or your code is bad designed. And that may be the case we read above.
Tests rarely break because they help you not breaking the code and functionalities, and they are so fast and efficient on making you realizing that that you don't feel the pain of it.
I can't imagine any example where "easy to unit test" != simple
Functional tests now, that's another matter. But a lot of TDD dogmatism is centered on unit tests specifically. And that results in a lot of code being written that doesn't actually contribute to the product, and that is there solely that you can chop up the product into tiny bits and unit test them separately. Then on the test side you have tons of mocks etc. I've seen several codebases where test code far exceeded the actual product code in complexity - and that's not a healthy state of affairs.
I remember when i realized that TDD shouldn't have such weight in our development as it had gotten (when it was high on the hype curve).
It was when we starting using a messaging infrastructure that made everything much more reliable and robust, and trough which we could start trusting the infrastructure much more (not 100% though, of course).
It made me realize that the reason why we did this excessively large amount of tests (1800+) was because the fragile nature of a request/response-based system and we therefore "had to make sure everything worked".
What I'm trying to get at here is thar TDD assumed the role of a large safety net to a problem we should have addressed in a different manner. After introducing the messaging, we could replay messages that had failed. After this huge turning point tests were only used for what they should have only been used for - ensuring predictable change in core functionality.
(our code also became easier to understand and more modular, but that's for another time...)
Integration tests easily survive refactoring, on the other hand
The problem with integration tests is they are slow and grow exponentially. If they aren't growing exponentially then there's probably large chunks of untested code. Unit tests suffer their own problems, like you said they can be useless because of a reliance on mocking, they can also be brittle and break everywhere with small changes.
Ultimately any good suite of tests needs some of both. Unit tests to avoid exponential branching of your integration tests, and integration tests to catch errors related to how your units of code interact. I've experienced plenty of bad test suites, many of them are because of poorly written unit tests, but its often the poorly written integration tests that cause problems as well. As with most things, its all about a healthy balance.
Then there are the "write once, never fail ever" tests. Okay, so the test made sense when I wrote the code. I will never touch that part ever again because it works perfectly. Why do I keep running them every time?
I personally run my unit tests every time to confirm my assumptions that the unit of code under test hasn't changed. I also assume all code I write will inevitably be changed in the future because business requirements change and there's always room for improvement. Actually can't think of a single piece of code I've written (apart from code I've thrown out) that didn't eventually need to be rewritten. The benefit of running unit tests is less than the benefit of running integration tests, but the cost of running them is also significantly less. Current project I'm working on has 10x as many unit tests as integration tests and they run 100x faster.
My workflow is usually run my unit tests for the code I'm working on constantly, and when I think things are working run the entire test suite to verify everything works well together. Thats my workflow whether or not I'm doing TDD.
Like, are the two points neighbors? I mean, I'm not going to write a version of this function for a spherical board in the future. Nobody plays on a spherical board.
It's also a really boring unit test. Yes, (1,1) and (1,2) are neighbors. Do I really need to test this function until the end of time?
Programming is a craft. Good programmers write good code. Bad programmers write bad code. No methodology will make bad programmers write good code, but bureaucratic bullshit can and will prevent good programmers from working at their best. The only way to improve the output of a bad programmer is to mentor them and let them gain experience.
Of course, this does require that the process itself has been set up with care, thought, and understanding of what's being achieved.
Some changes are simply not testable, period.
No, you cannot always write a test which initially fails, and then passes when the change is made, and when this is the case. You should understand why that is, and not try.
In some cases when you can, yet still should not. If a whole module is rewritten such that the new version satisfies all of the public contracts with the rest of the code, then only those contracts need to be retested; we don't need new tests targeting internals.
It's because the old version wasn't targeted by such tests in the first place that it can be rewritten without upheaval.
And I agree, that there are lots of anti-patterns that have grown in tandem with TDD, like excessive mocking with dependency injection frameworks or testing renamed identity functions over and over just to get more coverage. However, I'd argue that is equally the fault of object-oriented programming though.
Where I disagree is this: TDD and unit tests are still a very useful tool. Their big advantage is that you can isolate issues more quickly and precisely, IF you use them correctly.
For instance, if I have some kind of algorithm in a backend service operating on a data structure that has a bug, I do not want to spend time on the UI layer, network communication or database interactions to figure out, what is going on. Testing at the right scope you get exactly that.
I'm not keen on the "cult" of it, but if expectations of what the output should look like are available from the onset, it would appear to be of some benefit, at least.
The field of software engineering has matured a lot since then.
There are quite a few very old projects that don't have the same level of cruft as Oracle; it epitomises a Sales Division driven culture.
How many of those switches (that now need to be supported and tested) are because some functionality was promised to a large contract, and so it just had to be done? I would wager a good number.
I can't imagine a C/C++ application where the test suite takes 20-30 hours on a test farm with 100-200 servers. And if you can break 100-1000 tests with a single change, it doesn't like things are very modular and isolated.
And 30 hours between test runs! I would definitely not take that job. That sounds like hell.
Q: Are you busy? A: Yes, in the middle of running tests...
If I submit my test jobs today to the farm, the results would come one or two days later, so I work on another bug tomorrow, and submit that. Day after tomorrow, I return to the first bug, and so on.
Branch from master, and rerun tests before the final merge, like you should in any other software? (Many processes fail that criterion, see https://bors.tech/ for something that gets this right).
Ideally you work on a different enough bug that there's limited interaction, and ideally that's figured out before you fix it, but those criteria are indeed harder to satisfy in a bigger software.
The best way to guess this is to extrapolate from the interview questions. If they ask you a lot of low-level debugging/macro/etc questions..
Further rounds of interviews covered data structure problems (trees, hashtables, etc.), design problems, scalability problems, etc. It was just like any other interview for software engineering role.
When I was interviewing for an SRE position with Google in Dublin, I had about 10min to ask questions in each of the 5 interviews that were conducted on-site.
In between the interviews, a sixth SRE would take me to lunch for about an hour. Anything discussed with him wouldn't be evaluated as part of the interview.
So there was plenty of time for questions, I would say.
Oh ya, the markers in here are pretty run down, let me pray to the old ones for some more"
Forgotten dreams like snowflakes melt on hot dusty ground, soon to turn into hard dry mud beneath a bitter polluted sky.
Wouldn't you just ask the developers interviewing you outright, "can you walk me through an example of your day? How long does it take you to push out code? What's testing like? Do you run tests locally, or use something like Jenkins?" etc.
Which is when I got back on HN regularly :-)
(PS I did tell the internal person that there was no way that reading HN was related either to a BFOQ or other job requirement; and thus while it's not illegal, it'd be highly suspicious.)
What the fuck? Am I a spoiled tech-bro, or does that sound completely insane to anyone else? I would 100% not take a job if I didn't get a chance to talk to my coworkers and future manager during the interview process.
Seems like a trap set up for fresh out of college hires. I don’t know any senior developers who would even consider a job under those circumstances.
I worked for a mini-computer company in the 1980's that ported Oracle (I'm thinking the version stamp was 22.1 when I was there from 1986-1990). It was one GIANT mess of standard "C" with makefiles that were in some ways larger and more complex than some of the actual kernel code it was building!
Took 24 hours to build the product... lol
Enterprise software sales cycles are slow, but once they start turning, it's hard to turn them back.
Contrast PostgreSQL or... uhh... virtually any other database. Oracle's mess is clearly a result of bad management, not a reflection of the intrinsic difficulty of the problem domain.
The executable is ~ 500kb.
Enterprise software is gross.
So... is this good or bad? A Hello World in Go is on the order of 2 MB. That doesn't say anything about code bloat, it just says that Go prefers static over dynamic linking.
This is also a factor for independent developers (who build airline reservation systems) who need to choose a RDBMS for their product - they'll choose oracle, because ... Oracle can provide support guarantees in a way a Postgres contractor can not.
Which makes Oracle not a different class of product than Postgres, but a different class of support for the product. (which could be considered part of the product, so ... maybe you're right)
https://medium.freecodecamp.org/the-biggest-codebases-in-his...
Size of software reflects the number of people working on it (and for how long), not essential complexity.
What more could we want?
The notions of tests written in C that take 20 hours I can't even fathom.
At that point, for DB testing, I doubt it matters what language test are written in, it's going to be mostly about setting up and tearing down the environment over and over.
There is a clear difference between code developed AND maintained in the US vs. code that was developed in India, or code developed in USA and given to Indian developers to manage support. Nothing against Indians, but Ive been around the block and there seems to be a lesser quality of code from that part of the world and companies justifyvit in cost savings.
The actual damage was done much before I had joined Oracle. It appears that somewhere in the early 2000s, the Oracle codebase went from manageable to sphagetti monster. The changelog showed more changes from US developers than Indian developers at that time. Once the damage was done, all developers whether from the US or India now need to follow this painful process to fix bugs and add features.
The category-leading product is probably from one of the earliest companies in the field, if not the first. They have the oldest and cruftiest code - and the manpower to somehow keep it working. It is definitely not the fastest and definitely not the most stable. But they do have the resources to make sure it supports all the third party integrations and features important for the big customers.
I have encountered exactly this same situation on several different fields and categories.
At at time when I was a complete open source fanatic in the early 2000s it suddenly made me realize how Microsoft actually had much better quality software than most big proprietary software vendors.
And yet, even SQLite has an awful amount of test cases.
It's a bit heartbreaking at first (you spend hours/days/weeks working on something, and then a fellow hacker comes and cuts of the unnecessary pieces), but in the long run I'm grateful we do that.
The single hardest thing about programming, I'd say.
Lesson learned. Always write tests. Your business will depend on it.
But on the other hand, over-reliance on tests is one of the reasons they ended up in this situation in the first place. It's like the car safety engineer's joke - How do you make cars safer? Install a knife in the middle of the steering wheel pointed at the driver.
When we're too sure that some safety feature will save us, we forget to be careful.
The test cases are written in a domain specific language named OraTst which is developed and maintained only within Oracle. OraTst is not available outside Oracle. The OraTst DSL looks like a mixture of special syntax to restart database, compare results of queries, change data configuration, etc. and embedded SQL queries to populate database and retrieve results.
I don't know how many more millions of lines of code the tests add. Assuming every test takes about 25 lines of code on an average (every test used to consume about half of my screen to a full screen), we can estimate that the tests themselves consume close to another additional 25 million lines of code to 50 million lines of code.
Not sure if Oracles is already following this or not. But this is necessary for scalable projects.
I have seen smaller versions of what the OP describes. My plan was that every new piece of code that was checked in had to meet certain guidelines for clarity--like can the dev sitting next to you understand it with little to no introduction--and particularly knotty pieces of existing code deserved a bug report just to rewrite and untangle the knots.
In the end, whatever your plan, I think what you need is a cultural change, and cultures are notoriously difficult to change. Any cultural change is going to have to start high up in the organization with the realization that the codebase is an unsustainable POS.
(ASML makes machines that make chips. They got something like 90% of the market. Intel, Samsung, TSMC etc are their customers)
ASML has 1 machine available for testing, maybe 2. These are machines that are about to be shipped, but not totally done being assembled yet, but done enough to run software tests on. This is where changes to their 20 million lines of C code can be tested on. Maybe tonight, you get 15 minutes for your team's work. Then again tomorrow, if you're lucky. Oh but not before the build is done, which takes 8 hours.
Otherwise pretty much the same story as Oracle.
Ah no wait. At ASML, when you want to fix a bug, you first describe the bugfix in a Word document. This goes to various risk assessment managers. They assess whether fixing the bug might generate a regression elsewhere. There's no tests, remember, so they do educated guesses whether the bugfix is too risky or not. If they think not, then you get a go to manually apply the fix in 6+ product families. Without automated tests.
(this is a market leader through sheer technological competence, not through good salespeople like oracle. nobody in the world can make machines that can do what ASML's machines can do. they're also among the hottest tech companies on the dutch stock market. and their software engineering situation is a 1980's horror story times 10. it's quite depressing, really)
Reading a few sentences, looking out at Lanai. Reading a few more sentences, and looking back at Lanai...
15 years later the app was close to half milion lines long of huge bowl of spaghetti code. Only comments in whole codebase were timestamps. I don't know why he dated his code, but I find it fascinating: he never deleted basically anything so you can find different timeframes of when he discovered various concepts. There is use-exception-instead-of-if period, there is time when he discovered design patterns, there is time before he learnt SQL so all the database queries was done by iterating over whole tables and such. I am sure I will find commented Hello world somewhere in the code someday.
I am working on this codebase for 10 years. Code quality improved and major issues get fixed, but there is not enough budget to actually rewrote whole system, so after all it is more or less huge spaghetti monster and I get used to it.
I would say that rewrite would cost about 2 millions of euros. Which is really big price tag for company that use this system as a backoffice tool.
The company is quite fascinating, they started around 1945 and during the years they've became small conglomerate. There are three or even four generations workign together and once they like you as a person they will find something for you to do.
Company did tried to migrate to other software few times, but the software is just too specific for given industry and legislation of small country that the companies who tried to create similar software usually went bankrupt soon.
A spaghetti monster which solves a real business problem can be improved, chunked into pieces, gradually rewritten, whatever improves maintenance. If need be, there will be funds and time for doing so.
By contract, an impeccably architectured, layered, no-design-patterns-omitted, product which solves no business problem .. oh, the horror.
As pointed out elsewhere, this is definitely solving a real problem, the longevity of the app is the proof.
Instead of rewriting, can you replace it with newer idioms? A MVP/PoC for a newer way of solving a problem (AR may be here) that the software solves with some tangible gains, the latter is more important, can lead to approval of a mini budget for that MVP and who knows what that can lead to.
App consist of maybe 20 different codebases that generate around 30 executables, kind of randomly, fetching source files from different codebases as programmer find a fit + random "fixes" of system modules/component make it all very very hard to do much groundbreaking work.
The basic was so old it only had two (yes two) character variables - the Fortran code made liberal uses of Arithmetic IF staements !!!!
An example of one is IF (S12 - 1.0) 13, 13, 12
Excel is an incomprehensible maze of #defines and macros, PowerPoint is a Golden Temple of overly-object-oriented insanity, and Word is just so old and brittle you'd expect it to turn to dust by committing. There are "don't touch this!"-like messages left near the main loop _since roughly 1990_.
I had a bug in some IME code causing a popup to draw behind a window occasionally, and a Windows guru had to come in to figure it out by remote-debugging the windows draw code with no symbols.
I learned there that people can make enormous and powerful castles from, well, shit.
Frequently there are critical code sections where it is much easier to tell people "don't touch it" rather than training people how to work on it safely.
Getting it done > Getting it done properly as far a management is concerned.
Facebook was a spaghetti code mess in the beginning. I'm sure it caused them some growing pains, but moving too slowly early on would have likely been more costly.
Only in startup land which is still a small fraction of our industry.
Most places will never have 10 times the manpower to fix things and are hurting themselves by not doing them properly in the first place.
> Facebook was a spaghetti code mess in the beginning. I'm sure it caused them some growing pains, but moving too slowly early on would have likely been more costly.
Survivor-ship bias, for every facebook how many potentially viable companies never got off the ground because users couldn't tolerate using their steaming pile?
If you've ever seen a software product where something that should take a weekend takes months to get out, it's often not because the problem is more complicated than you'd think, but because of a mangled, complex codebase which prevents anyone from getting real work done.
Edit: Removed a bunch of redundancy.
The long as short of it as a contractor I have to get it done. It will be probably me making the changes later and I make sure I put these things called comments in.
Also developers pretending code quality is an either or proposition is a false dichotomy. You can write 80% of it in a correct manner and the other 20% could be just hacks to get it done in time. You can't write the perfect system.
So I am sorry you are the one being fallacious.
Coworker: "I hadn't enough time to do it right"
You: "Given enough time, how would you do it differently?"
Coworker: "............" (crickets)
IMHO it's not related to deadlines only ; the "not enough time" argument is often a comfortable fallacy, keeping us from facing the limits of our current skills.
I found it to be especially true with testing. I've lost the count of how many times I heard "we didn't had time to write (more) tests". But testing is hard. And when given enough time, these developers don't magically start doing it "right" overnight.
Bingo. It's not necessarily only skills though. It can be myriad reasons and "no time" is just the easiest excuse they can think of. In big companies I've often seen the company process prescribing such bad tools that a good TDD testing strategy is impossible to do with those tools, but they won't move away from them, because somebody in purchasing already bought 10,000 licenses for this bad tool (which is often just a bad GUI, which doesn't really help, except for selling the thing).
The worst tool was a bad GUI where you couldn't even define functions you want to use, that had slow (>1h), non-deterministic, test execution, for a unit test.
In C# this normally means using IOC + DI.
Also almost nobody I know does proper TDD. I know it is very convincing when one of the TDD evangelists shows you how to write something like a method to work out the nth number in a fibonacci sequence using nothing but just writing tests.
In reality most 95% of developers that even write tests write the logic and write the test afterwards to check the logic does what it should.
>In C# this normally means using IOC + DI.
I've become quite partial to functional programming in the last few years. Side effect free functions with functions as interfaces for DI lend themselves perfectly to TDD and data parallel async without worrying too much.
C# is now slowly taking over most of the good features from F#, but I think the culture won't transform so easily.
e.g.
I had to write a bespoke popup window launcher for a large gambling company in the UK. The games were mainly the awful slots games that you see in motorway service stations. These are basically one arm bandits on steroids.
There is a lot of logic that was in JavaScript that should have been in C# and I had to design it correctly to work with a third party Proprietary CMS system and I had to manage session tokens on 3 to 4 third party systems. Not easy.
It took me about 2 weeks of just reading the code and absorbing it, drawing lots of diagrams of how data flowed through the system and then porting that logic over to C# in a way that would work with the CMS system in a logical and OOP fashion and handling auth tokens effectively.
If anyone feels that there is enough time then (depending on the position dev/manager/client/etc) he starts slacking, moving focus, moving deadlines, moving staff, demanding more features/support/documentation or new requirements analysis, start being pissed more about smaller bugs.
Related to that is the "obvious performance fix" that doesn't perform faster that keeps burning up time for years long after it was proven to not be faster because freshers never found out about it and the oldsters forgot.
But also unlike your coworker, I probably _figured out how to do it right_ in the time given. It's pretty rare that I don't have time to do it right; it does happen (especially with extreme instances of scope creep and requirements drift), but it's rare.
Which I guess is your point? The time excuse is just an excuse, and a good developer writes good code.
For example, code running in JVMs on top of non deterministic operating systems sometimes behaves in really odd ways. Sometimes a main loop is stable for reasons nobody understands.
http://blog.zorinaq.com/i-contribute-to-the-windows-kernel-w...
> ...remote-debugging the windows draw code with no symbols
Why, specifically, were no symbols available? I can't come up with an explanation. Surely old symbols are kept. Do checked builds take longer to iterate on (ie build), or something?
Office and Windows are different teams and units, so one dev on one team typically wouldn't have access to all of the symbol info for the codebase of the other. Setting that up takes some hoop-jumping, so he tried without and ended up figuring things out just fine over a few hours.
What I wanted to demonstrate was that in that moment all he had available was shit, and he still managed to push the castle higher.
I must admit, I do very much wonder what kind of environment Microsoft would be if teams were less segregated. I found http://blog.zorinaq.com/i-contribute-to-the-windows-kernel-w... in the comments, which seems to hint at the same sort of theme somewhat - particularly the bit about contributing to teams other than your own. There's a strong notion of isolation.
This is just thinking out loud, a response is not required. Everywhere has pros and cons. I'm (even with all this moping) actually less hesitant about MS as a whole than the rest of FAANG (except for N, which I also don't see a problem with) - not because of the whole "new MS" thing, or GH, but because everyone else seems to have fewer scruples than I consider to be a viable baseline. So there's that. :)
It's just kind of sad to see these kinds of inefficiencies, and it would be cool to eliminate them. Of course, it'd unleash organizational chaos for a while, but of course it would be totally worth it.
https://news.ycombinator.com/item?id=15745250 ?
It appeared that a bug in an office component was fixed with manually binary editing. Is that probable?
The first thing I did was chuck it into VS2010 and run some code metrics on it. The results were, 10 or so Methods had 2000+ lines of code. The maintainability index was 0 (number between 0 and 100 where 0 is unmaintainable). The worst function had a cyclomatic complexity of 2700 (the worst I have ever seen on a function before was 750 odd). It was full of nested in-line dynamic SQL all of which referred to tables with 100+ columns, which had helpful names like sdf_324. There were about 5000 stored procedures of which most were 90% similar to other ones with a similar naming scheme. There were no foreign key constraints in the database. Every query including updates, inserts and deletes used NOLOCK (so no data integrity). It all lived in a single 80,000 line file, which crashed VS every time you tried to do a simple edit.
I essentially told my boss I would quit over it as there was no way I could support it without other aspects of work suffering. Thankfully it was put in the too hard basket and nobody else had to endure my pain. I ended up peer reviewing the changes the guy made some time later and a single column update touched in the order of 500 lines of code.
There was one interesting thing I found with it however, there was so much repeated/nested if code in methods you could hold down page down and it would look like the page was moving the other way, similar to how a wheel on TV looks like its spinning the other way.
Hahhahah. That is the funniest shit I have read in a long time!
Meaning I capture all the I/O and recreate it. SQL, HTML, PDF, CSV, whatever. Serialize everything, before and after. And then do diffs on the outputs to see if new code reproduces expected behavior.
Much easier than dead code removal, code deduping, incremental changes, backfilling tests, etc.
Once someone captures, documents what the code is actually doing, the real refactoring begins. Removing unnecessary queries. Grooming data. Simplifying schemas. Aligning the app with the biz. Etc.
It’s my default approach with thorny scientific code: get everyone to agree on what output should be for a bunch of relevant inputs, get the original systems output, hash it, and then write a bunch of tests in a new project that all assert that these inputs produce outputs whose hashes are as follows..
then never look at the guts of the horror again.
There were two ingredients in the recipe for disaster. The first is that Lua comes "batteries excluded": the standard library is minimalist and the community and set of available packages out there is small. That's typically not an issue, as long as one uses Lua in the intended way: small scripts that extend existing programs with custom user logic (e.g. Nginx, Vim, World of Warcraft). The second is that Lua is a dynamic language: it's dynamically typed, and practically everything can be overridden, monkey patched and hacked, down to the fundamental iterators that allow you to traverse data structures.
This was the playground for the guy to create his own reality.
Lacking a serious standard library, he crafted his own. Where a normal world e.g. file rename function would either do the job or return a error to the caller, he chose a different approach. Functions were autonomous highly intelligent pieces of code that tried to resolve every possible problem, entangled with external logic, so grokking the behaviour of the most fundamental things was challenging - let alone understanding fragments of code composed of library calls.
Lacking a OO model in Lua, he built his own. I can spend a lot of time describing with what was wrong with it, but it suffices to say that each object had SIX different 'self' or 'this' pointers, each with slightly different semantics. And highly entangled with external unrelated logic of course.
I'll save the stories about the scheduler and time series database he built for another time.
Queue the Neil Gaiman sandman styled comic about his dark adventure
To be fair, I've also seen it happen in C, C++, and JavaScript.
Part 2 please???
For replacing nginx with lua, that is curious. Openresty is not being used? No web framework? (lot of lua dead web frameworks in the road, by the way)
Also, "creating your own reality" is usually a bad thing in any language. That usually happens when you're developing something alone, for long and didn't give maintenance to other peoples code much in the past.
In another related point: lua is NOT supposed to be used as glue/extension code. It was designed so that it would be easy to do so. It is that "you can" more than "you should".
Lua doesn't come with batteries included, by design and doesn't have a plethora of libraries available or (coff coff) easy to pick from, but, they do exist and they do solve most problems. Nonethless, truth be told, most of the good api for Lua one sees nowadays, was not available 1 or 2 years ago or not mature enough.
To conclude, lack of type hinting in Lua or optional static typing can create problems for bigger problems if good design, testing and documentation is not enforced from day 1. Most scripting languages suffer from this. You guys could try "ravi" to get this (almost) for free.
The only maintainers of this code, ever, have been grad students and postdocs. I estimate there have been about 12-15 generations worth. This code has supported hundreds of publications in its lifespan.
A codebase that began life in 1987, in C. First ported to matlab in 1999. First source control was added (as SVN) in 2015. Between 2015 and 2018, there were 6 commits total, yet 3 people graduated out of the lab from it. Probably 100,000 loc total, of which I estimate maybe a third is ever used. 1400-line matlab functions are normal-ish. I've found loops nested 11 levels deep.
It's a series of psychophysical experiments. Each experiment exists in at least 4 different versions side by side in source, each named slightly different, often by incorrect datestamp of last modification. Version control across machines is not well maintained, so you have to diff everything before you can copy or move files lest you accidentally blow something away completely.
Oh, and it's mexed and wrapped for use on a mac on exactly one snow leopard machine, hardware from 2007.
edit: I think this counts as a job, not a student experience, because I am not a student. I just have to clean this mess up once in a while.
It's a little more work sometimes, especially when your experiment changes things structurally, but it pays off over and over.
There's a great article somewhere about how the normal version control flow doesn't really work for this style of computing.
You want to keep both "versions" of code live and active in the same place at the same time (often in the same notebook).
People end up with methods named methodName, methodName2 etc, which isn't very good. But once you see the workflow you understand why normal version control doesn't work either.
There should be a solution to this, but AFAIK there isn't yet.
By all means, branches are great for super-prototypey early code, but once you know that you want to keep the ability to run the experiment around, guard it with a flag and merge it into mainline to avoid nightmare merges later!
Now add on the fact that different teams had varying levels of frontend competence. This led to some webapps being in (badly written) React, some in Angular, some in JQuery and one in Angular2 as well. Some were java API backed, some were NodeJS backed and one used Ruby in the backend. Oh yeah, and each had different datastores as well.
Now alongside this, there's no central auth framework, so each webapp had their own way of how to determine user auth (there was a shared cookie thankfully on the *.company.com domain), so there were 6-7 possible login and logout pages as well.
When the company had a brand redesign and needed all customer facing stuff to re-align with new design, we had to literally re-design 6 dashboards which have used different paradigms and different tech stacks.
To this date, the dashboard still exists and is used by customers (Main dashboard for a company valued at $500M+) and the most common issues are issues related to Auth (eg. random logouts when switching tabs), data inconsistency (They have crons which update data from one DB to another, but it's not immediate) and an inconsistent design and UI behaviour (since JS is also different for each app) which pisses off many users.
Till date, I'm not sure who signed off on this pointless dashboard design.
0: https://www.quora.com/How-is-JavaScript-used-within-the-Spot...
Software is a mess. I've seen some freakishly smart people capable of solving very hard problems writing code that literally changes the world at this very moment. But the code itself is, well, a castle of shit. Why? Is it because our tools (programming languages, compilers etc) are still stone age technology? Is it because software is inherently a harder problem than say machines or chemical processes for the human brain? Is it because software engineers are less educated than other engineers? .....?
I honestly sometimes do wonder. Judging by the elevators in my building some things even barely work at times. They go down often and require parts replacements that take weeks to arrive. They're barely 3 years old.
Isn’t that one of the two main features of the product?
In fact, you could build the exact elevator almost identically. Software is never like that — it’s dynamic and constantly changing throughout the life cycle of the software.
Ha! As a mechanical engineer, no, nothing in reality is ever precisely defined. Consider that every single part in the elevator has a tolerance: none of the parts are exactly the same in every elevator. Did you account for thermal expansion? What about wear? Fatigue?
> Software is never like that — it’s dynamic and constantly changing throughout the life cycle of the software.
But elevators need to be shipped right the first time and can't make mistakes; with software, it almost never works perfectly no matter how much time passes.
There is a person inside me -- whenever this is said -- that wants to shout "No! You can prove correctness of your program!". But this complicates the issue even more, since afaik no elevator's correctness is proven but it just works. Mechanical stuff somehow just magically work without proof whereas it's still debatable if proven software works (did we prove the software that proved it?). I don't know what's the difference. Maybe elevator has a well-defined structure, it goes up and down, opens the door etc, even though implementation can change (as you explained). But maybe software isn't like that. Idk.
Software is... not.
It might seem like magic, but, of course, there is actually a science to it. Depending on the situation, a part can work if a certain length is 10.000 or if it's 10.001. In software, that is never the case: a value that should be 10 but is actualy 10.001 can stop everything. In engineering, there is a limited amount of leeway at every step of the process, and everything is slightly overengineered by some factor of safety to ensure that this is the case. For example, if an elevator cable is rated to hold 10,000lbs, the elevator will be sold as having a maximum capacity of 8,000lbs. In correct usage, (weight less than 8,000lbs), there is a sufficient factor of safety on the cable so that it won't break even if the cable is cracked, worn, etc.
What should we make these buffer sizes? There's not really a right answer, but there are definitely some wrong ones. Make it big enough to handle the expected use cases and pad a bit extra. Works exactly like a tolerance in physical engineering.
Building things is really hard. I don't think software engineering is in any way special. It's just that the problems we solve with software are often more complicated than the problems we solve in meatspace and failure is much more benign so that QA is not done with as much diligence.
E.g. your cable is rated for 10,000 kg and 6-month inspections, your driveshaft can go 5 million revolutions or 9 months before it needs to be greased...test all those things separately, put them together and you have a stateless system that if it works today will work tomorrow.
Solving complex stuff takes a lot of effort, and usually most companies do not have the resources or the motivation to go back and rewrite a product.
In my experience, code needs to be re-factored 3 times before it starts to look coherent and clean.
But 99% of projects can't justify that kind of expense.
But good code is very feasible, just takes time.
After all, if work inside the codebase does not make a difference to the people who use the software, in terms of reliability or features, is it really worth doing?
The trick is to find a balance where efforts to improve code quality actually improve outcomes for the business and the users -- otherwise it's not justifiable to take time away from feature development, which is what the users actually care about.
When they attempt to build something new, it often ends up like software – tremendous overruns in both cost and schedule.
C.f. the only new nuclear power plant being built in the West, which is $4 billion over budget and ten years late: https://en.wikipedia.org/wiki/Olkiluoto_Nuclear_Power_Plant#...
The reason why this feels more commonplace in software is that we're usually designing something new. Software has essentially no reproduction costs, so there's no reason for anybody to design software that's a carbon copy of something you can already download or buy off-the-shelf. That's not the case in engineering of physical products or works. New buildings are needed all the time, even if they're performing exactly the same function as the building in the adjacent lot.
Exactly this. I like to use an analogy of building bridges :)
- OK, we have this valley and we want to drive cars over it. What do we
do?
- Hmm, we could just make the road along the bottom of the valley
- Wouldn't that cause the cars to fall because of the steep angle?
- Good point. Maybe if we built the road with a lot of twist and turns
- Then it's a slow and long drive.
- Maybe we could build some sort of catapult setup to throw the cars
across the valley
- That would make safety a concern though. Is there a way we could use
helicopter rotors to suspend a road in mid air over the valley?
- Or what if we attach the road to each side of the valley and make a
road that's strong enough to not crack in the middle under its own weight?
- Yeah, that sounds like a good idea. How do we make sure it can hold its
own weight, though?
- .....etcI must admit that when I saw valley in
> OK, we have this valley
I immediately thought about SV, and then as I read
> - Maybe we could build some sort of catapult setup to throw the cars > * across the valley*
I imagined some kind of scenario happening in an alternate reality where a startup was trying to dream up a viable way to actually achieve this in SV.
It was funny because of the fact that a lot of the ideas people do come up with are probably ideologically very similar in magnitude and ridiculous impossibility.
This was played for laughs by German satire news website "Der Postillon" (like The Onion, but in German): "Department of Traffic to replace ramshackle bridges by jumping hills"
Nicely photoshopped picture at: https://www.der-postillon.com/2014/11/lander-ersetzen-marode...
Programming also involves things like "formalizing and automating the representation of knowledge in general" which is a holy grail of philosophy since Leibniz's time.
We're always building on top of preexisting ontologies and logics which sometimes fail to even make sense, or which make it tedious to express things that we would want to consider elementary (Unix, TCP, Java, GTK+, SQL, etc).
And we're always vulnerable to being smacked on the head by yet another "Falsehoods Programmers Believe About X" post detailing the myriad ways in which we routinely falsify and oversimplify to deal with the boundless complexity of actual reality and the horrendous details of legacy bureaucracy.
If you hit something with a hammer the result and response is instant. You can feel the nail went deeper before you look, you can hear you hit the head correctly before you think about it. You can feel the vibration and hear and feel and see the wood crack before you could read a sentence about it.
You don't have to compile the hammer and run the hit and read the result from the screen and if you logged the right things you will see something about the result based on what you logged but not quite everything because that would be an unintelligible mess. If you compiled the right version of hammer that is.
You don't even know for sure if you are holding a hammer. Of course that's not called a hammer but a unique tool you downloaded because they said it is the new best tool of the year - and there are several famous new tools every year. Though you cannot be sure if that tool helps in your job until you try to use it as you don't know if you are using a hammer or an excavation machine and both can be suitable for the task one being slightly bigger though.
EE here. It's not. The entry bar is much higher: most of the time difficult projects are given to senior engineers. It takes a degree and many years of work to get there.
> Or chemical engineers synthesize medicines in way nobody but a rockstar guru understands
Been there. Not at all.
> cellphone is made by machines designed in early 1990s because nobody was able to figure out what that one cog is doing
Same. Part of the reason is that any physical component adds cost, weight, and so on to a project.
In software you can throw code and random libraries at a problem and nobody will notice until vulnerabilities start to pop later on.
Maybe it's a good thing that I'm not an engineer.
All elevators accept a standardized fire/security key that enables "exclusive" mode that lets you tell the lift "go to this floor" and it will do so: https://www.youtube.com/watch?v=1Uh_N1O3E4E (a reasonable sink of 1hr)
As for traffic lights I'm aware (at least in Australia) that they all tie back to a realtime system that lets a control room instruct lights to turn green and so forth. (Presumably the immediate action is that the lights that are currently green immediately turn orange, then red after the normal delay.) You can get summarily fired for misusing this though. I've also heard (IIRC on a TV documentary-type show) that ambulances tie into this system with live GPS tracking such that lights turn green as they approach, but this is sufficiently fantastic that I'm waiting to trust-but-verify it before I quote it with confidence.
Under normal use some traffic lights are traffic based while others have timers. The ones that are traffic based will take into consideration whether the pedestrian crossing button has been pushed and reduce the threshold value needed for the actual traffic lights to turn red. The other timer ones - like the really annoying ones outside my local mall :) - are on a fixed timer, and I try to run for that one when I know it's about to turn green, or just accept and twiddle my fingers while I wait if I miss it.
It'd be nice if elevator simulators were satisfying to write. Although, actually... I've just realized that with Unity you could have _quite_ a lot of fun :D... (hmm, I've had GPU on my wishlist for about 15 years now, now I really want one)
This despite the fact that if additional time were allocated to come up with a better solution, tomorrow's software would be easier to integrate on top of today's, and with a higher quality of code.
But the reason that managers behave like this is because they can - and this I think has more to do with the ethereal and timeless nature of software than anything else.
In the real world it simply is not possible to put together a physical artifact with the sort of compromised constructs that appears in software.
There are costs of material to consider; costs of production. There's degradation over time.
All of these aspects of physical constructs provide a strong motivation to produce a quality product - not out of beneficence, but because (quality engineers aside) it's cheaper and it's really the only feasible option.
Incidentally we had a small falling out with this company, and they were refusing to update their executable until this issue was resolved. This looked like affecting some hundreds of schools and their timetables. I did some checking, and it turned out their 'non-updated' executable was doing a simple date check on the local PC; if it was past a certain date, the executable refused to run. So I did a quick hack in our application that involved: - setting the local PC date to prior to the 'cutoff date' - running their executable with the required parameters, and grabbing the results - setting the local PC date back correctly
This led to interesting negotiations as they were puzzled why their 'gun to our heads' no longer appeared to be working, and things were resolved to the benefits of both parties soon after.
The largest madness was a J2EE mess where persistence was achieved by taking your current object, sending a message bean to the server, it would touch the database and return the result which was being polled for (making it synchronous). The amazing thing is that the client and the server were the same J2EE instance. So Class A could have just called Class B. Instead it was A -> turn it into a message bean -> send it to the "server" (same machine) -> unwrap it into A again -> transform it into B -> message bean it back to client -> unwrap into B.
Literally three months of 8 people ripping all of that out and replacing it with A.transform() // returns B
Oh, and at the same time, none of this was in source control. It was being worked on by 12 people up until that point. They didn't know who had the current files which were in production. So my first job was taking everyone's files and seeing which ones would compile into the classes being used in production, then checking those into source control. Up until then, they just kept mental track of who is working on which files.
The java mess was a pattern copied from within the company where it was used correctly. The main product of the company was built around some very complex scheduling software. It was a black box and communication was entirely through inter-server beans. This was copied to intra-server, which makes no sense at all, and it was used everywhere.
But my most auspicious entry to a company was where the guy I was replacing had been arrested for stealing data from a competitor. My first two days were spent recovering logs, downloads and other traces of what he had done and handing it over to the lawyers. (the circumstances were not mentioned in the hiring process)
and
"Oh, and at the same time, none of this was in source control."
Owwww, that's painful! 12 people spaghetti Perl that did its DB lookups via Java and non of it in source control?
Owwww, owwww, owwww....
What was horrible about it was that it controlled everything from who got a website, active domains, what POPs users could dial into, metered billing, you name it. And it did all of this by manipulating flat files of pipe-delimited data on a central server, then rcp’ing those files to the various machines, then rsh’ing to the various machines and kicking off THEIR scripts, which parsed the source files and generated their own files, which called another set of scripts that parsed THOSE files and generated the software config files.
This included doing things like updating init scripts so that new IPs got added to interfaces, and what email server a user was provisioned on, so it had to generate new exim configr with routing rules.
All this to say that it all worked, but I dreaded having to go in to manipulate anything. Adding a server at least had a dedicated procedure so that was fine, but anything else was a nightmare.
Case in point - as part of a gradual plan to remove this nightmare, I swapped out the radius server that they were using for one that could support a database backend, and modified the local config generator script to make a new config for the new software as a stopgap until I could get it into a database.
The config file had a series of fields that just had numbers in them, and after much digging, it seemed like that controlled whether a terminal dial in user was presented with a menu of options, and what options. I had to reimplement that logic for the new software, made a mistake, and accidentally removed the option for UUCP for the 10 customers that were still using UUCP. One of them was on an ISDN line and their mailer decided to continuously redial looking for the UUCP, tacking up thousands of dollars in carrier rate charges for the weekend that it took anyone to notice something was broken.
I got given an IDE that was written in korn shell to maintain. Not as mission critical as this sounds, but was the only way to edit, compile, link and deploy around 6000 COBOL programs that made up a very large and expensive financial services platform. It also integrated with the SCM (unix RCS!), did checkout, checkin, merging, branching and all manner of amazing things.
There was probably 30 devs who used it, all running on a HPUX server.
It was very powerful, but a total nightmare to look after.
The rewrite was 100 lines long.
That made me LOL.
I was called in because, while most of the application worked, some of the requested features were not yet complete. When I made my initial recommendation (scrap the whole thing and start again) I was told the client's board would not agree to that because of the money already invested and the fact that the board had seen a demonstration proving that "most of it worked".
It took two developers eighteen months to beat this sorry mess into a maintainable state while ensuring it remained "usable". It would have taken one third of that time to rewrite it from scratch.
I had a similar experience with an offshore company. This was during the early-mid 2000s, at the height of the offshoring era when $10/hour programmers in India were aplenty and everyone felt their job was soon to be outsourced. Turns out, no joke, they were being paid per line of code.
Despite having to maintain the heap of crap, I was amazed at the brilliance of their maniacal dark-art ability to implement as little functionality with as many lines of code possible. It was like code golf in reverse.
function isTrue(v) {
var result;
result = v;
if (!isFalse(result == true)) {
return result;
} else {
return isFalse(result);
}
}
function isFalse(v) {
var result;
result = v;
if (!isTrue(result == true)) {
return isFalse(result); // Tail recursion so this is fine.
} else {
return result;
}
}
if (isTrue(myBool) == true) { // TODO: Could this be refactored to isTrue(isTrue(myBool))?
return true;
} else if (isTrue(isFalse(myBool))) {
return false;
} else if (isFalse(isTrue(myBool))) {
return false;
} else {
return isFalse(isFalse(myBool));
}There's also a bug on line 7 that I didn't notice - it should be isTrue rather than isFalse. I can't edit the comment anymore to fix it.
As horrible as this sounds, this is actually good from a refactoring point of view because it should be straightforward to rewrite it to use actual queries.
* It was over 20 years old by the time I started
* It was written in Fortran
* Variable names were single and double digits
* Each fortran program would run in isolation but had a shared memory process
* It was formally a terminal program but a weird Java frontend was created so everything looked like Windows GUI
* All program names were four letter acronyms
* All data was stored in fixed width binary "flat" files
* It previously was under CVS version control, but each install slowly drifted apart, so each site had it own unique features and bugs.
* I once had to move a significant feature from one install to another using only patch files generated from the work done on the original install.
I would like to say "cue the stereotypes for Indian developers" and we could all have a good laugh. But no. This is more like Heart of Darkness. They must have traveled to the darkest corners of the subcontinent to find a mind capable of the eldritch horrors we found there. We started keeping a wiki of design patterns, to save us WTF time. Here are a few.
* Memory management. What's that? This site required 16 GIGABYTES as a PHP memory limit in order to load the front page for an anonymous user.
* Security. What's that? Part of the reason it required so much memory, is that it would include a complete debug log of the site's entire transaction history, including PII and credit card numbers, in every session. Meaning any vulnerability was a critical vulnerability.
* They would arbitrarily include the entirety of Zend framework to access a single simple function. This happened several places in the codebase, with different versions of Zend committed.
* Can't reach the ERP to get the price for an event? Let's set it to $999,999.00 and proceed with the transaction.
* Invoice numbers were random numbers between 1-1000. Clashing numbers meant a fatal exception that would fail to store the invoice or payment record... But not until after payment had been processed. Birthday paradox means this happened a lot.
* The developers used arcane bits of Drupal API to do totally mundane things. Like, if you know about hook_page_alter, you know there's a setting in the UI for the frontpage. But we'll just use hook_page_alter instead.
* Write queries using Drupal Views, rewrite the query in code, override the views query with their custom (identical) version using an unusual API hook, just to add a sort.
I could go on, but I think you get the picture. Eldritch horror of a codebase.
> * Memory management. What's that? This site required 16 GIGABYTES as a PHP memory limit in order to load the front page for an anonymous user.
On one infamous occasion we were making a, relatively small, patch release. The debug version worked fine but the release version crashed systematically. Even when we backed out all the changes we had the same behaviour. We were screwed.
Until one of the team had a bright idea. She stripped strings from the debug build and tested it. To our surprise it not only worked, it was only slightly bigger than the previous release version and it was also slightly faster! We shipped.
This experience was the trigger to make me go all-in on a full re-write that I had been contemplating. One of only a couple of times in my career that I've made that decision on a major piece of software.
The re-write was a huge success. It was also about 10% of the original in terms of LoC. The day our testing finished, we held a ceremony where we deleted all the old code from the current version.
This caused a slightly different issue. At the time, code metrics were starting to get fashionable but LoC wasn't yet the pariah it became.
So, a couple of days later I got a concerned call from the metrics guys. Apparently, we had deleted more code than all the other teams combined had added in the previous measurement period. This caused their metric calculation to barf. Their solution? We should add all the code back in! This led to a somewhat heated argument that ended up with me persuading them that deleting code was good and they should, at least, abs(LoC) it. It didn't make the metrics any more useful but meant that we had an application we could reason about. Happy days.
ah yes, when bad metrics become targets
Debug versions would never get compiled, as I'm told the resulting file was too large for the filesystem to handle.
Apparently a great deal of the code was actually inline XML.
They knew this was a bad pile of technical debt, too: at one point, a senior/staff engineer gave a company presentation where they brought a fanfold printout of the main header file for this monstrosity, and literally unrolled it across the entire stage.
The architecture looked like something from the Early Java days.
The system used Iron Python / C# in the following way.
1. A web request would hit the CMS 2. There was a massive switch statement to work out how the query would be rewritten 3. If it the url was prefixed with processor it would attempt to find the processor in the db. 4. The code would then find the python script associated with the processor. 5. The processor would then spin up a command line instance in a hidden command window on windows server. 6. The processor would have to return XML that had to be built up using strings (not element tree for you). 7. This would return the XML to the C# which would then try to render into the XSL Transform.
If at any time this failed. Silent failure. There was no way to debug easily (There was a magic set of flags that had to be set in Visual Studio or wise you couldn't debug the python scripts).
To get the software to build on a new machine. It took a contract 4 months to reverse engineer an XP machine. None of it was documented anywhere.
It used ImageMagick to generate thumbnails on the fly which doesn't work to well with windows server.
The lead engineer was an alcoholic. He used to go to the local pub for 4 hours in the middle of the day and come back smelling like a brewery.
I like my beers but this guy was on another level.
Uncovering what the hell was actually happening was like peering into the mind of a psychopath.
Once myself and 3 other engineers spent 4.5 hours trying to figure out how to send an email from this treacherous app (a feature which had stopped working months before we showed up). After those 4.5 hours, none of us even came close.
To top the whole thing off, page load times were in the minutes. The standard joke from the users was that they "get to take lots of coffee breaks."
Much to the (new) manager's credit (and my sanity), we got buy in to just build a new version and let the old one die. I left the team shortly after the 2.0 beta launch, but God I wish I would have stuck around a little longer so I could have seen the official end of life for that calamity of an app.
I've worked with lots of proprietary CMS systems and I kinda got into a groove with working with them. I knew exactly how to manipulate the system within the parameters of said system.
I kinda found it challenging. When a system is well built. I find it boring.
I had never written in LISP before, so I bought a book!
I was having trouble reading the program on the computer, so I printed out the roughly 15,000 lines. Not a lot, I know, but it was about an inch and a half thick stack of paper. I started going through it.
It consisted of LOTS of subroutines. Thousands. Each one neatly formed; no more than a couple hundred lines. It gave me hope.
It read in the text file, created a blob of a string, and then passed that blob to the first subroutine. Then it passed to the next subroutine. And then the next, and so on. As far as I could tell, it never called a subroutine a second time, and it never returned to the starting method. Given the strangeness of LISP, I couldn't figure out what it was doing, or why.
The guy who wrote it had retired, and we didn't get along anyway, so I didn't try to chase down what his thinking was.
I gave up.
To my knowledge, they're still using that program 17 years later.
I suppose you were referring to the statement "Customers in that space don't have many options". Your statement is reasonable and there might be market opportunities to create new software. But I somehow have the feeling that it would take a tremendous amount of time to recreate something that checks the same boxes as the old system. But I do not work in that sector so my judgement might be totally off.
I agree it is an oppurtunity, but the barrier to entry is very high. Reaching feature parity is a multi year project with a large team and domain experts.
There's a lot of engineering that goes into/went into mainframes.
Now I've put it in perspective, I went on to an intern position between years 2 and 3 of uni. I was handed a lovely piece of code which had:
- Around 300 classes - 3 or 4 layers of nested generics - Factories, factory factories, generator factory factories - 90% of the parameters were passed in from the build engine running the code, so it was impossible to run locally, ever. - 0 tests - Some 100 pages of documentation, which had been lost until I was about halfway through my placement (and mostly documented how to set up and run it, not how to maintain it)
Seriously, this thing was designed to the extreme, made to be generic to every single scenario in existence.
So what did it do? It took items from a customer facing system, transferred it onto the internal work tracking system. Then when they were updated in the modern system, mirrored the relevant updates back to the legacy.
The best part? Every time the internal work tracking system updated (once every 6 months), this thing broke horribly and it was practically impossible to fix. Even if you managed to set up stuff so you could work in a development environment, it still connected to the customer facing system, so you had to be incredibly careful what you did during testing.
It wasn't the biggest in terms of LOC, but it astounded me just how much effort (apparently the guy who wrote it squirreled away for a year to write it, then moved to Canada, and was famous in the department for having one too many beers during a outing to a local Indian) went into designing this behemoth.
I still have the occasional nightmare about it!
It was a huge folder (not repo - and there were zip files of different “versions” of the code in there). The main monster was a huge Visual Studio solution with hundreds of targets, one would be an application for entering some data, the other was for entering data from a hardware device (a scale if I remember right), etc.
The main source of truth was an MSSQL database to which all these apps would connect as root. There is no backend as such to ensure access control & consistency, and any misbehaving app could essentially trash the entire DB.
Database credentials were hardcoded in every app’s main entrypoint, with earlier “versions” of the credentials commented out below.
I thought that surely these must be either staging DBs or at the very least there would be network-level access control meaning the DB wasn’t accessible from outside... but no - I managed to connect to their production DB as root from a random, untrusted location. I do not know if MSSQL uses encryption by default but I would bet good money there was none and they were essentially connecting to their DB as root, over plaintext, from hundreds of different locations across the country without any kind of VPN.
In terms of code you obviously have your standard & expected “spaghetti monster” with UI & business logic scattered everywhere. What struck me the most was an empty exception handler around the main entrypoint.
In the same folder there was also source for an iOS app. Didn’t look at it but I don’t see any valid reason why this should be in the same place as the Windows apps.
Thankfully I no longer work there and even if I were I had no major C# experience (which gives me a very convenient excuse not to touch this mess).
A true monorepo
Are you me? :-)
Had almost the same experience, minus the database. Friend of the owner wanted to buy a company and asked us to evaluate their code to see if it was maintainable enough to add new features. I got a zip of hundreds of firmware projects each representing a different version. They were all on the same basic platform but with different hardware features #ifdef'd, or customized for a particular customer. The code itself wasn't that bad (not that good either!), but their developer clearly had no idea what Version Control meant.
In the end I gave the thumbs up and he bought the company, then ended up having to redesign the product from scratch since much of the originally designed-in components were no longer available. He did his Due Diligence for the software, but ignored the hardware side!
The software was largely a standard base with modifications done for the individual customer to suit their business needs.
In this particular project, it was a batch sortation, meaning we received a large batch file from their mainframe, which would then be parsed and executed by the controller software.
Everybody feared this particular project, and estimates on new functionality were sky high, but I didn't think much of it until i had to modify the batch parsing code. I was met with 22000 lines of C code in a single function. This single function was modified multiple times each year, usually adding more code in the process.
It took me the better part of a month to refactor into "manageable" chunk sizes, and in the end i was left with a 1700 LOC function that was still "too big", but nobody really understood how it worked, and we couldn't test it, so i just left it at that.
After my refactor could implement new functionality somewhat faster, but in the end it was still a very complex algorithm, so despite being in smaller functions, you still had to be very careful when modifying it.
Previous job. Was sysad. This code runs most of the academic US internet.
Everything was Perl5. There were over 150 different major applications written with it, ranging from 40 lines to 500k lines. The older the recent commit, the worse. Touching any of this would cause errors, either in itself OR in associated applications! You'd be working on thing A, get it working well, and 4 weeks later thing B would fail horrendously.
The worst was a tie between a variable that was a 6 line long SQL query, which was packed to the brim with function calls that ended up expanding a query to something like 50 lines long.
The tie was a gem in code that wasn't touched for 7 years. This wasn't at the top, or even in a config file. It was hardcoded in the middle of the perl5 program...
$db_server = (servername);
$db_user = root;
$db_password = (password);
1. No primary keys for the main database....
2. Goober had the idea of storing pictures in said MySQL database. 70 GB of pics...
3. Redhat 4 still in production, along with RH 5.
4. Its everyone for themselves. The goal is you hobble it along enough for the next oncall. Let them get hit with it.
5. Running iron from 10 years ago. Contracts pull in $$$, but you're dealing with paleo-datacenter crap
6. Just retired LTO3 tapes. Now they have "shiny" LTO5....
Now, I work in a heterogenous Windows/Linux shop with non-paleo hardware. We're even deploying on AWS in a limited fashion.
This place has its warts too, but everywhere has something. But that previous place... I'm surprised it still hasn't crashed and burned. Their networking was solid tho. Just anything with Linux was a tire-fire.
And as one more gem, there was a program that terrified the fuck out of me. A fellow engineer showed me this update tool that would update a router remotely. Great. Well, if you add the -n (customer) flag with the customer number, it would update all the routers for that customer!
It was spectacular, and terrifying at the same time. I asked them for their testing procedure, and it was a -t (customer-testing). If they forgot the -testing, well.....
Is that really a bad idea? The pics would not be any smaller outside the database.
This framework was given to an offshore team, and they were told to use it to farm out hundreds of requests. The framework was inflexible enough that they each started adding snippets here and there to make it work for those projects, with no code review.
When I joined there were well over a hundred different projects, all using this framework at the core, most having little bespoke tweaks to make the code work. Every failure was essentially a new failure in a new way.
It was a useful experience - it's one of those experiences that teaches me via a negative example. This was the worst example of how "roll your own" resulted in incredible technical debt I've seen.
He came back with a custom python/c++ framework where literally every function was called handle(), and in c++ it used type inference to figure out which handle() to call.
Apparently this is what he wrote for his last firm.
We quickly found something else for him to do but he didn't last much longer after that anyways.
We divided the work and I ended up working on the formula parser. I spent a week thinking about it and couldn't figure it out (I wanted to work it out from scratch). Eventually I had a flash of insight: I know how to parse simple formulas, so I can use string replacement to recursively rewrite a formula until I can parse it.
By the time I had written all of it, I didn't understand how it worked anymore, but it did work!
FormulaParser ended up being longer than the rest of the codebase combined, and I eventually learned the other groups did it with a regex and ~50 LoC...
Millions of lines of code (or even 10's of millions of lines) are really not all that exceptional. The original programmers are typically in some home or are pushing up the daisies.
Thanks for all the sharing. Young devs (like me) should really read and appreciate this thread.
Accruing technical debt was a process feature. More bad code that everyone is afraid to touch means more budget for terrified developers and testers, and insane networked database design means more budget for servers and sysops. The fear leads to meetings, the meetings lead to suffering, and suffering leads to the dark side. It still works according to spec, and is human-fixed quickly whenever it doesn't, but the poor quality of the codebase is likely costing at least $2 million per year.
In my opinion, if it works, and provides the utility it was designed to bring, then it doesn't matter. If it makes money, then who really cares!
This makes sense. However, if the code 1) is hard to understand (according to the developers available) and change and 2) needs to be changed, it costs money.
Eg, read about the expense banks are now incurring trying to maintain COBOL systems. Whether the code is "bad" is debatable. But the fact is that they have a hard time finding people who can work on it.
I think you're right that every long-lived code base will have warts. And I don't think that means that the original builders were wrong-headed.
But if you've got a decades-old system that nobody understands anymore, you've got a huge liability. You can't ship features to compete, you can't fix bugs, you can't comply with new regulations. You can't even rewrite confidently because you don't know what the old system does.
There must be things you can do as a code base ages to keep it maintainable, allow incremental rewrites, etc.
"Ugly" is subjective; I'd define "bad code" as "difficult to reason about". If you're introducing someone new to the project, how long do they have to stare at it until they can grok it? Good code is code that makes sense, where things are documented and clearly named and encapsulated and have a flow that makes them understandable.
This doesn't necessarily mean it isn't ugly. Often, it's uglier than "clever code" which does something succinctly but less obviously.
If it makes money now, you might care about continuing to make money in the future. Being able to reason about and change your product should help with that, but so will more money to throw at it, as this thread shows.
The company had bought Ellington, a Django based CMS, but the team basically rewrote the entire thing using multi table inheritance (unintentionally), so everything in the database had two copies, and we had over 70 tables, hundreds of gigabytes, disasters every week and tons of bugs... I discovered this more than a year later and nobody was even aware. Even the DBA wasn’t aware the tables were duplicates.
At an old employer there were 15,000 lines of batch script across 14 .bat files on a Windows laptop. Old director of IT used it to onboard new customers. It basically copied a DB and turned "CHANGE ME" in some columns to the client's name.
It had it all. 5k lines of date validation, 3k lines of "UI", 400 goto statements, hard-coded passwords, versioning by incrementing the file names (leading to a bunch of code that was never called), and to top it off a static IP granted to the laptop that used as a part of authentication.
Took me two weeks to unravel it and replace with ~20 lines of Ruby.
Later, all of my complaining on Facebook led an old professor to invite me back to give a talk on the importance of code quality!
Instead, he wrote web applications in R, instead of Python or Ruby, which my company had many developers who had expertise in, and eventually handed it over to to me. He even persuaded our bosses to invest into R Studio Server and had an instance installed in one of our machines. It's not only the choice of the programming language that made me furious, it's also the quality of code. He also mixed up snake case and camel cased variables all over the code. In addition, the same name would refer to different things, eg. `abc` and `Abc` and `a_bc` would mean totally different things. And stuff that could be written in a simple Sinatra or Flask application were written in R Shiny.
As a non-R person, I quickly learnt the language, (while mentally cursing it all the way for the bad choices it had made and the terrible inexplicable syntax it had) but getting used to this bad code was quite a challenge. We had several top tier clients whose reports were critical and reliant on this R code and it would frequently, randomly fail while maxing out on memory, no matter how much you threw at it. Debugging was another issue and I struggled with this codebase for 8 months while this junior developer had moved on to other technologies.
Eventually, my main role almost switched to devops which I hated, because I enjoyed writing web applications and good code that doesn't require maintenance nor devops much. Eventually, I realized I couldn't take responsibility for this anymore as it would cost me my reputation and I really didn't like the way the company handled the situation as well. They were quite supportive of the junior dev encouraging him to move on to newer technologies while he half-assed everything and threw it on other people's heads who already had other responsibilities. They did this so that they could show off at meetups "We use the latest tech stack..blah blah" while adding 0 value for clients.
So, I quit the company, along with a dozen others and never looked back. But, I did learn quite a lot..my my.
That's not too unusual. In Java `Camel` would usually be a class and `camel` an object, and in Prolog `Camel` is a variable wheras `camel` is an atom. Not sure about R though..?
> And stuff that could be written in a simple Sinatra or Flask application were written in R Shiny.
I don't know R Shiny, but the examples looks neat and simple.[0] Are you sure this is not just a case of "I don't like X" rather than the code being bad?
As for R Shiny, those examples all look fine, but wasn't the case with my code.
Is it possible for any codebase to NOT eventually (given enough time) become a crufty pile of garbage?
I suspect (but have no real evidence... yet) that SOME of this spaghetti garbage is due to the traits of procedural, OOP, mutable languages. But this would then imply that things like functional-language codebases have much longer lifespans... and I don't have evidence for that... but I'm hoping someone can chime in
It certainly has a lot of cognitive overhead since you need to always keep in mind the context in which the code you are writing will be running (to reason about concurrency etc.), but it’s relatively easy to understand and well written.
Most organizations don't see the value in that.
Also the quality of the test suite makes a huge difference in combination with the willingness of developers to refactor. No testsuite automatically means no refactoring. With a testsuite the question remains whether the organization penalizes or encourages larger code changes (sometimes with old code bases managers demand pinhole-surgery only).
And then there is this funny thing that perfectly acceptable code bases age without a change to their code. Idiomatic C++ code from the 90ies is from our perspective not clean, even if the person who wrote it was a dedicated and smart programmer following the best practices that they could get a hold on.
With functional code bases you might see similar patterns. A Haskell person of today may look at an older common lisp code base with similar reservations as a java programmer to the visual basic 3 app.
I just sat down and wondered how a tech stack potentially used by a startup could age. In the future i think we will see much more dependency problems. When you get to maintain a Django/nodejs/ruby on rails application, you always take pypi/rubygems servers for granted (or your local mirror on artifactory). Think about the time in 40 years when the dependencies are not available. Or the small languages we sometimes see and still are able to find tutorials for, how will it feel to take over a Lua codebase in 40 years? I hope that enough docs stick around, but already when I browse the web on Smalltalk stuff most links are broken because actually information can disappear from the web.
Or database technology. How will that NoSQL db appear to a maintainer in 30 years?
So while today we are unhappily having to maintain software that was written without version control, it may be that future generations will have it even worse because they dont even have a monolithic code base in front of them but something with dependnecies they cannot install anew anymore.
It's comfortable to blame the tools but in my experience it's a people problem, not a tool problem. I've looked back on my own code from years ago and wondered what I was thinking! I'm currently rewriting one of my own projects because it was done in a hurry with the requirements half-specified and my heart was not in it.
But I've seen things from other people, insane twists of logic that can hardly be imagined.
One of the projects I don't work on originates from the 80's. There is tons of actual code from the 80's in this product. It has a Windows GUI. It has a web interface. It started out on Unix at some point but now runs on Windows. It's written in a language that doesn't exist anymore. It costs millions of dollars. OOP vs. Functional is not really the question.
Edit: I actually like the following article much better, found it in the See Also section of the link above. https://en.wikipedia.org/wiki/Software_rot
Currently, I work (among other things) on a 25 year old code base that acts as an interactive, terminal-based interface for a CMDB.
It has its problems (like mostly not using the exception mechanism of the programming language it's implemented in; probably wasn't very reliable back then, or maybe didn't exist), but all in all it's OK to work with. Most changes touch only 1 to 3 files, most of which are pretty short (<100 lines of code, typically).
There are "here be dragon" areas, but not too many.
It didn't really need to be that big. A lot of its size was a result of pathological over-engineering. Apparently a previous (of course) engineer who built much of the original app thought boost::bind and functional binding was awesome. He also absolutely loved template meta-programming.
The code base was full of templates that contained templates that contained templates that contained... layers upon layers upon layers of templates, generics, bind, and so on. The horror. The compile times were awesome too. Local builds on an 8-core machine took 15-20 minutes.
I once made a commit to that code base where I replaced two entire subdirectories of files and tens of thousands of lines of code with one function. It took me weeks to understand and then finally to realize that none of it was necessary at all. It was all over-engineering. The function contained a case statement that did the job of layers of boost::bind and other cruft.
I definitely had a net negative line count at that job. The experience helped to solidify my loathing of unnecessary complexity.
It also made me respect languages like Go that purposely do not give the programmer tools like complex templating systems, dynamic language syntax, etc. It's not that these features have no uses, but honestly their uses are few. I've used Go for a while now and have found maybe two instances in tens of thousands of lines where I missed generics. I imagine I'd miss operator overloading in heavy math code, but that's about it. The problem is that these features are dangerous in the hands of "insufficiently lazy" programmers that love to over-engineer. I'd rather not have them and have to kludge just a little than to deal with code bases like the one I described above ever again.
Just let that sink in for a moment.
There were 20+ tables all modelled after how SQL Agent does its own scheduling, 30+ stored procedures for interacting with it, and it was all intended for use with a GUI that was (naturally) never written. A relationship diagram of the tables sat on my personal Wall of Shame board for most of the time I was there. And yes, I had to use it from time to time. It was installed in every database for 200+ clients, in production, UAT, development, you name it.
The most delicious irony is, the only way such a thing could run automatically... was to use a SQL Agent job to poll it once a minute.
Yeah...
1) How did it get that way, and what can we learn from that? 2) Is it inevitable that all software will end up like that? 3) How can an organization ever successfully sunset or move to a more maintainable system? Or should they even aspire to that?
I don't know exactly what they should have done and when, but it seems like rewrites are going to be necessary, and it sure would be nice to start rewriting a system while you still have someone who can explain what it does.
Also: Well-designed components are easy to replace. Eventually, they will be replaced by ones that are not so easy to replace. -- Sustrik's Law
Edit: Added Sustrick's Law
It was a 3 months nightmare with an arrogant as fudge client/developer.
It was a C project, but the second developer was coming from a Java background. So his half is all written with Java notation and naming conventions.
The thing was maths-heavy and had no comments. Instead they were consulting to a documentation book which also had file names and approximated line numbers.
The error tracing was done with function call stacks, so every function pushed some data into a global stack before a critical operation and, popped the same data if everything went without any problems.
Some libraries they were using were ad-hoc patched to taste, and grafted into the code tree. So, debugging was 5x harder and longer.
The functions were not divided into headers in a logical way, they were ad-hoc. So finding something was dependent on an source indexer or a grep sprint.
Last but not the least, the developer was so arrogant that the bugs were resolved by threatening to force him to clone his code and sent it out to another developer to debug and clean. Otherwise he pretended that the bugs were not present, because it was running on his test bed.
... and yes. It was in production.
He did't understand the concept of a join. So he'd nest queries in VBScript with join key supplied from the outer query to the inner. Row by row. Essentially, a manual cursor.
Same programmer wrote an ASP portal app. The login of which got most of its security because they didn't know how to iterate over a returned dataset. Same code would set a cookie for access IN THE PRESENCE of a password. It could be wrong, you would still get access. Worse, the logout function didn't delete the access cookie, it just redirected you to the login page. Meaning you could impersonate anybody if knew thier username. Included admin.
I once corrected bug, by using a view. I sent him the view. He had no concept what a view was "That's like a stored procedure right?'. I'm shocked he knew what a stored procedure was.
He's still in business and the software is deployed worldwide. He refuses to fix it. He's a multi-millionaire.
Lots of microservices, before it was cool, that was fine. Shitty code everywhere, commented code, dead code, 5 blank lines here and there. Many lines over 100 chars, lots pushing 150 or 200+. Didn't understand how to use argparse or logging but tried. Crazy mixed-case WTFExtremelyLongSillyNamesEverwhereSendThis at the command-line interface, instead of verbs like send.
Had a custom ORM that didn't want to look like one and took 5 times the code to do similar things. Little handling of exceptions, things like wrong permissions or IO like tar file creation might cause a 6 hour job to crash.
Daemons couldn't be shutdown gracefully, had to tail their logs until they paused for a moment, cross fingers, and kill process, often kill -9. Old daemontools made it more difficult. Bad timing could mean you are in for 3 to 6 hours of manual job cleanup work. Would happen a few times a week anyway, cutting into dev time. Still you could count on it to work about 90% of the time.
Token test suite and docs. Embarrassing web interface that would look amateurish in the 90's. Original developer made us do a standup everyday at 10:30am just when getting into the zone. They felt worthless for a while, and then it dawned on me why, we were all working on different projects.
The punchline: spent three months of 60 hour weeks taking out the trash, writing tests, paying down debt. Spent the next month or two with another dev designing/writing a vastly improved V2 with graceful shutdown, Django-style ORM, and quality as headline features.
A few weeks before we're about to knock it out of the park and deliver, the old author of V1 comes back in a panic, says we need to finish at end of month as a huge project is finishing. Doesn't seem to make sense, big changes at end are a bad idea. Takes over control of project, designs/implements V1.1, pushing aside our improvements and whips it up in a few weeks while I sit there with nothing to do. 6 months work of 60 hour weeks flushed down toilet.
After picking jaw off floor and offering a few choice words I left the job by mutual agreement a few weeks later and didn't look back. Good times.
How about barbed wire? https://news.ycombinator.com/item?id=15910263
To give you some examples, I originally came on as a contractor because they had some refactoring they wanted done. The entire system was home built (including the programming language) and there was a file size limit of 32,767 lines. They had many functions that were approaching this limit and they didn't know what to do, so they hired me. Probably you can imagine what I did.
One time I went to a code review. They were writing a lot of data into some pointers. I asked, "Where do you allocate the memory"? The response was, "We don't have to allocate memory. We ran it in the lab and it didn't crash, proving that allocating memory is a waste of time". No matter how much I tried reasoning with them, I couldn't convince them. The code shipped like that.
One of my more amusing anecdotes is that when I worked there the release life cycle was five years long. The developers would work on features for 3 years. The developers were responsible for testing that their own code worked. There was no QA. After 3 years, we would ship the code to the telcos (telephone companies) and they would test it for acceptance for 2 years. We would fix the bugs that they found.
I started working there at the end of a release cycle, so people were only fixing bugs. I got an interesting bug in that I couldn't find any code that implemented the feature. The feature had apparently been implemented at the beginning of the cycle (so around 4 years before), by someone who was now my C level manager. I started looking at the other features that person had implemented. There was no code. It seems that this enterprising person had started work and realised that nobody would check his code for 3 whole years. He just checked off all his work as done without actually doing anything. Since he was an order of magnitude faster than everybody else, he was instantly promoted into management. When I reported my findings to my manager, he made it clear I wasn't to tell anybody else ;-)
Such a messed up place. But the switch worked! It had an audit process that went around in the background fixing up the state of all the processes that ended up in weird states. In fact, when I worked there, nobody I worked with knew how to programmatically hang up a call. If you were using a feature like 3 way call, etc, they would just leave one side up. Within 3 minutes, the audit process would come by and hang up the phone. Tones features "worked" that way -- by putting the call into weird states and waiting for the audit process to put it back again. You could often hang up after a 3 way call, pick up the phone and still be connected to the call.
Most people don't know it, but because of some strangeness with some of the protocols, telcos used to "ring" their main switches with Nortel DMS switches. This would essentially fix the protocols so that everything could talk to everything. So, if you ever made a long distance telephone call 20 or 30 years ago, it almost certainly went through a DMS switch. The damn thing worked. Somehow. I have no idea how, though ;-)
... and it'll probably load a lot faster than the PHP+CSS nightmare that is WP.
The developer who was leaving the company dropped a copy of Michael Feathers' Working Effectively with Legacy Code. There was a small amount of Python to wrap the API to the C++ code using Boost against which a small suite of unit and functional tests were being developed. I learned a lot on that project.
I never fully understood how the C++ code all worked but having that API interface in Python helped to grok parts of it (and eventually replace it with a few hundred lines of Python code at a time).
1) Read a Wikipedia page for vi.
2) Noticed that Bill Joy used a Lear Siegler ADM-3A terminal.
3) Discovered that Lear Siegler was the result of a merger between Siegler Corporation and Lear, Inc.
4) I know that Lear, Inc. was founded by William 'Bill' Lear.
So, not only do we have Bill Lear to thank in a way for vi, but your username is oddly familiar...
One client wanted to hire me to continue developing his AutoCAD plugin written in AutoLisp (a cut down version of lisp for writing macros in AutoCAD). He had all his code in a single file which was around 63,000 lines long. (I calculated this as 504 meters of screen/paper top to bottom).
This was for a product that was in production, used commercially, and he'd been going for 15 years, adding bits as he went. There were loops hitting 300 lines, well over a hundred global variables, no consistency, and duplication like you wouldn't believe because he hadn't grasped the idea of moving code out to reusable subroutines.
I asked him what he did when his clients found a bug. The answer was "knuckle down for a few weeks".
I just couldn't believe that this was his daily work for 15 years, and he never thought to learn anything about programming other than what was immediately required to solve the problem in front of him...
Another unbelievable part of this, is that AutoCAD has an absolutely superb IDE built into it (i.e. it's fricking free!) which has features like the ability select & run code in current scope with minimal clicks which make for the fastest development environment I have ever played with, as in, you have no idea how much functionality you can churn out in a day. But he didn't like it, so he edited his code in....wait for it... WordPad! (That's a Windows built in Rich text editor where double clicking on an underscored_word only selects up to the underscore because it's not made for code, and that on its own makes it impossible to work with).
I tried to modify his code for several hours but had to stop myself because it was madness. So I told him the only way forward would be to rewrite everything from scratch after which point I could make it do anything he wanted. I reckoned it would only take 4 weeks to rebuild his 15 years of mess - partly because the IDE makes development so damn quick, and was even willing to do it on a fixed price, but he declined, which I think was utter madness.
The thing is, he had 15 high paying clients, and two other part time employees helping with other aspects of the business, and drove a brand new Audi A3, which means he probably holds a world record for highest ratio of money earned to quality of code written, at least in the CAD subcategory :-D
As time went by, everything started to make sense and I was able to grasp almost everything about the environment and the tech stack. Even perform a little bit of system analysis in COBOL to try and identify some gaps in the code base.
But what always intrigued me is that even those senior developers in the team with 25+ years of experience with COBOL wouldn't ever touch this one program. The program responsible for 90% of the logic of the product of that company. Every now and then, this program would ABEND for whatever reason and sometime, no one could figure out why. They respected (cute way of saying they were afraid) this program so much that, instead of refactoring it, they would just throw an if statement and let it run.
This program was built in the 60s and had I don't know how many hundreds of thousands of lines of code. It is still running to this day.
Now I don't have the expertise to say if that was bad code, and even if I had, I didn't deal with this program enough to say this anyway, but I was very intrigued why would this particular program ABEND out of nowhere, and no of these super experienced developers would have the guts to touch it.
Boy did I regret that decision!
- 100KLOC
- Initial development outsourced to India. Comments, variable names in Indian.
- Subsequent development outsourced to Belarus. Add comments, variable names in Russian.
- "Why use ObjC OO features when we can write buggy and incoherent C"?
- Global, implicit state everywhere. Tapped on something? Hope it didn't mess up the state you are relying on.
- Obviously no tests, and no testers.
- Inheritance chains of up to 20 classes.
- iOS kindly forces MVC, but you can obviously write empty controllers and all spaghetti-logic into your views. Needless to say, that was what they did.
- Complete lack of proper structure. Several views iterated up their parent controllers to the desired one, grabbed its views, iterated over them until the right view was found and something was done to that view.
- Building and running the application was controlled by 10+ env variables (the other dev was fired after he pushed a dev build to the App store, which mysteriously passed review. "Whoopsie, forgot to set one of the env variables correctly").
- about 80% of the logic was copy-pasted for the iPad build instead of reusing anything. It was not a different target, it was a separate project.
How long did you work on that before you threw in the towel?
A while later I heard they threw everything away and started an in-house rewrite in Swift. Some people do seem to learn from their mistakes.
EDIT: clarification on the opening statement.
1. Millions lines of messy Java code with annotations and dependency-injection form a multi-shard distributed job with tens of distributed downstream backends, some of which provide fake always-success synchronous RPCs hiding the fact that the underlying operations are actually asynchronous and may fail very often. What makes it worse to work on these code is it was built with a single transactional underlying database but then due to reliability issues of the single database, data are now across at least three different transactional databases. This causes endless race conditions and concurrency issues to fire everywhere. The production release of this distributed job used to be once every week, now multi-months is normal, and quarter rollback is not a surprise to people.
2. Almost million lines of messy C++ code with several .cc/.cpp files containing tens of thousands C++ code, some class implementations are across multiple .cc/.cpp files. I have always been scared when touching some of these giant .cc/.cpp files. People who have been working on the code for years can still easily make ignorant mistakes when adding/modifying a small feature (with so-called fully unit-test overage of course). There are multi-million lines of testing code, which is almost 10x of the code be tested, but most of them are bogus and almost test nothing, hence silly mistakes are everywhere, everyday. Even the original author of the code base needs 5 follow-up fixes in order to make a 10-line behavior change work.
For both code bases and the jobs running these code, people are now talking about breaking them into microservices, by converting function calls in the existing code bases into RPCs. I can foresee a tremendous number of service outages are coming...
Nobody could even describe definitively where all the code could be found in version control. The best part was that nobody trusted version control either, so even if you did find something that looked relevant, there was a good chance it was dead and only served the purpose of confusing you or forcing you to memorize unstructured structure just to be able to get around.
It was madness inducing.
One of my first gigs actually getting paid to code was getting hired on as a last ditch effort to save what was (unknown to me at the time) a failing business. It was a company that basically relisted real-estate auctions on their own site, coded entirely in PHP, by a single developer who had read "How to Code PHP in 24 Hours". They had one client that was basically keeping them afloat. It was so disorganized that at one point I was tasked with making a quick YouTube commercial in Adobe Premier for a client, showing off the features of our whitelabel product with their logos, edited from a stock template. I do not know Adobe Premier. I digress.
The main PHP file (yes) was 20,000 lines of code. Want to add a new feature? Copy that file into a new file and save it as newfeature.php. Database operations weren't transactional, there was no change management, and for about a week we were using production systems to code until development environments were made for us.
There was other shady stuff going on too, like using over a hundred proxy accounts to scrape content from other listing sites. I refused to touch or even look at that logic in the codebase, and it was always talked about in kind of a hushed way. I was young, didn't know any better, would nope the eff out if a similar opportunity came along at this stage in my career.
They folded shortly after laying off pretty much everyone but the CEO and the lone coder. Dumpsterfire would be an understatement, but my coworkers were chill and helped make the best out of a bad situation.
EDIT: Oh yeah! I forgot to mention the hardcoded password the was site wide that we used as a sort of "impersonation" feature. You could type in any user account and this password, and it would log you in no problem. No, we did not have auditing controls.
It was interesting. The database was Mongo and they had some horrible Entity Framework port that they forked on Github and used which wasn't maintained and at the time in 2015, 3 years old.
The data layer had business logic. Let's say they were getting a user record from the database but they didn't want to include the first and last name they'd write a method called
GetUserWithoutFirstAndLastName();
That would be in the User repository. Another requirement would come up to get the user but not include the user's language for instance and they'd create another method
GetUserWithoutFirstAndLastNameAndLanguage();
Ended up with about 70 or so methods which basically gave different levels of hydration for the User object.
The frontend was written in Extjs which took about 25 seconds to load.
They had no tests.
Since the environment provides realtime feedback, it's a drag to try to refactor the current diagram into a reusable abstraction. Instead, many users optimize their creative time and just keep adding functionality to the diagram in a single graphical window. By the time they are done there is text overlapping other text and a bunch of lines obscuring most of the diagram.
Even with the ones that have a simple set of controls for a sequencer or whatever, there is usually a "guts" module that hides all the spaghetti.
Also, for any running program you can instantly make it twice as spaghetti by doing "select all" and "duplicate." :)
For more traditionally defined bad code, I worked with a team that rewrote an existing service from scratch using with 40000 lines of spaghetti. The service that it replaced was 80000 lines, and was replaced despite working perfectly fine because it was a total clusterfuck with dependencies interwoven across every service anyone had ever heard of. This was a payments system for a major online retailer. All of this code has been removed since.
I also used to participate in Java4k, a competition to make games fit in 4 kilobyte jar files. Writing almost everything inside a while loop in a single function is basically table stakes for that.
Your post should be at the top of the thread. Rewriting projects from scratch tend to do more harm than good, and the only reason this problem isn't addressed very often os that the people invested in reinventing the wheel don't admit to having caused more problems than they solved, and the ones who developed the old systems have moved on and thus are unable to say anything in their defense.
The crazy part is going into the commit logs and seeing that a lot of the people who wrote this have been very successful. They've become vice presidents, retired rich, and written influential papers.
We've got another codebase of similar vintage that is preventing us from deploying spectre patches...
We tend to think that harsh deadlines and unthinking managers are to blame, but writing good code over the long term is incredibly difficult. Group dynamics are hard to deal with and as you add (or replace) people on the team, you are bound to eventually go off the rails even if you started off well. Which is not to say that you shouldn't try, but our industry is really immature. Most developers are quite young and even when you have a couple of older people on the team, they may or may not have the skills for long term design evolution. Maybe 50 years from now it will be the norm to write good code, but I think we'll still be writing legacy messes for a while.
I should point out that even the worst code I've seen written this decade is at least an order of magnitude better than the average code I saw when I started my career. As an industry we are improving!
I replaced it with an 11-line template.
Dim Hermes As String, Artemis As Long, Odin As Range
Original code also computed the most time-consuming routine twice to thrice each run and had many many typos, e. g. "calculte_infomration". Imagine this, but in every second function/variable.
However, it worked.
The trophy should be a plate of spaghetti, in gold.
An IVR application would typically play a recording to the user, then wait for the DTMF signal, then maybe take the user to a sub-menu, prompt the user to choose another option and do a database query and then play another recording and so on.
The IVRs had to repeat a menu if the user made an invalid selection, and "press 'star' to return to the main menu" and so on.
So most of the applications I maintained were thousand line C++ functions that looked something like this (paraphrased):
void ivr_main(int ch) {
main_menu:
play("mainmenu.vox");
d = get_dtmf(ch);
if(user_hung_up(ch)) return;
switch(d) {
case '1' : goto menu_1;
case '2' : goto menu_2;
case '3' : goto menu_3;
case '4' : goto menu_4;
}
goto main_menu;
menu_1:
play("menu1.vox");
d = get_dtmf(ch);
if(user_hung_up(ch)) return;
switch(d) {
case '1': goto menu_1_1;
case '2': goto menu_1_2;
case '*': goto main_menu;
}
goto menu_1;
// etc...
}Not too many LOC but the badness/LOC ratio is terribly high.
Mind you, it may have got better than my experiences with it, which as you can probably guess weren't very positive.
And it was only a small portion of the OS.
The first day on the team, before even looking at the code base, I ask our development lead what unit testing framework we are using for the Java code and what we are using for the front end code. He gives me this funny look and tells me to speak the guy onboarding me. I of course go and ask the lead onboarding me and he gives me an answer that turned my world upside down.
"Automated testing is a waste of time. You will spend too much time writing test then developing code and delivering stories to the business"
I need to point out that this guy eventually goes on to an executive level position. proving that it is not the quality of work you deliver, but the optics of delivering quality software that counts in large corporations. I digress.
I receive my SVN credentials later that day and come to the realization that I have made a very poor career choice. The front end code alone is over 5k Javascript files with functions spanning thousands of lines long all full with 100's of nested asynchronous callbacks. Not only is the code crap, but the tools that we had were not used correctly. For example comments.
//01/01/2018 - Fix bug - Begin //01/01/2018 - Fix bug - End
The code base was riddled with these. What story was this for? would have been useful to pull the Jira story and see what you were trying to do. I guess I'll spend the day going through your 5k line function that does everything, but nothing. Or, better yet, let me check the SVN commit date. Maybe they committed some good notes for me there.
01/01/2018 - Fix bug
Damnit!
I did my year, which is the minimum you can be in a job before posting out. I left without a second thought. When I left the business was complaining because the testing team was larger than the development team. I wonder why?
I'm maintaining two semi-large applications right now that I wish I had the time to fix.
The bad stuff is basically no documentation, no attempt at pep8 compliance, written before decorators existed in python and very dependent on Python 2 syntax.
But it works and it's actually a very good distributed monitoring system that was ahead of its time. Closed source but it would remind people of prometheus if it was released, yet it was several years ahead of that product.
The app ran on Mac servers.
Barely any tests and several hand-rolled components, some of which were merged only because the author managed to implore somebody to give them an R+ after a few sprints of the branch just sitting there.
A total of 21 people were involved in creating this system and for some insane reason we still had daily standups with almost full attendance.
Notable among the ones in this project was the date picker - there's a decent number of those available and yet somebody made the decision to hand-roll it.
The result was a mess that for some reason had a 300 LOC service as a part of it. Needless to say minimal tests.
It was a waste of man-days in a project that was already over budget.
There's so many integration points (business logic, i18n/l10n, styling, keyboard navigation, accessibility, etc) that I think it's often easier to write your own. Hopefully now that most browsers support <input type="date" />, we can just use that most of the time.
Drupal 7 started the migration to object-oriented code and was halfway through the messy rewrite when it was released. Drupal 8 finished where Drupal 7 left. That's where the majority of the developers left as well (pun intended).
There's lots of materials on how to refactor code (improving its structure without changing functionality), including blogs, books and video lectures.
- Line should not be too long (120 max)
- Function should fit into 1 page in the view port of screen, even when your console and debugger take 1/3 bottom part of the screen.
- Variable name should be pronounceable and longer than 3 letters (to prevent name like `i`, `x`, `s`)
- File should not be too long (1000 lines max).
- Return/Exit/Throw as early as possible.
- Comment:
- If an `if` take more than 2 condition, it's worth commenting.
- If a funciton is longer than 10 lines, it's worth commenting.
- All file's worth commenting.
- If you ask yourself "should I add comment", add comment.Robert Martin, Clean Code.
Most people that have read the book (and it is a classic so many have) will swear by it. I most certainly do.
Another great tip that saved me from tons of refactoring: "The wrong abstraction is worse code duplication". Meaning sometimes duplication of code is better than trying to create the wrong architecture (as long as you mark it in your quick and dirty list).
Microsoft support was basically "yes it does that sometimes".
The Guy Before Me™ decided that the best way to implement this would be to split the user's search into individual words, perform a separate search query through Solr's HTTP API for each individual word, and then do a bunch of very clever and complex post-processing on the result sets to combine them into a single set of results.
This led to endless headaches due to horrible performance. Imagine if you wanted to implement web search this way. How would you synthesize the results for the search "boston plumbers" given the search results for "boston" and the search results for "plumbers?" You would need tens of thousands of results for each search term to find even one match that applies to both terms. Now scale this to getting hundreds of results to present to the user. Now scale this to n search terms.
I was tasked with making this take less than 8,000ms for a simple query. I spent a while getting to understand how this code worked and building out performance tests so that we could determine how it would behave under load (we didn't have any users yet). The results were pretty grim. I presented two possible options for moving forward:
1. Move this crazy result-set-intersection logic closer to the data. I could build a custom Solr plugin to do this stuff inside the Solr server so that we didn't need to copy gigantic result sets across the network from Solr to the application server for every query.
2. Delete ALL of this nonsense because literally exactly what this whole mess of code was meant to accomplish is already implemented in Solr. They call it highlighting. It's one of the marquee features of the program. I can't stress enough that this is precisely, perfectly, unequivocally, the exact thing that all of this complexity was meant to accomplish.
My manager thought it would be a shame to throw away all of that very expensive code and lose the flexibility of an in-house solution. So we went with option one. I spent the next month writing a Solr plugin that reproduced the original logic. It was still slow as mud so I sharded the data across multiple Lucene servers and distributed the algorithm across them with a map/reduce sort of scheme.
In the end, it all worked great. It was fully ten times slower than the solution already built into Solr, but it worked.
The startup later ran out of runway trying to build a big-data-sized in-memory distributed database from scratch to speed up search. The founder (also the lead developer while I was there) insisted that everyone use raw C-style arrays and a custom in-house hash table implementation because he thought STL was too slow. Basically, "not invented here" was in the DNA of that company. I'm surprised we even used commodity hardware and didn't design some kind of in-house search coprocessor that would do everything in silicon.
Maybe, if we pitch the idea to Randall, he may prove it. :)
I was at a project were every new developer said "i have never seen such bad code". Srsly! 3 new People said it and me as well.
- Bad testability - no tests - hidden 3 bad bugs surfaced in just one year - ...
I had very little trust in that code. And it was not much fun. On Upside: Cleaning it up was a great feeling.
Wish you all the best
I once worked for a reasonably large business that ran all of their invoicing and stock management through an MS Access project. The whole thing was wacky. half the business logic was implemented in VBA. The other half was stored procedures, but there was no obvious pattern (predictably, anything that involved a task the DB was optimised for was written in VBA). "Deployment" consisted of saving to a network drive and waiting until everyone opened it again in the morning. Data integrity seemed optional and inconsistent. The symbol naming convention could be described as cryptographic. The only documentation was a comment over each function stating the original developer's name and a timestamp. It had a proud splash screen stating "Developed by Dave Davidson" that was shown for 5 seconds on startup - this was of course completely simulated and there was no reason to have a splash screen. I could never quite fathom the magnitude of this guys delusions of grandeur or why they would want to put their name to it.
The worst part about it all was that for the most part, it worked. The parts that didn't work were well known to the people using it and worked around. So processes were developed around it, and over time these became so deeply ingrained in the teams using it that they couldn't imagine working any other way. Most of this consisted of taking telephone orders, printing off the invoices that were generating in Access, and then rekeying that information into an accounting system (for efficiency purposes of course, these printed copies were passed to another team member with notes scribbled on the original document. Mistakes were commonplace and accepted as a CoB).
Part of my role was to implement a web based ordering system. We did a reasonably good job but of course had plenty of our own WTFs. The biggest pain was integrating this with a particular team. They could not imagine a process that did not involved printing off orders and rekeying. After a while we realised that the reason we got so much push back was that once fully integrated, our system would make half of the team members redundant.
With support from management we went ahead, and over time the wrongs were righted. When I left there was still a deep level of distrust in the new system. Mistakes that were daily occurrences in the old world were "proof that the new system won't work". Orders were still printed "just to make sure I don't lose it". New bugs were treated as if the sky were falling. I would spend more time managing expectations than writing code. But we got there in the end.
Buggy software can be fixed. Buggy humans are an entirely different kettle of fish.
<puts face in palms, starts weeping gently>
function a_nodes_list_trees(
$site_url,
$base_url,
$mode,
$site_id,
$max_subtree_depth,
$nlevels,
$node_id = 0,
$tree_id = 0,
$nleft = 0,
$nright = 0,
$nlevel = 0,
$current_subtree_depth = 0,
$flags = 0,
$order = 'tree',
$inline = false, /* needed when loading stuff via ajax*/
$skipped_types = [],
$items_per_page = 10,
$min_tag_count = 2,
$min_author_count = 2,
$is_site_root = false,
$skip_pagination = false,
$ignore_404 = false,
$skip_display_options_and_batch_ops = false,
$display_bottom_pagination = true)
$mysql_the_rest = 'FROM
'.DBTP.'node n
LEFT JOIN
'.DBTP.'node n2
ON
n.tree_id = n2.id
AND
n2.perm_view & ' . A_PERMS . '
'.$mysql_join.'
WHERE
'.$mysql_where2.'
'.$mysql_where.'
n.perm_view & ' . A_PERMS . '
AND
n.site_id = ' . $site_id . '
';
$total_count = $A->db->fetch_one_column('c', ' SELECT COUNT(*) c '.$mysql_the_rest);
Yet, it works like a charm, PHP updates made it faster even, and I can use it for everything I needed so far, and use its output in a variety of ways. I still want to rewrite it, but it seems a lot of work to just shave off a few ms and have nicer code, with the same result for the visitor, and also having to write something that migrates the content. I suspect with enough content, it will slow down, and then I'll think about the next iteration. But it's still a mixture of pride, plain being happy to have it, and groaning whenever I fix a bug or add a feature.
I just thought about that when I saw a youtube comment saying "stolen" in response to a funny joke. I wondered, do they mean they intend to "steal it" because it's a good joke, or did they mean the person who posted the joke stole it from somewhere? Nobody will ever know, at least I for one won't sign in just to ask.
I notice that since mobile devices, a lot of "communication" on the web these days is kinda like the "small talk" from Kevin from The Office US.
https://www.youtube.com/watch?v=_K-L9uhsBLM
If I hadn't asked what you meant, I and anyone else who read your comment would have had their own interpretation of it. I see a lot of comments like that on HN, where you would have to ask "what do you mean?" because it's totally unclear. A variation is stating something that is factually true but doesn't really refute anything, but the commenter clearly seems to mean something by stating that triviality, but don't say what it is. It's like dog whistles, but not for others to hear, but only the posters themselves know what they mean. Count me out.
No need for 'atleast'. I'm not criticizing the length, its fun-ny because its another form of life and way of expression that others use, brings me joy when i see patterns in life, expressed in many forms, one being someone who is verbose writing a hilarious god function then inadvertently backs up that 'digital persona' posting with a god function-esque bio lol!
options = NodeBuilder .author({ min: 2 }) .flag({ .. }) .pagitation({ .. })
function a_nodes_list_trees(options) { ... }
Because in the end, if your function is fast and working correctly, it's fine even though it's a little bit messy. The problem is more all the code calling this function and having to pass dozens of params in the right order, and then it's a pain to start adding/changing parameters. Also, when calling this function, you probably need a bunch of temporary variables to "build" all the params; all those temporary variables could instead live inside that builder object.
Making and then really using the CMS helped me with knowing what I would want in my next CMS, and just keeping the whole in mind from the start would probably make it a lot better by itself. My thinking is that the longer I put that off, the more languages improved and the more I hopefully learned in the meantime ^^
So it might be job security but it's also vulnerable to the hit-by-a-bus problem