Roll Your Own Frameworks
blog.startifact.com
blog.startifact.com
The first half boils down to (a) research existing solutions, (b) avoid NIH when you can and (c) refactor your code as you find commonalities and points of re-use. The second part is mostly about designing a system that could be used/extended by someone else (separate the generic from the the specific, treat it as open source).
Those are very valid points. Thing is, in following those very valid points one could probably write another article with a far greater number of examples (and a far better lesson) where the best course of action is to NOT roll your own framework and prefer to adopt some existing code and make the necessary extensions/modifications.
To illustrate, I checked the morepath "framework". It seems to tick all of the boxes one would expect from a proper open source "framework". It even has more documentation than I would expect from such a small open source project. The one point where it fails is that it never gets to give one simple example of a pain point that this project solves that all of the other million web frameworks don't.
In the end, it seems that the author is looking for an ex post facto rationalization for some of his projects that had a lot of his time invested but no external interest afterwards. Too bad that it seems that he also took the wrong lesson out of it.
[0]: https://morepath.readthedocs.io/en/latest/compared.html
If you have a large organization that can absorb the cost of maintaining the framework long after you have gone, and can ensure the entire company is invested in this framework? Sure, someone will hate you in five years because they will hate your opinions, but you won't sink a company.
But if it's in between, you are doing a disservice to whoever is paying you. You won't document it like a great framework. You'll be fine with a hack or need to skimp on testing during a deadline, and three years later people will be stuck around decisions that seemed right at the time.
I have seen how bad of a drag this can be up close.
It pretty much echoes your statement that unless you know you and others will be documenting and maintaining (and that is a big commitment), your probably just doing everyone a disservice long term.
Other than that fully agree, the power of documentation stackoverflow and forums can be easily overlooked but is incredibly valuable. This even if we all feel a little miserable every time the framework fails at any of the magic things it does and we just patch it up and end up never fully understanding the root cause.
In some cases I think it's a tricky call. I'm currently working on a "framework" and am worried about all the issues that you mention. The reason I'm still building a framework is that as far as I can tell there is really no existing framework for the task at hand, and we do need the economies of reusable code (consistency across deployments & avoidance of double work). I hope the organization is large enough to absorb this and keep it going down the road, and am aiming to make it open source in the hope that a community can form around it as well. Fingers crossed, I guess.
So, to be fair, many of us have done it. The best case is that you can open source it and everyone can benefit from your work.
I've recently started a new application in PHP which has very strict time / performance requirements. I could build it in another language, but I'm most familiar with PHP, the thing would take me 4x as long to build in Rust.
The default frameworks (Laravel and Symfony) are very slow out of the box, and doctrine / propel are pretty slow as well.
I've instead built something entirely brand new using Swoole and an assembled set of very fast and very light Composer packages (namely Fastroute and PHP-DI) and everything else I'm building myself (compromising on reusability and full-featuredness in return for raw performance).
I've split the code in to two: one half is the framework, it's concerned with routing, dependency injection, models and hydration, event handling and responses. The other half is focussed on the business concerns: creating users, creating authentication token, performing searches.
There were a few challenges: moving from single row, single model queries to hydrating complex joined collections was a major multi-day challenge, but the end result is that when a request has all the data it needs cached, it responds in ~3ms, when it needs something from the database it responds in ~10ms.
I was apprehensive when I started, and a half decade ago this codebase would be an unmaintainable mess, but I'm finding it a joy to work in: there are few layers of abstraction, not a single line of code is wasted, and when something goes wrong there's no magic obfuscating the true problem.
The code for the framework is its own composer library which is pulled in to the vendor directory, the code for the application sits within src/{BundleNamespace}.
If there's a service, it's a service, if there's a controller, it's in Controller, etc.
The aim is to maximise ergonomics and familiarity while minimising surprises. If a developer can pick up your microframework and be instantly writing code then you've done your job right.
It’s faster. See https://yang2020.app/fast.php
I've "rolled my own" quite a few times this way, with web frameworks and ORMs/query engines, and while I agree it feel that way, I would add the caveat it feels that way at first. Eventually, whether that's 6 months or 6 years, you end up cursing the framework because either you're spending a significant chunk of time implementing features that you didn't need but now do, and would have either been included in a framework when you started, or you would have just been able to upgrade your framework to a newer version to get it.
Unless the project has a very tight scope, the eventual drift in needs causes friction when it comes up against the very targeted capabilities of what you designed for yourself because it only implemented what you needed when you wrote it.
Writing a web framework is useful and fun, as is writing an ORM. At least for the first 75% of features. People should do it, if for no other reason it makes assessing other ones for whether they are suitable for your needs much more easy and accurate as your knowledge increases. But 5 years from now when you (or whoever inherits it) are realizing there's a security issue you need to patch in your home rolled framework, and you aren't quite done patching the security or performance problem that was found last week, you might reassess how good of a choice rolling your own was.
Symfony and Laraval might be slow, but there's a slew of other PHP frameworks to choose from that range from full stack to micro frameworks[1], and those probably deserve a look for anyone in the same position considering the same move. Maybe rolling your own still comes out looking good, but it's not something I think should be attempted in production without quite a bit of consideration as to the eventual implications.
1: https://www.techempower.com/benchmarks/#section=data-r18&hw=...
Is it anything like this?
There are some superficial differences (we don't have Twig as it's serving as an API, etc) but it mostly follows what the repository you posted lays out.
We found Phalcon to be a bit too opinionated in how to do things, and we fell in love with how Swoole has implemented Coroutines and (practically) free context switching between long running asynchronous processes and short running network requests.
Looking back, I do actually wish we had chosen Phalcon (the documentation is a lot better than Swoole's, which is lacklustre to say the least, and occasionally intersperses random characters in Simplified Chinese halfway through sentences), but we've been happy with Swoole's Table implementation which allows us to share memory between network request Coroutines and hit the Redis cache a whole lot less.
PHP had a strong re-invent culture, probably because it didn't have a package manager back in the days.
When I started with Node.js and NPM I often just installed something well known.
A few years ago I had the feeling things would get a bit out of hand, so I only installed packages I couldn't code myself, because the lack of domain knowledge or time.
- they're using Composer packages
- they're using the fairly well-documented, popular Fastroute for routing
- they're using the well-documented, PSR-11-compliant PHP-DI dependency injection container
- they're using the well-documented Swoole for async programming
In other words, they're rolling their own framework not from scratch, but on top of well-understood components that largely follow -- and largely require programs that use them to follow -- what's considered to be "best practice" in the PHP world.
I get that there's a lot of crap PHP out there, historically, and that frameworks have helped dramtatically reduce that. But it turns out you don't actually have to start with a framework in order to write well-structured, well-commented PHP code -- you just have to have a commitment to doing it.
I'm working on my own PHP microproject right now and I'm pretty flagrantly violating some "PHP the Right Way" conventions; the project doesn't even Composer and -- horror of horrors -- the autoloader isn't technically compliant with PSR-4. I'll pause for everyone to recover from the fainting spell. I don't think anyone else is going to ever have to maintain this -- but I'm also pretty sure that when I'm finished, it will actually be more understandable and maintainable than some Symfony and Laravel projects I've worked on in the past. It's not the fault of those frameworks that people build inscrutable undocumented modules on top of them, but it still happens.
And frankly, I understand why someone can appreciate Symfony and Laravel and friends and still say, "Wow, that's... just... a whole lot." (If you initialize a new Symfony 5 project, before you have written a single line of your own, your project is already at around 160K lines of code.) And they can look at microframeworks, especially modern we-must-comply-with-all-PHP-FIG-mandate ones, and say, "That isn't a whole lot, yet it kind of feels weirdly heavyweight for what comparatively little it does."
> Please stop reinventing the wheel...
You learn a lot by reinventing a wheel. Sometimes what you learn is that a lot of the existing wheels are actually monster truck tires, tank treads, or jet skis.
We run and maintain several Symfony applications, we're very familiar with frameworks, how they work, their strengths and weaknesses.
This was a carefully considered route:
We wanted to be writing PHP code with high performance and strict response time requirements, as such:
* Symfony was instantly ruled out.
* We knew we would struggle to do this with raw PHP but didn't want to retrain the team to learn a new language (our thinking being we would likely write something inferior and insecure).
* We considered Phalcon and Swoole and ultimately decided on Swoole as we found Phalcon a bit too opinionated. That being said, Swoole's documentation isn't great so we've had to internally document it. (We will probably push a few documentation PRs for swoole-src when we're done).
* We didn't trust a big framework to have zero memory leaks, which is a key issue for long running Swoole processes, since it's not a typical issue for the single-use PHP processes we are used to.
With regards to documentation, it's very well documented (with documentation explaining the reasoning behind each class, usage, pitfalls and how it fits in to the larger application, and documentation for the wider ergonomics of the application, debugging, performance considerations, etc), with a 96% test php-unit coverage for the Framework.
Your concerns are very well received and are concerns we have had from the beginning. Constant vigilance and a deep respect for my co-workers is what keeps this little project on the straight and narrow.
Have you considered sticking to it and applying some caching layer on top of it?
We did originally consider writing it in Rust (it would have been my personal preference) but the consensus was that since the rest of the tooling for this customer is in PHP we didn't want to add a skillset they didn't have internally if they were to move away from us in the future (we aren't big on needless vendor lock-in and prefer to compete on quality and service).
We experimented with PHP7.4 and Symfony / Laravel and found that we didn't have the performance leeway we needed, even with preloading / opcache.
We opted for Either Swoole or Phalcon, tried them both and found that Swoole was more to our liking and was somewhat more performant than Phalcon in our real-world testing.
We are exposing Swoole directly to them within their VPC, so we were able to do away with NGINX as a reverse proxy as well and have Swoole handle the HTTP request phase and HTTPS. We were able to make heavy use of Swoole\Table to save superfluous trips to Redis and run a Redis synchronisation process in a coroutine thread when the concurrent number of network tasks drops below 16.
Our end service response times (including network overhead) are around here:
- Mean response time: 4ms
- Fastest: 2ms
- 99th percentile: 9ms
- 999th permille: 10ms
One unforeseen side effect is that it has a crazy amount of scalability built in by default, the 8core machine the process runs on can easily scale up to 15,000 concurrent requests (as in 15,000 requests per second) without going over 15ms for response times, and can scale up to 45,000 concurrent requests before things start to "break".
Nope, existing frameworks are designed to solve the problems in 100,000 different applications across every domain known to man. Your "framework" (which if you're doing it right is probably more like a set of patterns and a carefully selected set of libraries that do one thing well) solves your own problems, and only your own problems.
If something tailor made to solve your use case specifically looks exactly the same as something like Django, then you're either Google or you've overengineered it.
The end result has its own pros and cons:
* We don't get the brute-force security you get when millions of websites are running the same framework
* Our implementation is much faster (and there are many fewer memory leaks, as that was a core concern), as it only has the features we actually need.
* Without middleware layers for cross compatibility there are fewer abstractions, this allows the entire framework and application to be reasoned about with a lot less magic obfuscating the code which is actually running.
* If something is slow, it isn't an opaque problem of "symfony being slow", it's "@helldritch can you take a look at \App\Framework\DB\Query\JoinHydrationStrategy on line 112? If you unset the fields you no longer need, the hydration runs in linear time - much quicker when joining collections of Users against the Groups they're in)" <-- an actual message from our Discord.
* Each feature takes a little bit longer to develop (as the Framework needs the implementation before the application can use it)
That's true only if you're looking at it through the lens of the current paradigm; for instance, assuming that you need "typical" routing, HTML templates, reactivity, etc. But, these each represent just one way to solve the problems they address.
So, "the right way" here is kind of self-referential and assumes there's no better paradigm to be found. The current crop of frameworks are conceptually all very similar and represent just one more iteration. We'll move on to a completely new paradigm at some point.
Let’s say GP and his coworker switch jobs in the next couple of months. Who is going to maintain an application built on a framework no one understands? At that point you’d have no choice but to rewrite it except now you don’t have the knowledge of how the existing system works.
All of this without even getting into the security implications of using a framework that no one else is.
The difference is this framework still runs on a modern platform (assuming it uses JS and other Web standards). So, it's dated only WRT other frameworks, which come and go anyway.
So, unlike with COBOL (and "Cobalt"), it's supported by the platform of the day with standards many current devs know.
A 2-decade old project probably would take such a long time to re-write into a modern framework that the time would be better spent investing in devs willing to work with your system
Particularly in JS, you may find after the re-write your framework of choice moves on without you and you're back to square one (though this is slowly stabilizing)
Obviously, there are exceptions, maybe the new language has persistent data structures or strong concurrency primitives that would be difficult to home grow that should move the needle of your decision making, in which case you may want to use new tech at the edges of your system, but you'll need to make peace with the fact you'll likely never finish the full migration and it will complicate onboarding, ultimately this should align with the personality of your team I think
A simple guideline might be to only roll a new framework if it's significantly advancing the state of the art WRT modern frameworks. That is, addressing conceptual weaknesses or the like.
OTOH, if it only offers incremental improvement and/or could easily be adapted by a current framework, then probably not worth the effort.
So, we're talking paradigm shifts here. For instance, we probably don't need any more HTML template-driven, reactive frameworks.
At the moment for example I'm doing exactly that in plain Racket (and also learning Racket along the way) and so far it has been a fun experience because you realise how much functionality we take for granted everyday when doing real work in the industry. I wouldn't definitely do any real work from scratch without a framework because using the least powerful framework out there surely is going to be way much more productive than starting from plain scratch.
Being that said there's I think another reason why there could be room for this: after you have tasted many of frameworks out there and having substantial programming experience you might want to roll your framework only if you are able to provide something innovative or extra value that none of the alternatives have, after all this must have been the way many of frameworks out there were born and became popular, but even in this case you might be able to achieve that by either contributing to a well known project or extend it through a plugin.
I do believe this is also referred to as simply "building an application."
But most people do, I can't count how many bad reinvented wheels I met in my PHP time. It was rather refreshing to use Node.js with NPM, where I could simply install all I need.
1) Unless you have a lot of experience, you don't know why various patterns have been implemented, or various compromises have been made, in existing frameworks. Frameworks aren't on the whole built by idiots, and if you think a framework has made an obviously poor choice, count the days until you realise why they did it. If you're lucky your choice will still make sense for your application. If you're not, I hope it doesn't take you too long to redo it all.
2) As the author says, frameworks are by nature declarative. This is great if you have really good documentation, but if not you're going to scream trying to work out the control flow of your application. "Why does changing variable X result in behaviour Y? What is even checking variable X? grep's returning nothing, the stacktrace is twenty levels deep and I don't recognise any of these functions. FUCCCCCCKK!"
You're going to forget how your framework works. If you don't forget there was no point building it at all, because the point of a framework is to abstract you from having to think about things.
3) Your project's not that special. The framework your considering will probably work just fine. It's worked for several million other projects, it'll work for yours. Yes, you might spend a bit of time trying to shoehorn something in. It'll be an order of magnitude less time than building an entirely new system.
4) It's much less useful for future work. A high-level understanding of building frameworks will get you a job at the Django Foundation. An high-level understanding of Django will get you a job everywhere else. And also at the foundation, come to that.
It was a fairly informative process. It taught me things I didn't know I didn't know. I value it in much the same way I value building interpreters. But much like building interpreters, except in a few niche areas it's probably not a good use of your time.
If you are interested, I did a live recording of building a blog from scratch [1]. From the content, to a decent framework to run it. This is more of a purpose oriented framework where you build the features you need.
[1]: https://idiallo.com/video/website-from-scratch-part-1-conten...
The 'framework' of your app is a collection of patterns and interfaces that allow you to implement your actual features as easily as possible. Whether you get that by importing 2M loc from the same source, piecing together tiny libraries that do one thing well, or write the entire thing from scratch, it's the same result.
I'd agree with you if I was using the definition everyone else is, which seems to only cover the fist case (importing a shitload of code from one place).
Using any pre-made library is a trade-off where you incur some technical debt in return for up-front speed. This is obvious if you look at the extremes (i.e if you have infinite time then I'm 100% sure that using Django isn't the optimal way to build your app, and if you're writing your own framework then by definition you're spending a bunch of time writing code that doesn't directly implement features. Everything else is somewhere in-between).
If you've written a framework that you can just sub in across multiple projects, then it falls victim to the same trade-off you'd have to take with an existing framework. Unless there's something about your framework that's straight up better for the general use case than existing frameworks (in which case well done, but "just build something better than Django/React/Whatever" isn't really advice that's useful for the average dev).
I think it's far more realistic to have familiarity with a set of patterns and small libraries for solving specific problems, and when starting a new project, port over the parts which are relevant to it. I've written tons of this kind of code in my career, and I don't see any point whatsoever in releasing my own open source framework. Instead, whenever I implement something that I think is interesting or might be useful later, I build a minimal gist that implements the concept in-case I need to refer to it later and re-use the code.
I think if I had my own framework that was robust enough to just copy wholesale between projects, and I had to train all of the devs on my team to use it each time, it'd just be the worst of both worlds. I'd much rather either use something that's reasonably standardized, or piece together the right set of tools for the job and educate and familiarise everyone else with it as I go.
But that also doesn't fail mysteriously in ways that even StackOverflow can't diagnose (like all of the open source frameworks do).
Bonus: it both translated text in HTML elements and JavaScript strings.
NIH at its best? But still today I couldn't find something so simple even for the non-technical translator. Because of CSV merging we could work with the translator in several rounds without hassle.
See, it seems like that _should_ be true, but in my experience it's the other way around. If I inherit somebody else's code, I can always (or at least, I have a perfect batting average so far) make sense of how things work and usually put together why he did things the way he did even if he's not available for me to talk to. I may not agree with his design choices, and I may criticize the way he approached things, but I can always work out how/why things are put together the way they are and how to move them around. Frameworks like Angular and Spring, on the other hand, are the product of dozens if not hundreds of people and are so generic as to be like a cloud of smoke: it takes weeks just to get my head around what (if anything) the framework is even offering. That would be fine if it wasn't for the fact that the framework ends up failing in mysterious ways that require days of debugging to work out.
Because it seems like with attitude like this, the art of creating a better framework will be lost to everyone but maybe some Brahman-like caste in ivory towers, who also don’t mind others leeching from their effort and telling everyone how you’re not supposed to roll your own.
I was tasked with delivering PHP code that would serve pages to the customer. For whatever reason I wasn't allowed to use CodeIgniter, which was what we used elsewhere in the company. I wound up creating a mini-framework to serve pages. It was extremely basic and limited, but in this situation it was what was needed.
It is the only time I've ever had to create a framework. I doubt I'll ever have to do so again. The beauty of frameworks is that they free you up to focus on your application and the logic to support it, rather than figuring out the best way to handle things like routing or database access.
See, I'm always a little concerned handing something like database access over to an unaccountable framework for fear that they'll do the wrong thing and a) it'll be my fault and b) I'll have to figure out how to work around it under time pressure.
> the only way to make a framework do what you want is an ugly way
Most of the time it gets ugly because the developer(s) doesn't know how to do that thing with that framework, not because the framework doesn't allow it, and would rather find an escape hatch than improve his knowledge of the framework and solve it the right way. I've seen this happen many times, both while studying and in a work environment. And if that wasn't the case it would be way more beneficial to tweak the existing framework and contribute upstream instead of creating "yet another framework" (obligatory xkcd: https://xkcd.com/927/), especially if you'd like to give back to the community.
Some frameworks have been around for decades (Django and Rails are good examples) and are still in active development, used by people in all sorts of industries, all over the planet. It's quite presumptuous to think your special use case hasn't been covered by at least hundreds of other developers, tens of which came up with what would probably be considered the right way to do that specific thing.
The only acceptable reasons for rolling out your own framework are: performance constraints that are imputable to the code architecture itself (hint: most of the time they are not), licensing issues, the lack of a framework for the language you're using and visibility (yes, releasing your own framework is a cool way of advertising your skills).
Is that because he's being lazy or because he has a 50-deep backlog of JIRA tickets, a project manager breathing down his neck and an expectation that every problem can be resolved in about an hour and definitely, positively, no developer should ever be "wasting time" reading documentation because we pay you to code, not to read, now get back to work you lazy worthless developer.
Of course you can still fuck that all up by starting a project and insisting on whatever is popular or that you're familiar with. There's plenty of 'framework-y' libraries out there for React that constrain you in all the wrong ways if that's how you like to roll.