You need to be able to run your system
catern.com
catern.com
Just this morning, I’ve been extending a Firebase app I developed for a very large enterprise customer, and I’ve been constantly reminded of some of the things the author speaks about as I tried to recreate the FaaS environment locally.
Not only did I waste valuable time setting everything up locally when the same would’ve been trivial in an old-school web framework; I now have an explosion of environment permutations and smells in my code, and I don’t completely trust that local and prod will behave the same so I will have to do extra testing.
Despite Firebase coming a long way - some of these local emulation facilities plainly didn’t exist or barely worked a couple of years ago - there is no way I would choose it for a new project. The only module that has genuinely saved me time is auth; none of the rest have made me happier or more efficient than a web framework. Even for prototyping (the reason why I picked Firebase initially) I would still pick Django or Express or whatever else over this. (And I haven’t even delved into the big elephant in the room, which is lock-in.)
Regarding costs and scalability there are a gazillion (fairly easy) things I could’ve done to achieve a similar cost range bringing my own stack.
FaaS and the like may have their particular use cases but for anything that has even the slightest potential to turn into a full-blown app I’d go for a good old monolith every time.
https://github.com/supabase/realtime/blob/master/examples/ne...
Even then, if I were to build an app with a predictable realtime requirement I would (a) find a full-stack framework with this as a core feature (b) use a framework without this core capability, but look very hard for the simplest library that would allow me to pull off the requirements, and only if all else fails do (c) which is to isolate the realtime component into something like Firebase and build everything else in the core stack. (I haven't tried Supabase, but from what I understand it might fall more into (b) than (c), since you can bring your own infra)
Why can't you just use a "live" development environment in Firebase?
Obviously this isn't practical for some cloud services due to cost (AWS Redshift is the first example that comes to mind), but I don't see this being an issue with Firebase. If a company is paying six figure salaries to their developers, surely they can afford to give them dev environments on whatever cloud provider they use.
Requiring an internet connection doesn't seem like it would be an issue either. Who in 2021 is building web apps with no internet access? I can tether to my phone practically anywhere, and even when I'm on a flight I have Wifi access.
The product I work on uses a few live AWS services in dev, and for the past 5 or so years everything I've worked on has needed some kind of internet access to run. I can count on one hand the number of times this has been an issue. Dev/Prod parity is a good thing in my books. It's been more common for me to run into issues on prod due to slight differences in how things work between running things on my MacBook and on AWS, not just due to services emulators being different to the real thing, but due to intrinsic differences between running things on a MacBook and on remote Linux servers (even with docker).
Defense contractors? Government entities? Entities worried about chain of trust in packages? Choice of upgrade path instead of forced? Ability to keep working when the net is down?
There's a reason for lots of people to still want 'on prem' and to not want to have the whole Internet as a dependency.
I think you partially misunderstood the parent comment - he was talking about requiring the internet for building the app ("Who in 2021 is building web apps with no internet access [while developing]?"), but you responded to using web apps w/o internet ("Who in 2021 is building web apps [for use] with no internet access?").
A bit of your point still stands, but, as the sister comment correctly pointed out, when you're building a web app for offline usage, Firebase is not the right choice of technology.
If your org has not approved any cloud services, then you can run everything on your development machine (database, webapp, cache, etc.), or a provisioned dev environment if the full stack is too heavy for a single workstation.
In utilities/industrial... docker, but not s3.
I have a live (as in online) dev environment with its own DB, which is good for most database reading/writing.
However, for Firebase Functions (FaaS), I really don't like the dev experience of using an online environment. You have increased latency for everything, but above all, logging is terrible.
So you have to set up the local emulators, which is not nearly as easy as doing `python manage.py runserver` or `nodemon` or what have you. I had problems with the setup this morning, and I've had to modify the code to write to the emulators instead of live functions depending on the environment (there is no way to do it purely through env variables, you have to do it though code).
I'll grant that it's improved massively over the last year or so - when I started the project I couldn't get the emulators to work at all, and SO and Github were riddled with issues on the topic.
On the topic of databases, I say it's good for most reading and writing, but last time I checked the data from emulated databases is not persisted so it's only good for fleeting tests, but not for playing around with data locally. I'm sure I'm not alone in that I often like to play around with bulk operations (such as big initializations or fetching) and manually explore the data (the "live" data explorer in the form of Firebase console web app is atrocious, there are infinitely better local solutions like pgAdmin, or the JetBrains' database explorers)
All in all the DX is just about acceptable, but I'm not impressed given that the product is almost a decade old, massively adopted, and led by Google. My original point is I believe web frameworks give me "tighter" DX, and I don't see a reason to pick Firebase over other options as a complete stack most of the time.
I agree that it's just barely acceptable. I wish the web console wasn't so slow, and the data explorer needs a major overhaul.
Over the past 20 years, a lot of software development has shifted to apps that run in the client's browser.
But, there are still a lot of computers which are not connected to the internet. Sometimes it is because of security, sometimes environmental factors, sometimes it is simply because the use case does not necessitate it.
I have a computer that uses a web app that isn't connected to the internet on my desk right now -- a printer.
A fast-paced edit-compile-debug cycle is a huge development productivity benefit, and doing it via unit/integration tests and what have you is nice, but not equivalent.
With a local instance, I can just spin a new one up for each test run. You can't do that with firebase projects (they do now provide an emulator which is just about workable).
Overall, I hope this is a trend that continues. The fact that these things are sometimes baked into their clients, which are usually freely licensed and developed in the open, available for patching, is something that hugely eliminates some of the lock-in risk associated with building on top of platforms. Worst case scenario? The company either goes out of business, deprecates some corner of its platform that you're relying on, or becomes prohibitively expensive to use, so you spin up a VPS and daemonize the client. This doesn't work for large projects, but for anything that is relatively small (but not so small as to characterize the development costs as trivial), it's sufficient.
Anyway sharing here just to note that while thankfully it was a CI issue and not a production issue, it did convince me that "test locally on your own machine" tooling still had some subtle gaps vs. what was happening in "the cloud."
I'm interested, what does Heroku offer in the way of "local testing" anyway? I just use a web framework's normal test runners in my local shell.
If you or anyone else wants to discuss this more and give us some specific feedback on what went wrong, please reach out:
* https://groups.google.com/g/firebase-talk - our mailing list, good for open-ended discussion
* https://github.com/firebase/firebase-tools - GitHub repo for the Firebase CLI and emulators if you have specific bugs to report.
I always run into this problem with microservice architectures, and it's a pain. There might be a B2B opportunity there, "recreate my distributed architecture as a service" or something...
Isn't that essentially Docker Compose?
I get the feeling that Firebase (and some parts of AWS) are meant as services for larger enterprises that are really hooked on GCP (or AWS).
Why do you want to be able to run the app elsewhere? To test new features? To reduce regressions? Reasonable goals, but at the end of the day the only thing that is identical to production is production. Unless you have a real time copy of all of the data and you continually run copies of all real time production requests, it's not production. It might be the same code and a very similar infrastructure, but without the same data and load, you are going to get regressions in production. Maybe it's flaky historic data or unexpected patterns of load, but whether we like it or not, we're all already testing in production.
I'm a huge fan of unit tests, CI, and all of the other common best practices to reduce the number of bugs that are identified in production, but you also need to have the kind of tooling and processes required to minimize recovery time and to be comfortable with testing in production. Small, easily testable, quickly shipped units of work and some flavor of feature flagging so you can dark ship code, recover quickly from outages, and do things like canary roll outs to ensure the new queries don't break with the production data at scale!
Additionally, you might want to create custom or corrupt datasets. Yes, you can theoretically add a flag, but then this flag needs to be checked everywhere and might come with its own bugs. Using the "customers_debug" table instead of "customers" table (for example) works as well, but then you replicated staging in your production environment - added complexity which is definitely _not_ needed.
Lastly, this misses the other points the article makes - a local copy allows you to freely play with the configuration, shut down related services etc. You can not do that in production.
But I'll give you that - resiliency in production is still very necessary and you usually won't be able to replicate everything locally. The ability to run local isn't everything - but that was not the point the article was making.
I was unable to run the codebase for a project I joined once and switched to using heroku one-off dynos (servers from that app's cloud provider -- these nodes are identical to production nodes excepting that they don't receive external requests and can be segregated in logs) as my development environment ... did some dumb things with ssh and tmux and was able to persuade all my tools that all the remote node's processes were accessible from localhost -- with a little finagling being able to use all the tools normally used for local development and debugging directly against production provided an extremely productive set of lenses for peering into the behavior of production resources -- especially when onboarding into the new company's codebase ...
That is letting the perfect be the enemy of the good. Sure, there are classes of bugs you'll only be able to find in production, but there are also large classes of bugs you can find in a development/test environment. In general, it's a good thing to minimize the bugs you find in production. It's certainly less stressful.
For me, the article mostly concerns about the stateless services. Let me rephrase the article's main "ethos" in a different way, if you may..
If all your stateless services are running in the same kubernetes cluster in production, and you accidentally destroy the cluster (ahem terraform ahem), would you able to automatically re-deploy them from scratch? Does anyone really understand the dependencies? Is there any circular dependency which would require some hack?
And if you can't do that in the local development environment, what makes you think that you can recover from a production incident like this?
You need CI and regression tests running on smaller stages with mock data, because a bad rollout can take some time to revert, even if it didn't include a rogue database query or migration that impacted your data. Knowing how to rollback quickly is important, but a penny of prevention is worth a pound of cure with large services.
And IMO you shouldn't let people "test in production" when the changes involve a production database; those should be off-limits. It's all fun and games until someone forgets which instance they're logged into.
You'll still get plenty of regressions, but the goal is to avoid major issues that impact most of your users/customers/etc.
Most organizations, for example, do not have hyperscale volumes and face no insurmountable barriers in setting up parallel environments for testing.
Why? Because business doesn't give two shits about your productivity. They care about one thing, the productivity of the software. And so they will buy other goods and services that will at some point be closed source. And then you're the one that has to integrate them.
Are you deploying to GCP? Already you can't run your full stack locally. My last week was spent hacking Google's Anthos CLI tool, nomos into a kpt function. Kpt is open source, nomos is not, it's not even source-available. It's just proprietary. And the platform it interacts with, you can't run locally. Go ahead and check out the troubleshooting page for one Anthos tool:
https://cloud.google.com/kuberun/docs/troubleshooting
No sections on spinning up a local env. You can't self-host or run locally, Google won't let you.
This isn't isolated to just Google. People need to get it into their heads that open source is the realm of business and not the realm of hackers. You're a cog in their machinery.
If you can't run free software for your whole stack, then your troubleshooting steps will necessarily involve understanding at least two environments, one of which will be a very black box.
Devs, if you want a better world, start contributing to copyleft software.
Your version of "a better world" is one in which authors, musicians, movie producers, artists, and all other creative types are free to make money from their works, but not software developers. When they try to do it, it's immoral. No thanks.
The more we can build up free software as an alternative to open source, the more the business world will be forced to use it. They need your dollars more than anything.
Contributing to open source software is one of the most effective forms of altruism in history -- but it is altruism! Permissive licenses like MIT and LGPL embrace and enable that altruism. Copyleft licenses and the FSF are built around the idea that you are entitled to it, and it is immoral to write commercial/proprietary software. That is fanatical and unhelpful.
Unless you think it's always immoral for anyone to make money from a copyrighted work? In that case I'd be a lot more interested in hearing that perspective, because at least it's not hypocritical.
Also, with a novel on paper, you generally have similar rights as the GPL grants on software, such as reselling it, making modifications (I can just write on the paper!) and that sort of thing. While scanning and distributing copies is not allowed, quite some GPL companies also crack down on this by simply not providing updates anymore (I believe this approach is taken by the Linux grsec patchset developers).
Did we learn our lesson from the outage? A big _nope_. I suspect it's because being able to run a somewhat complicated system on local requires thinking in brand new ways with benefits that aren't very obvious from the outset. After that experience, I sympathize a lot with the author's points and hope to work in an environment (ha!) where spinning up a docker container is all it takes to have a _full_ dev environment.
IBM got a bad shipment of laptop hard drives that exhibited a MTBF of about 2 years, and our equipment dept bought a stack of laptops from that batch. Over a summer we had 6 machines go belly up. Mine was number #5. People still looked at me like I announced that I had stage 3 cancer. Oh you poor poor man. I found this reaction disappointing.
By then the process was about as documented as any I've had. It just took me a day to get it up and running (because the base image left a very slow step until after 2nd boot, which I still maintain is dumbness squared).
A coworker from that cohort had an experience that I still use as an example. He tried to put his work laptop in the back seat of his car. He missed and hit the door frame. Killed the laptop. Similarly, taking your laptop down the stairwell could be a one way trip to the trash bin.
If the information is important for us GET IT OFF OF YOUR COMPUTER. As soon as you know. Put it in storage, or at the very least in some coworker's head/computer. If you do this, consistently, then losing your machine is a shitty inconvenience, but nothing more dire than that.
My co-workers were appalled, but I had less than a day of downtime each time. With continuous backup to an external drive (always on while I'm working), a password manager I can also store certs in, and keeping everything important in a cloud-backed git repo, there isn't a single thing on my work computer that's hard to replace. It's been fantastic.
Do you have any good links?
I only write code on my laptop, but I sync my projects folder with my cloud desktop. It's saved me from several crappy accidental losses.
I built a thing with a local first view and I still battle this everyday because whether we succeed or not will not be dictated by whether it worked locally, but by actually shipping a product that people want and are willing to pay for.
Tradeoffs. Sometimes you have to just let go of the purist view point.
Whenever I hear these statements, it always sounds like "I need to have an identical copy of this skyscraper in order for me to be able to replace one tap on floor 42."
Also good luck running a system that operates on few hundred of terabytes of for instance YouTube data locally.
Also running the whole system locally is usually a pretty good way of creating a "distributed monolith" -- yea, there might be microservices, but also a dozen of assumptions here and there that different parts of system are being deployed simultaneously (usually they are not), or that certain distant parts of a whole system share some behavior that can be changed simultaneously.
So no, you don't need to run the whole system locally. On the contrary, you need to be able to run smallest part of it (hello, microservice) locally, and that part should be responsible for one thing. APIs and frontends can easily share JSON schema to make sure they send and receive valid data, and each service can have tests against that schema, the ultimate source of truth for them.
Boom, suddenly I can develop "big system" piece by piece in isolation on my 7-year old macbook with no problems against tests / storybook / debugger.
The building industry also has the problem the OP describes:
https://i.stack.imgur.com/yHGn1.gif
The problem as I see it is that people who go all in on unit tests tend to be dogmatic about it and suffer the above type of issue whereas the people who want, broadly speaking, to run things as realistically as possible are pretty aware of the real constraints.
Moreover modeling larger is also frequently cheaper because the real thing often comes for free while creating elaborate frequently changing unit test mocks has very high opex.
Why can't that be scaled down to a few hundred megs of data running through a system, end-to-end, on the local machine?
Not being able to scale down seems like a code smell to me. Is there so much overhead in the microservices that even at idle, or very low usage, they can't even be started?
Again, talking here about the "run the world and click on things" approach vs developing against tests / API schemas / whatnot.
Postgres performance will be different to prod, yes. That's all part of the realism - cost trade off. All models are wrong, some are useful.
The point is that diving headfirst into making matchstick model unit tests is dumb when you could build something to test against at reasonable cost which is a lot more representative of reality.
IMO this is an obvious point if you adopt a cost/benefit approach to development but it's often impossible to see for the dogmatists.
To be honest, short of issuing every developer with their own production-grade environment to debug against I am not sure what exactly would satisfy this line of questioning.
You don't need hundreds of terabytes to run an app with test data.
Monolith's aren't bad.
NASA had two versions of the space shuttle software developed in parallel by teams at IBM and Rockwell. None of the devs had a whole space shuttle.
Well you could just mock the data and code, but then the according to the author:
> "Run an individual service against mocks" doesn't count. A mock will rarely behave identically to the real dependency, and the behavior of the individual service will be unrealistic. You need to run the actual system.
What matters here are the odds that the production system will behave the same way as your tests. For functional requirements (not performance), the normal situation is that if you replace your data, those odds do not decrease a lot. If you want to test performance, that changes, and you may need an environment as large as your production one.
Yes, but isn't that exactly what the author suggests?
If you replace all of your data, it's still your code running there. But you must have some data, or your code won't run, and it must look like real data, or your environment will be fake again... where "look like real data" is completely problem dependent.
> Also running the whole system locally is usually a pretty good way of creating a "distributed monolith"
I'm not sure how you came to this conclusion. Nowhere does he mention monoliths or microservices. Even if that distinction is made it is good practice to be able to run your architecture on a single box as a goal.
Going through that exercise alone will force the designer to think about how the application scales up and down, having representative samples of data for test suites, coupling between services and so on.
The orders of complexity increase dramatically for each component running on a separate machine for the system under inspection.
Being charitable, the things you mention do have their place but they don't address many of the problems you'll face when building a complex system.
The key is to be able to quickly undo some scary new thing and get back to a known-good point. Also, you need to make sure the testing you are doing is easily distinguished from normal business activities. For us, the humble feature flag is all it took to get unblocked on moving mountains of complex scary things to production. Being able to reassure the customer that we can instantly revert a piece of experimental functionality has been a game changer.
We used to spend months agonizing over how the test environment is such a poor facsimile of production and complaining about vendor XYZ not setting up various things that we would need to prove our code works as expected. We used to make insane bullshit promises about how if things worked well in staging the move to production should be flawless. Those were some dark times for us.
1. Developers can be far more productive when they can run everything locally. 2. Until you’ve deployed into and tested against production, you don’t know if your shit actually works.
In fact, they’re complementary: developers should have excellent local dev facilities and excellent facilities for running experiments in production.
Everyone on the team is aware of the caveats as well. We say things like "It worked on simulator, but we still need to test in live".
Nowadays, I accept this reality is largely impossible and you must always draw some boundary. This doesn't mean all your developers should use a shared MySQL because nobody knows how to bootstrap the database- but it means you have to consciously decide where you sit on the continuum. Always expecting to run the whole system on your laptop (or even in a cloud) is also an unreasonable expectation, unless you're Netflix and your revenue per employee is into the millions. The reasons for this are many and complicated, but at a high level the work required to make it happen will cost too much and won't be a priority.
I'll quote here from the excellent "Testing Microservices the Sane Way" by Cindy Sridharan:
> asking to boot a cloud on a dev machine is equivalent to becoming multi-substrate, supporting more than one cloud provider, but one of them is the worst you’ve ever seen (a single laptop)
https://copyconstruct.medium.com/testing-microservices-the-s... "Full stack in a box- a cautionary tale"
That said, such design is perfect for precisely this kind of customisability.
Edit: I thing thought that Apple is to blame here by setting the defaults to ridiculous values.
If the browser's default style was a little better, it could be even more elegant - not a full window width, for example. Could browsers make such improvements to their default style? Or are assumptions about the default baked into sites in such a way that changes would break them? Could browsers detect those assumptions and provide an appropriate default accordingly?
There is another blog post written by the author that docker containers are harmful that I strongly disagree with but not totally unexpected given the design choices of the site.
Clicking through their profile to their business/project website was a loading screen for 5+ seconds. I never got a true "professional" impression as I closed the window before it finished.
Project 1, Elixir.
> Run an individual service against mocks" doesn't count.
We got an error in production a couple of weeks ago. The code was pretty old so it was a surprise. Found the problem, a rare case we never thought about, and the culprit: almost all the tests using that function where run against a mock of that function because it requires quite an elaborate database setup to work. The function itself was tested only in a couple of unit tests which exercised only the naive cases. Big mistake!
Project 2, Python.
We spin EC2 VMs to run some long running CPU bound task and kill it. Of course we can do that in a development environment. We're already using SQS in development. And yet spinning remote VMs takes longer than locally. So we build a couple of scripts around VBoxManage (VirtualBox) to run those VMs locally. Booting them from an SSD is really fast and it works when working offline. We have two classes with the same methods, one wraps VirtualBox, the other wraps EC2. The configuration file of the application picks the right one given the runtime environment.
> Developers of large or legacy systems that cannot already be run in their entirety during development often believe that it is impractical to run the entire system during development. They'll talk about the many dependencies of their system, how it requires careful configuration of a large number of hosts, or how it's too complex to get reliable behavior.
> In my experience, they're always wrong. These systems can be run locally during development with a relatively small investment of effort. Typically, these systems are just ultimately not as complicated as people think they are; once the system's dependencies are actually known and understood rather than being cargo-culted or assumed, running the system, and all its dependencies, is straightforward.
In my 28 years of professional experience working in 9 different organizations, only one of them did fully run the core system in development, and another two of them could have done that with enough time and effort. For the other six, there was no reasonable way to run the full system in development.
> Typically, these systems are just ultimately not as complicated as people think they are
In fact, most long-running systems are quite a bit more complicated than people think they are.
Granted, a variable but usually large chunk of that complexity can be removed/simplified, but at their core, these 'legacy' systems, the ones that make real companies work, do their job because they have, over many years, figured out how to successfully come up against all kinds of real world complications.
To be clear: I get what (I think) this person is fundamentally trying to say: the more expansive/end to end a development environment is, the better. That it's better to have as wide a view into the system you're working on as possible. Good stuff!
Most places don’t have a way to run in development because they haven’t sat down and gotten the application to a running state. Every year it gets ignored, things get harder and more tedious but never impossible
I think docker is an obvious enabler of this because people can just spin up an image from 6 years ago in a container and keep running against old libraries/dependencies. Even worse is depending on that container makes it difficult to upgrade dependencies to modern versions of software because the container is so old.
Maybe a corollary to the article is, "without constant maintenance, software becomes un-runnable."
If I'm creating tax filing software, do I need to spin up a local instance of the IRS to test with?
No, integrate directly with the test endpoints provided by the IRS [1].
[1]: https://www.irs.gov/pub/irs-schema/Portal%20Transition%20Gui...
> "Run most services in a mostly-isolated development environment, calling out to a few hard-to-run external services" doesn't count.
Only ~20 years here but I've worked almost exclusively on three or four large systems with test environments despite their complexity. I think it may be the nature of the companies/business that makes the difference.
Ancient (1980's code) hospital ADT+billing system (medseries 4 if anyone cares) on an AS/400. We kept the last generation of hardware around to run the test/demo environment which was end-to-end connected via HL7 to all the other test environments. That included the other thing I managed, Cerner Millennium, a hodgepodge of a medical records system on AIX and Oracle, where again the old hardware got used as the test and build systems. The test environment was critical because even though it was made up of a dozen commercial software packages talking to each other over HL7 and flat files (and serial for lab devices and who knows what else) it was critical to make sure the whole environment worked when doing upgrades or new development. We kept a full copy of prod data for final integration testing. The tooling for (most of) the software supported replicating environments.
The largest example is Photon/Ubiq at Google; end-to-end test environment running with deterministic sampling to save resources at all stages of the pipeline so everything from injection through logjoining and aggregation to the final database updates were continuously running the latest code for testing before a release to prod. Thousands of prod machines devoted to it. That was in addition to the integrated tests and unit tests during the development process.
At FB the statistical counter system I worked on also kept a continuous test pipeline running, again with event sampling to reduce cost, to ensure everything would work properly before a new release.
My takeaway is that where it matters companies are willing to spend the money and people to have full end-to-end testing, often in realtime, and have some leeway to save on resources with subsampling, smaller datasets, etc.
If I understand right (and I may not, both your description and the OP), this may violate the OP's prohibition I've quoted below. What do you think?
> "Run an individual service in a shared stateful development environment running all the other services" doesn't count. A shared development environment will be unreliable as it diverges more and more from the real system.
The OP seems to be awfully strict.
But isn't it the "shared stateful development environment" the OP is saying is unacceptable? Since every developer doesn't get their own "last generation of hardware"?
But I may be misunderstanding the OP?
I am not trying to critisize what you did; I am trying to understand and possibly critique the OP. The OP seems unrealistic to me, despite it's protestations that it really isn't; but I may be misunderstnading the OP.
In that sense, sure; we had one active directory, one physical network, one physical workstation to trigger commands, one office chair, no test version of me, etc.
So if the OP's language is not careful then it could include things in "the development environment" that don't make sense.
Be prepared to run _any part_ of the system locally, like databases, message queue thingies, web applications & services, etc. This gives you lots of flexibility and control you wouldn't have otherwise. The up-front work can be painful, but it pays off over time.
Know how to proxy missing parts: For example, have a local nginx server proxying requests to a shared server that can handle some things.
Set up integration tests that actually bang on running applications (over, say, HTTP) instead of testing little blocks of code. Even if these only run on-demand (full runs may take hours, require overnight scheduling, bleagh) they'll be super-handy.
Ultra-distributed systems are a nightmare, but you do what you gotta do. In the end you'll end up being your own "devops" engineer and know your architecture better.
Edit: A lot of people think of "mocks" as unit-testing types of things, but "mocking" an entire web server is often the way to get out of an integration-testing pinch, especially when some production-only remote thing interferes with the critical path you're on.
After working on developer infrastructure at a big company for a few years now, what I've realized is that most companies don't spend very much time at all designing the 'molds' and 'fixtures' their software teams need in order to manufacture a worthwhile object that they know will work. They pay lip service to the importance of testing and solid dev tooling, but they're reluctant to invest in it. (I don't blame individuals here — they're behaving rationally given market pressure — but it's a sad state.)
Plot twist: can't run any virtualization software on my development laptop and we are cut off from "the internet"(so no ssh or remote cloud).
Advice?
Developers need full access to their machines or they are stifled.
This organisation is not interested in you doing quality work.
Your options are therefore to either resign yourself to not doing quality work, or leave that organisation, ideally making sure that management knows exactly why you're going.
(Of course, the chance of management actually caring is about one in thirty seven trillion - if they cared about quality of work or employee satisfaction, your machine wouldn't be locked down. So not bothering to tell them is fine too. Tell them if you're feeling charitable)
Barring that, if you have admin rights and your laptop is running Windows you can enable Hyper-V[0] and get a great hypervisor that comes in the box. (If you’re already using WSL2, you’re already doing virtualization). If you’re on macOS, try to find something that uses Hypervisor.framework (multipass[1] is good if you can get along with Ubuntu) so you don’t need to install any kexts.
But this is really something you should bring up with your manager, it sounds like it’d be difficult to get any good work done without being able to run tests in a not-production environment.
0: https://docs.microsoft.com/en-us/virtualization/hyper-v-on-w...
Within the boundaries of the space is a solution, we just don't know where the boundaries are most of the time.
A bunch of years ago we were under a similar predicament, we needed to tune some software that was data dependent, yet we were cut off from prod data, and the test data as hot garbage.
It took weeks to get new software into production. We had one code push left before launch but we still needed to tweak with production traffic. So we did what everyone does, ship remote firmware updates. In the last code push we included `eval` and the ability to POST code to an endpoint. We were just doing some query rewriting, but the code could have run anything.
Once the last build made it through dev and test, we had a serverless deployment platform that allowed us to dynamically update the code in realtime.
yolo
But if you want a solution to this specific problem, the first suggestion I'd have is to install a non-root-requiring package manager like Linuxbrew or Nix (which is better if you can get a directory /nix writable by you, but workable even if not). Then you can install software without contacting the sysadmins and waiting.
Alternatively, do you have unprivileged user namespaces, i.e., does "unshare -Urm" work and get you an apparent-root shell? If so, you can install rootless Docker in your homedir, or use https://github.com/containers/bubblewrap . (A few years back I wrote https://github.com/twosigma/debootwrap which ties together bubblewrap and debootstrap to get you an unprivileged Debian chroot.)
In Elixir, what you've got is a lot of Elixir. Sometimes you need other external services and there's some work there, but you can often get a surprising amount of your system in one codebase, running on one more instances of the VM. It makes testing SO much easier.
Just imagine an Amazon developer running amazon in its own laptop (or cluser, or server, or whatever... multiply it by the number of devs... it is insane).
I also came on once to a set of projects where there wasn't a good separation of the environments; test and prod shared the same database, b/c it was "too hard" to replicate the prod data back to test. Consequently, no real DBA work could be done b/c it couldn't be tested, only small hot-patchy kind of work.
Separation of concerns is a slightly different problem but the root cause of the issue is similar, you need to be able replicate the complexity of your system in production as much as possible in a lower environment, otherwise you won't be able to make as significant changes as you'd like to your system.
I don't think this is terrible advice, while I recognize there are limits to what is possible to run locally/in lower environments, the central notion of 1) separation of concerns and 2)matching the complexity of the lower environment to production as much as possible is I think good advice.
> Developers of large or legacy systems that cannot already be run in their entirety during development often believe that it is impractical to run the entire system during development.
This is just a flat out ridiculous statement this author makes. My wife works on an automation system intended to respond to incoming events in the form of positive tips in surveillance system collections that in turn cues further tipping rules to retask those collection systems. One of the external dependencies is the entire spy satellite control system of the United States. How are you supposed to run this in your development environment? Some things you have to mock.
They emphasized that this was their first time of running everything in the real environment although I'm 99.999984% certain that they've tested it in the closest setup as possible. I'm also certain that there must have been a few glitches here and there, but the overall system worked. There's a process of mitigating and containing the faults and failures, and NASA is known for that.
I don't disagree with the principle of OP, but just repeating it in a dogmatic way doesn't make things better. I want to see practical examples and ways to fill the gap (between the dev and prod).
The fact that Perseverance and other Mars rovers have been successful is amazing and took an incredible amount of work to accomplish. These are complicated systems that were vetted using models and simulations without ever having been run "in production". This comes at a high cost.
Critical software is never tested in production or run "in system" before it is deployed. Airplanes, banks, medical systems all require extensive validation through testing on models and simulations. You can't test your changes for the first time on a live aircraft or living tissue. Costs reflect that.
The truth is, a lot of software is not critical. You can get away with hacking / trial and error type development and never fully understand the system you are helping build. Frankly there is a lot of money to be made providing brand new services that are unreliable or quirky or ephemeral because those services never existed before.
My point is that how you test and validate your software product depends on your application. Sometimes the costs don't make sense to "run everything" and sometimes its physically impossible. I agree that you should always advocate for the highest fidelity testing your business case can afford, but be prepared to settle for less than everything and rely on your engineering skills to buy down risks in the gaps.
I worked in so many places where nobody knew how the system worked and the most invaluable thing I do is to bring the production system into localhost.
It pays off real quick and people get amazed how fast I can solve long standing issues in the system by using tcpdump and co.
While I think it's useful to be able to exercise a system for real, I generally prefer to do it in production rather than attempt (and fall short) of reproducing production in an isolated environment.
Meanwhile, I'll invest my time in defining and understanding the contracts between services, and develop against those contracts so I don't have to run the entire stack of connected pieces every time I want to make a small change.
In my experience, if someone claims that someone else is always wrong, then he is wrong.
That's pretty harsh, as it seems to rule out any system that integrates 3rd party features, such as payments.
Running a service that handles payment without testing the entire flow is insanity.
I've been at organizations where local development required connecting to a MSSQL instance in a server closet at HQ; This is unacceptable, and devs will constantly be at war with each other.
Mocking your analytics provider (or whatever) seems much less problematic to me.
(Then, something more complex or special purpose, like say Amazon Elastic Transcoder or something).
Due to the hilarious nature of the business, getting an S3 account to use for 'local' (i.e. dev machine) development was a huge chore.
We wound up creating an Interface to abstract our usage of S3, and wrote a 'local' provider that just used the local filesystem. Of course, many use cases that would not work....
But the big -advantage- of it was, we at least knew that if we ever wanted to switch away from S3, it wouldn't be hard. The Interface had a spec that could be used to ensure that any plugin would work with our application's usage.
It wasn't -that- much more work.
Which violates these requirements, right? Using something that mocks S3 instead of actually being S3?
(If I was going to do that, I'd just use open source Minio S3 clone, assuming it adequately cloned the S3 features I needed).
All that said, I agree with the sentiment of the post, testing the system as a whole can reveal issues before they get to prod. But maybe that means you need to harden your contracts and internal SLAs.
This can probably work with more complex services too.
But I'm not sure that would comply with the requirements, they seem to forbid running against external services at all, no? '"calling out to a few hard-to-run external services" doesn't count.'
I think every team needs to decide how much to follow the original advice, there is certainly a balance between simulating external service locally (with possible associated behavior changes) or using an external service.
If you do want to run everything locally, there are servers which offer S3 API, like "minio" and "fake_s3". Probably would be very nice for localhost operation. But if you need a complex thing like Transcoder this is not going to work.
Not usually available for most other AWS services you might be using though. Say, Elastic Transcoder for transcoding A/V media.
One approach I've suggested in the past is that every time a developer has concerns about a design and is over-ruled by a manager, the developer should take a jelly bean from a jar that starts full of jelly beans. The team can keep track of the weight of the (contents of the) jar, and use the inverse of the "fullness" fraction as a multiplier for how long a ticket will take.
For example, if a story has a complexity of 2 points, but the jar is two-thirds full, then the ticket should be expected to take as long as a 3 point ticket would. That might not be exactly right for every ticket, but this process might at least be more accurate than ignoring the hidden cost of accumulating technical debt.
Of course this means that once the jar is empty, all estimates become infinity, or undefined, but if it ever gets to that point then at least the developers will have had some jelly beans to cheer them up. Similarly, a manager can have the satisfaction of "seeing" developer productivity increase when, at the end of a sprint focused on paying down technical debt, the developers add more jelly beans to the jar, to restore the fraction towards unity.
Eg You are writing software for a planned device which is operating on Mars. First, the device doesn't exist yet, second, it's hard to simulate the Mars environment locally.
I did such projects for very expensive devices. Think of nuclear power plants or formula 1 race cars. Thanksfully you cannot complain loudly "I need to be able to run this thing locally!". Your job is also to figure out how to simulate the environment, and to do a proper job without being able to test everything beforehand.
As long as you have strict contracts between parts of your system (whether that is in a form of microservices, IPC, classes or whatever else), a change made in isolation should be perfectly fine.
Microservices, no.