Hacker News Reader

Comment by fasteddie31003 | Hacker News Reader

I was interviewing for a platform engineer role for a company that from the outside their application doesn't seem too complex. Through the interview process, I find out that you cannot run much of their app locally to develop against and their staging environment does not reflect their production enviroment well. I asked them how they develop their new features if they cannot run the whole app locally and if they intorduce many bugs without testing locally. They response was along the lines that they hired good engineers that could understand all the complexity before deploying new features.

I believe having a solid development environment where you can develop and test the whole stack is key to making reliable software. Is this common at other companies that you cannot run the whole app locally to develop and test against?

View on HN

> Is this common at other companies that you cannot run the whole app locally to develop and test against?

Yes. I work in B2B banking software, so having an "authentic" environment to test against is extraordinarily complex. Even the big vendors in the space seem unable to maintain accurate facsimiles of production with their own decades-old core software, so smaller vendors stand no chance unless they adjust their entire strategy.

What we ultimately had to do was negotiate a way to test in the production environment in such a way that all the activity could be tracked and backed out afterwards. A small group of important customer stakeholders were given access to this pre production environment for purposes of validating things.

This is something that has never sat well with me. But, the way the business works the alternatives are usually worse (if you want to survive long-term, that is). We wasted 18 months waiting for a customer to have a test environment installed before we learned our lesson about how bad those environments are in reality.

Worked with banking software before; it can be incredibly complicated. I worked on proof of concept infrastructure for a Websphere monolith that ran on AIX servers. It had at least 10-15 services it connected to and each one of those were huge Java monoliths, too. Some of those apps had 8+ copies of /each/ environment (8x dev, 8x test, 4-8x stage/uat, 1-2x prod) so you could do integrated tests without stepping on other people or something software (software B might have a dedicated test environment of software A)

This works well. I've seen deployments where you have test1, 2, 3 etc, and you can "lock" the environment to yourself for a few hours.

I haven't found a wholistic integration testing framework where you can effortlessly mock out upstreams as running services, so that you don't have to ham-fist integration testing code into your app or rely on actual, live upstreams which are probably prone to breakage anyway.

I've done things like this in ad-hoc setups with Docker Compose and environment variables, managed with shell scripts. I would be very interested in a more thoughtful alternative that was integrated with testing.

I've done a thing automating whole environment setups with VMware cloning (undocumented feature because there is it's own product on top of it) and Docker via CI, in a bank. Basically we did golden images of VMs with databases having all accounts for test cases and prepared images for services to be deployed, while getting all necessary variables from CI and VSphere to write down in configs and Consul/Vault via scripts. Initial version control for the env could be done via same prefixes in the Git branches of services, after the deployment each env had a way to deploy whatever via CI's Web UI or API.

Running automated integration tests on the whole env is just an added step in the pipeline.

One way to balance is contract testing where you run a test suite against your mock/stub and the same test suite against the real thing. Your mock/stub would usually just have hardcoded data that aligns with the test suite to pass.

Another way is creating separate code libraries to abstract away external services. For instance, you might write ObjectStorage library that has real integration tests that use S3 but in your application you just stub out the ObjectStorage library. A lot of frameworks have verifying doubles or "magic mocks" or similar that will stub the method signatures so that can help make sure you're calling methods correctly (but wouldn't help catch data/logic issues).

Live upstreams are also prone to inconsistent state and race conditions--especially when you try to run more than 1 instance of the test suite at once.

For web services, I also like using "fixtures". You just manually run against a real service and copy the requests to a file (removing any sensitive data) than have your stub load the data from the file and return it. If you hit any bugs/regressions, you can just copy the results that caused them into a second file and add another test case.

Shameless plug: this is exactly the spirit behind what we're building at Kurtosis! (https://www.kurtosistech.com/) You get an isolated environment running real services connected to each other, so if you have a container with your mock then from your app's perspective it will look just like the actual upstream.

I have looked at your docs and don't quite get it. Is the idea that I have say 10 microservices that app-in-test will / may use, and then I set up the 10 instances (somehow all in one container?) and then can test that.

I think that seems sensible, but how is that different to 10 containers ? Is the issue something about simpler network management inside a container?

It's funny, I worked in banking about 15 years ago and the technology stack was the same. Java monoliths, Websphere on AIX (plus mainframes naturally)

Same here in e-commerce. We end up testing in production environments with an element somewhere that's in a development environment that can be tracked and actively watched. Amazon notoriously does not have a production environment, and eBay's is just unusable.

im pretty sure amazon has a production environment... just their staging is problematic. It's incredibly expensive to run a copy of prod and keep it up. You essentially have to get paged for it doubling your burden.

Yep, I mistyped. Amazon MWS doesn't have a staging/sandbox environment that's available for third party developers to test on.

My gf works in UX for a major bank and they simply issued her a bank account in her name (with her SSN etc -- it's a real account, with ATM and debit cards and the like). If she and some co workers want to test something out relating to accounts they actually use these accounts to minimize this kind of problem.

They must have some special code that says that these accounts can have their fees zeroed and probably are subject to some special out of band audits. I don't really know the details of course but I was actually kind of impressed.

(no she is not actually a customer of her employer)

Previously led data teams in banks and you’re right. There are dozens of flags for test accounts. When auditing, we screen out these accounts. Of course, with a proper data cleaning process, analytics gets a lot easier

That reminds me of "Mostly Harmless" from Douglas Adams where Ford Prefect breaks into The Guide's offices and gets himself an infinite expense account from the computer system. That's a dangerous thing to have.

Problem is, anyone with a good story to tell should really keep their mouth shut.

I worked at banks too but mostly segregated from the core business. Today I see observability tools as the solution for problems like that. Specifically tools like Lightrun where you can observe production, mitigate and fix directly from the IDE.

I wonder if other developers are familiar with this approach?

We are working on Videobug - it records code execution on production and plays in the developer IDE. Our Show HN here: https://news.ycombinator.com/item?id=31286126

Yep, work in b2b banking as well and there is a huge issue with banking ‘test’ environments. Our partners (100s of companies, mostly banks/brokers/insurers/etc) mostly don’t even have test envs and if they do, they don’t represent production at all (sometimes the api is a completely different version etc). Our software allows to spin up a test/dev env locally or where ever but nothing much works without the partners and to dev/test those, those, you will have to dev/test against their production systems with real money.

In the beginning (more than 20 years ago), I insisted that we implement test systems ourselves so we didn’t have to test in production; that was naive; it was expensive and there is so much complexity that this does not work well.

I'm in that boat. We have a "microservice" code base, and spinning up the whole "app" is a Herculean task.

Even if you do manage to get it all running, getting the multitude of database with useable data adds an odyssey to your day.

We have some parts automated, but ops cannot keep up with all the changes in the micro apps.

Back when I worked on a mono-rails app with a React front end, getting things up and running was as simple as running the JS server, rake db seed and then rails s. We used puppeteer to test FE, and rspec for the BE. Two test suites, but we knew if something was wrong.

That same place sometimes had a net outage, and I wouldn't even notice because I was running everything locally.

Now I can only get everything running locally with confidence, if I block out an hour or so.

Honestly... microservices are great in theory, but make dev a huge pain. Docker does not mitigate the pain.

That's the reason why I walk away as soon as I hear "microservices". It's cargo-cult mentality where complexity and the issues you speak of above are considered a feature.

Yep I've had the same thing happen - a company I worked for went from having a monolith that could run locally, then everyone drunk the microservice Kool-Aid and we ended up with no way to run the microservices locally in a way that integrated with the monolith. There were staging sites with everything but there was one staging site per team.

The worst cases are startups adopting microservices. I get where they are a solution to a large company with multiple teams hitting the pain of monolithic development, but a team of half a dozen adopting microservices is a big red flag.

Microservices are not even great in theory if you spend just a bit of time deeply thinking about the complexities that microservices introduce compared with simply using libraries/modules to organise your system.

Quite common.

Usually someone develops a remote or mixed local/remote development environment which is easier to use than configuring the full stack to run locally, and then the ability to ever do so again atrophies until nobody can figure out how anymore.

In some companies I've worked at this is fine and not a major setback. In others, it's been a mess. I don't think that "can run the stack locally" is a good litmus test for developer experience - the overall development experience is a lot more nuanced than that. "How long does it take to get going again if you replace your computer" and "what's the turnaround time to alter a single line of code" are usually better questions.

I work at a VOIP services provider and we operate multiple brands which are companies we’ve acquired over several years. One thing that has rang true of every company we’ve acquired is that there is typically barely even a way to run one piece locally, let alone the whole thing.

Some of these when I took over there wasn’t even version control, and they were being developed by FTPing files to production server or even editing them live with vim or something. Certainly no reproducible containerization or even VMs to speak of.

I agree completely with you about having a solid development environment. I’ve spent the better part of a year creating such a thing for some of the brands, and it has increased our time to ship features and fixes probably ten fold.

> they were being developed by FTPing files to production server or even editing them live with vim or something.

Many editors / IDEs (e.g. Notepad++) can directly connect to the FTP server, so working on a FTP servers looks almost same as working on a local folder. You just make changes to the files, and then refresh the web browser to see the changes.

lol, yes, sure. But why don't you sound horrified?

Version control at least is essential for something of any size with any number of people working on it. You must be able to "revert" a set of changes quickly and reliably if the application has any importance at all.

> You must

Don't want to be rude here, but version control is clearly not "required" in the strict sense. Plenty of software has been developed without it. Now I wouldn't want to work at a place without version control either, but it is as required for developing software as seat belts are required for driving a car. Less even, since at least for seat belts they are required by law.

People manually control versions all the time. Ctrl + C, Ctrl + V on a project folder to make a version. I have doubts that someone developed any nontrivial software without that approach.

I work at a company that has similar practices to what you describe (we don’t run our services locally). While it’s hard to get used to at first, I think the silver lining is that it forces everyone to write good unit tests, since it’s the only efficient way to run their code before publishing a PR. Having worked on the opposite end (at a team where we mostly tested locally and had almost no automated tests), I much prefer this environment.

My last job was working at a company where you could not bring up the app locally. It didn't start that way. When the company was smaller I could develop a whole feature from the backend to the frontend and see the whole flow. As we grew it became increasingly difficult to run the app locally. Then it became impossible. Bugs grew and development time increased honstly 4x to do the same feature I could have done 3 years ago. I now think a good clean developer experience is key to a stable app.

I'm at a company that works like this, and it's awful. There are also anti-patterns around git branching and container deployment. The staging environment is broken every other day and production deployment requires the full time effort of a very senior engineer who cherry-picks through git commits from one branch to the production branch (there are thousands of differences between these branches). I'm honestly done with them. They don't even know what sensible dev/test/deploy looks like.

I'm sure that paradoxically your company must have sky high hiring standards, with interviewers particularly grilling candidates about best practices?

This is not even sarcasm.

I saw a project get themselves in that position by using a lot of AWS services. At some point it seems they had a dev environment but it quickly faded.

Their solution was to have an anonymized copy of prod to a few (like 3 for 20 engineers) environment where features could be tested. Engineers usually paired in order use them concurrently for unrelated features and they were very expensive.

I did not live enough to see the on-demand environments arrive - devops was always busy with operations tickets to continue building those.

A lot of it depends on the complexity and the technology stack.

Microservices based architecture with massive amounts of microservices and other service dependencies, its a pain to do more than local unit testing and mocks.

Are you using FaaS or a lot of other cloud features? Also difficult to fully test locally. Extract as much logic out of the cloud dependant code.

Often there will be sub-sections of large apps that can be run locally.

And if the only way to test is in production hopefully they practice some form of limited rollout so you can control who adopts the changes or similarly some form of optional feature flags that are used to enable the new behavior until it can become a new default. If their release model is "We are so good it goes live for everyone all at once" then that tends to make for a stressful release days.

> Microservices based architecture with massive amounts of microservices and other service dependencies, its a pain to do more than local unit testing and mocks.

For microservices if you use something like Consul for service discovery within those microservices then you can hit them with, eg, Postman and change your local Consul registrations so everything goes to a main environment except the handful of services you're working on - which you redirect to your local machine for debugging.

Extremely common. For some reason it is cool to put stuff in lambda or azure functions or whatever. Or to use 11 different stacks.

It is the case in my company. Production and staging is bunch of docker-composes scattered around several servers. We had to create another staging environment, I think it took several days to configure it. People run services locally, but not entire stack, that takes too much effort.

Right now I’m trying to understand and deploy Kubernetes. Scalability and availability is good, but one of the reasons that I want to make it possible to roll out entire stack locally.

Basically it’s lack of devops. We don’t have separate role and developers do the minimal job of keeping production running well enough, they got other tasks to do. There should be someone working on improving those things full time.

This: "Basically it's lack of devops". Companies think they can avoid hirig any infra/devops people and make all of that stuff the job of junior developers. They should contract someone it to set things up, at the very least.

You can find developer first roles where the company puts a lot of time and energy into making developers the most efficient workers, but most companies are a cluster-fuck of resource management and you'll be lucky to find a healthy company that also puts developers first in this way.

In my experience, recreating staging as "production junior with caveats" comes down to cost and living with bad architecture/design. I'm not defending the practice, but I do think it's incredibly common.

At Google, it is very team-specific but it is generally not possible to run servers locally for testing and there generally is no staging environment, but it might be possible to bring up some part of the stack on Google's clusters. Those severs may read production data or some test data. Other than that there are static checks, unit tests, code reviews and the release process to ensure reliability.

How do you develop features if you can't run the server locally? Do you just write code, get it approved in a code review and push it to production?

Automated tests where you're just running a tiny subset of the code you're working on

Chipping in as another Googler - agree that it is team-specific, but you generally can bring up development environments (in terms of what traffic is served, closed to the public, etc.), but these aren't actually run locally.

Production is not the only alternative to local. Companies can set up servers, either in the cloud or on premise, specifically for development and/or staging purposes.

This is very useful if the production environment is hard to replicate on each developer's local machine. For example, a team where some people prefer the MacBook Air and others prefer beefy Windows desktops might set up a cluster of Linux servers for shared development and testing. A common alternative is to tell everyone to run a bunch of VMs/containers locally, but this isn't always feasible depending on the size and complexity of the production cluster.

I've been at a place where the whole stack couldn't be run locally at all but everything is separated by well-defined API boundaries and there are mature mock servers where required.

The architecture was probably more complicated than it needed to be but it was also decades old.

Yeah, once you have more than a small handful of dependencies (SaaS, databases, microservices). Some of the stuff I worked on will have built in stubbing/mocking so you can run parts of the stack locally and the test suite also uses the mocks/stubs.

In my experience, good local development environments are relatively rare. We had them at a couple of small <10 person B2B, SaaS startups I was involved with. In larger companies, there were just too many pieces to run everything locally. Or in companies with software development, but not focused on software, local environments seemed more of an afterthought. With increased dependencies on cloud services, "serverless" environments, etc. it can also be painful to run stuff locally. Obviously you can build around this (and should...) but it requires thinking ahead a bit, not just uploading your lambdas to AWS and editing them there...

Yaknow I actually transitioned from doing dev stuff to ops at a smallish startup. I was brought in to help with a green field rewrite. Eventually it became clear just how much of a mess the deployment environments were so I cleaned everything up and parameterized things so that when we went to stand up the new production environment it was quick and painless.

I got hired on at megacorp to do pretty much that with a suite of green field apps. And so I did, across a few apps and a zillion different environments because all of a sudden spinning up a new integrated environment was easy. The devs ran a lot of stuff locally but still had cloud based playgrounds to use before they hit the staging environments. Perhaps the best part though was having the CI stuff hooked into appropriate integrations, so even if you couldn't run it locally you sure as shit tested it before it hit staging. SOC2 compliance was painful for the company but not so much for us.

If you're sensing a theme with "green field" you'd be on to something. Even in a largeish company it's not so much the complexity as it is the legacy stuff that'll get you. Some legacy stuff will just never be flexible (e.g. the 32-bit Windows crap we begged Amazon to let us keep running) and even the most flexible software will have enough inertia to make change painful.

Tangentially this is also why I get apprehensive when I see stuff like that linux-only redis competitor.

Yea, it’s common that one doesn’t exist. I work as a test engineer and am usually the one to build and support those environments, I’ve yet to find a company that has it all working well before I join. I actually made a killing building integration testing environments of blockchains and sticking them in CI too.

Can you elaborate on the blockchain testing environments/CIs thing please?

Sure! So when testing you want a deterministic environment - starting from genesis, couple accounts, the ability to build or wipe state, etc… sometimes projects use multiple chains, so maybe Ethereum and Polkadot along with a bridge between them. Building this out in Docker, managing it in CI, and building a test framework to allow devs to easily complete actions like sending a tx or waiting for an event to occur in a block has been my bread and butter for three years (and taken me from a median salary to a top 1% rate). I’ve recently moved to another industry but it was good fun.

Wow, that's fantastic.

So basically: unit test fixture setup spanning multiple external services ("Chains?") in a scriptable build, with various common/comprehensive/programmable interaction flows, and then integrating with CI? I'm making some guesses here as I'm not very familiar with the space.

I think it's amazing that you have (had) made this into a career. I am a dev and spent a couple years as a QA developer. Do you think this is a space I could break into? (Decade plus of development experience plus the good fortune of growing up with a mother who was a software developer. I learned to code (badly) when I was 13. I'm more than double that age now :)

If so, I'm wondering if you'd be up for chatting some more like on IM (Google Chat or Whatsapp or whatever) to elaborate further if you think this is still a viable proposition for someone new. I'd be happy to pay you for your time if you think this could make me money. I know how to set up CI, I'm vaguely familiar with bitcoin (I used to own some), and I don't care for cryptocurrency but I'm happy to meet market needs with a smile to pay the bills and then some.

And here's another thought. I know very little of the space, again, but I've seen some headlines about smart contracts going wrong.

Do you think there's any space for low hanging fruit simple auditing of smart contracts? Maybe a linter?

EDIT: Here's what Ethereum suggests: https://consensys.github.io/smart-contract-best-practices/se...

In particular,

https://github.com/protofire/solhint/blob/master/docs/rules....

https://github.com/prettier-solidity/prettier-plugin-solidit...

Local dev is great until production gets complex enough that you lose parity. That'll happen even earlier now that we run x86 on the server but ARM locally.

At my company we don't support local dev nor have any sort of staging environment. We have development linux servers connected to prod services and dbs. There are no pragmas or build flags to perform things differently in "dev". Things are gated with feature flags.

It scared me at first but now I think it makes sense: staging is always a pain and doing it this way we avoid parity problems between staging and prod. Local development would be impossible for a system at our scale but I think even a staging setup would result in more defects—not fewer.

Local dev isn't the end, it is the beginning. With it it allows you to get further than you would otherwise - and most importantly, when (not if) you do discover a problem on a lower environment (or prod) it gives you a place to try and replicate it. You run everything locally so hopefully you can make a change to mirror the problem... Not a guarantee but at least you have a good starting point.

Also your local dev should try and mirror prod as much as possible.

Yep. This is one reason I've always ran linux on my dev machine anywhere that would let me - sure, there shouldn't be any difference between OSes, but otherwise it's one more place for differences between dev and prod to creep in.

Heck, in my previous $WORK, I saw differences from local (running Debian) to production running CentOS. Luckily one cluster of Jenkins build nodes were also running CentOS and those caught a subtle incompatibility between system libraries that would have blown up if we had merged and deployed to production. My local Debian testing env did not catch it.

As a Python developer, yes, there are differences - and those differences are sometimes a huge PITA.

The only reason I don't run Linux is due to other MS tech - Teams, Outlook, and primarily office products.

In my experience, yes.

Often it's due to complexity. Systems accrete stuff (features, subsystems, dependencies, whatever) over time due to changing business requirements. The more this happens, the harder it becomes to integrate them properly without breaking other parts of the system. And the system is running the business, often 24 hours a day, which makes migrations hard to do. So you end up of something thats basically too complex to run locally.

And particularly in regulated industries like banking you have the security teams locking everything down hard. Problems accessing systems and getting realistic test data, etc.

>Is this common at other companies that you cannot run the whole app locally to develop and test against?

It depends what you mean by the whole app. To reflect locally 100% of the conditions the big microservice app I am working on is very hard as I have to replicate locally multiple Kubernetes environments, Kibana, Data Dog, Elasticsearch, Consul and several databases and data stores.

What I can do, is to run one or more microservices locally, connected to remote development databases and speaking to other microservices and apps from some development Kubernetes environments.

I'm sad to say this even happens at FAANG. So many engineers seem to either not want to or not know how to optimize their own workflow as they're building it. It's really frustrating when you want to contribute something to their project and the development workflow for verifying the application still starts after making any changes is "upload the 5GB deployment bundle to AWS". Like come on, you can't seriously expect people to be productive that way.

> Is this common at other companies that you cannot run the whole app locally to develop and test against?

Extremely. We can't replicate our entire infrastructure in development - it's simply too complex and expensive. We use a combination of dev, integration, staging, and canary environments plus pilot environments to gauge quality before a full worldwide rollout.

lol. It sounds like they hire smart people. But smart people =/= good engineers. Good engineers keep things as simple as they can.

Simple is easy on a small product. Your simple sql setup doesn’t work so well when you have terabytes of data and need to be running tens of millions of jobs per day.

Why is it a big deal to run your app locally? You just give each developer access to a cloud account (either one account or one per developer) with wide enough guard rails.

Heck, when I am in a place with bad internet, I spin up a Cloud 9 (Linux) or Amazon Workspace account (Windows) and do everything remotely.

I’m not wasting my time trying to use LocalStack or SAM local.

It would be quicker for me to run things locally - in my team we all have access to an AWS account each but to test my code remotely I need to compile code, build my container, push my container to ECR then deploy the container to ECS - that last step takes minutes.

The micro service I was testing locally connected to remote services either via an https end point or via service discovery. It didn’t have to run in a container at all or be run remotely. If it communicated via messages, it simply read from or wrote to the queue using local IAM credentials.

If you’re developing locally to none public endpoints, use a VPN or develop within an IDE hosted on AWS behind a VPC.

Why does the app you’re testing run differently in a container than outside a container?

Standard disclaimer to expose my biases not to do the whole “appeal to authority thing”: I work in ProServe at AWS (cloud app development consultant). But I did the same thing when I was working in the real world. My specific job looks just like a senior enterprise dev + a shit ton of yaml, HCL, draw io diagrams and PowerPoint slides.

The other bias I have is that I work for the only company where I never have to worry about an AWS bill and I can spin up ridiculous remote EC2 based development environments when needed.

But I also didn’t have many constraints when I was the de facto “cloud architect” at a 60 person startup.

Unsurprising reasons for everything - security department would have been reluctant to allow development from inside AWS, and running in Docker was to ensure everyone (i.e. all developers, testers and TeamCity agents) were running on the same versions of runtimes etc (Docker was chosen as the easiest solution for this problem at the time).

Damn, I first got into AWS not just to avoid administering systems. But to also avoid system administrators.

That being said, how do you get on different versions if everyone uses the same package manager config file (requirements.txt/package.json/the wonderful system that Go uses/whatever Nuget/C# uses).

I’m also spoiled because I’ve never been in a position where I wasn’t the developer and leading the “DevOps” initiatives since working with the cloud - not bragging, I only opened the AWS console four years ago for the first time.

HN/Reddit is a good place to find out other peoples experience.

It's more to make sure everyone is using Java 11.x, Java 17.x, Python 3.x, Node x etc - we're on Macs but TeamCity agents are Linux and Docker is easy way to pin the runtime on both operating systems.

Docker also fixes the "I forgot to brew install a specific version" in this case.

> deploy the container to ECS - that last step takes minutes

Either you wind up an instance for each container, it takes too long to start the app with less resources or the image size is in dozens of GB, everything could be avoided.

Reply on news.ycombinator.com