You don’t need to be “enterprise-ready” or “scalable”

> Is this common at other companies that you cannot run the whole app locally to develop and test against?

Yes. I work in B2B banking software, so having an "authentic" environment to test against is extraordinarily complex. Even the big vendors in the space seem unable to maintain accurate facsimiles of production with their own decades-old core software, so smaller vendors stand no chance unless they adjust their entire strategy.

What we ultimately had to do was negotiate a way to test in the production environment in such a way that all the activity could be tracked and backed out afterwards. A small group of important customer stakeholders were given access to this pre production environment for purposes of validating things.

This is something that has never sat well with me. But, the way the business works the alternatives are usually worse (if you want to survive long-term, that is). We wasted 18 months waiting for a customer to have a test environment installed before we learned our lesson about how bad those environments are in reality.

I work at a company that has similar practices to what you describe (we don’t run our services locally). While it’s hard to get used to at first, I think the silver lining is that it forces everyone to write good unit tests, since it’s the only efficient way to run their code before publishing a PR. Having worked on the opposite end (at a team where we mostly tested locally and had almost no automated tests), I much prefer this environment.

Why is it a big deal to run your app locally? You just give each developer access to a cloud account (either one account or one per developer) with wide enough guard rails.

Heck, when I am in a place with bad internet, I spin up a Cloud 9 (Linux) or Amazon Workspace account (Windows) and do everything remotely.

I’m not wasting my time trying to use LocalStack or SAM local.

Yeah, once you have more than a small handful of dependencies (SaaS, databases, microservices). Some of the stuff I worked on will have built in stubbing/mocking so you can run parts of the stack locally and the test suite also uses the mocks/stubs.

I'm in that boat. We have a "microservice" code base, and spinning up the whole "app" is a Herculean task.

Even if you do manage to get it all running, getting the multitude of database with useable data adds an odyssey to your day.

We have some parts automated, but ops cannot keep up with all the changes in the micro apps.

Back when I worked on a mono-rails app with a React front end, getting things up and running was as simple as running the JS server, rake db seed and then rails s. We used puppeteer to test FE, and rspec for the BE. Two test suites, but we knew if something was wrong.

That same place sometimes had a net outage, and I wouldn't even notice because I was running everything locally.

Now I can only get everything running locally with confidence, if I block out an hour or so.

Honestly... microservices are great in theory, but make dev a huge pain. Docker does not mitigate the pain.

I work at a VOIP services provider and we operate multiple brands which are companies we’ve acquired over several years. One thing that has rang true of every company we’ve acquired is that there is typically barely even a way to run one piece locally, let alone the whole thing.

Some of these when I took over there wasn’t even version control, and they were being developed by FTPing files to production server or even editing them live with vim or something. Certainly no reproducible containerization or even VMs to speak of.

I agree completely with you about having a solid development environment. I’ve spent the better part of a year creating such a thing for some of the brands, and it has increased our time to ship features and fixes probably ten fold.

In my experience, good local development environments are relatively rare. We had them at a couple of small <10 person B2B, SaaS startups I was involved with. In larger companies, there were just too many pieces to run everything locally. Or in companies with software development, but not focused on software, local environments seemed more of an afterthought. With increased dependencies on cloud services, "serverless" environments, etc. it can also be painful to run stuff locally. Obviously you can build around this (and should...) but it requires thinking ahead a bit, not just uploading your lambdas to AWS and editing them there...

Local dev is great until production gets complex enough that you lose parity. That'll happen even earlier now that we run x86 on the server but ARM locally.

At my company we don't support local dev nor have any sort of staging environment. We have development linux servers connected to prod services and dbs. There are no pragmas or build flags to perform things differently in "dev". Things are gated with feature flags.

It scared me at first but now I think it makes sense: staging is always a pain and doing it this way we avoid parity problems between staging and prod. Local development would be impossible for a system at our scale but I think even a staging setup would result in more defects—not fewer.

Extremely common. For some reason it is cool to put stuff in lambda or azure functions or whatever. Or to use 11 different stacks.

I saw a project get themselves in that position by using a lot of AWS services. At some point it seems they had a dev environment but it quickly faded.

Their solution was to have an anonymized copy of prod to a few (like 3 for 20 engineers) environment where features could be tested. Engineers usually paired in order use them concurrently for unrelated features and they were very expensive.

I did not live enough to see the on-demand environments arrive - devops was always busy with operations tickets to continue building those.

You can find developer first roles where the company puts a lot of time and energy into making developers the most efficient workers, but most companies are a cluster-fuck of resource management and you'll be lucky to find a healthy company that also puts developers first in this way.

In my experience, recreating staging as "production junior with caveats" comes down to cost and living with bad architecture/design. I'm not defending the practice, but I do think it's incredibly common.

Quite common.

Usually someone develops a remote or mixed local/remote development environment which is easier to use than configuring the full stack to run locally, and then the ability to ever do so again atrophies until nobody can figure out how anymore.

In some companies I've worked at this is fine and not a major setback. In others, it's been a mess. I don't think that "can run the stack locally" is a good litmus test for developer experience - the overall development experience is a lot more nuanced than that. "How long does it take to get going again if you replace your computer" and "what's the turnaround time to alter a single line of code" are usually better questions.

At Google, it is very team-specific but it is generally not possible to run servers locally for testing and there generally is no staging environment, but it might be possible to bring up some part of the stack on Google's clusters. Those severs may read production data or some test data. Other than that there are static checks, unit tests, code reviews and the release process to ensure reliability.

lol. It sounds like they hire smart people. But smart people =/= good engineers. Good engineers keep things as simple as they can.

Yea, it’s common that one doesn’t exist. I work as a test engineer and am usually the one to build and support those environments, I’ve yet to find a company that has it all working well before I join. I actually made a killing building integration testing environments of blockchains and sticking them in CI too.

I've been at a place where the whole stack couldn't be run locally at all but everything is separated by well-defined API boundaries and there are mature mock servers where required.

The architecture was probably more complicated than it needed to be but it was also decades old.

A lot of it depends on the complexity and the technology stack.

Microservices based architecture with massive amounts of microservices and other service dependencies, its a pain to do more than local unit testing and mocks.

Are you using FaaS or a lot of other cloud features? Also difficult to fully test locally. Extract as much logic out of the cloud dependant code.

Often there will be sub-sections of large apps that can be run locally.

And if the only way to test is in production hopefully they practice some form of limited rollout so you can control who adopts the changes or similarly some form of optional feature flags that are used to enable the new behavior until it can become a new default. If their release model is "We are so good it goes live for everyone all at once" then that tends to make for a stressful release days.

>Is this common at other companies that you cannot run the whole app locally to develop and test against?

It depends what you mean by the whole app. To reflect locally 100% of the conditions the big microservice app I am working on is very hard as I have to replicate locally multiple Kubernetes environments, Kibana, Data Dog, Elasticsearch, Consul and several databases and data stores.

What I can do, is to run one or more microservices locally, connected to remote development databases and speaking to other microservices and apps from some development Kubernetes environments.

> Is this common at other companies that you cannot run the whole app locally to develop and test against?

Extremely. We can't replicate our entire infrastructure in development - it's simply too complex and expensive. We use a combination of dev, integration, staging, and canary environments plus pilot environments to gauge quality before a full worldwide rollout.

I'm sad to say this even happens at FAANG. So many engineers seem to either not want to or not know how to optimize their own workflow as they're building it. It's really frustrating when you want to contribute something to their project and the development workflow for verifying the application still starts after making any changes is "upload the 5GB deployment bundle to AWS". Like come on, you can't seriously expect people to be productive that way.

It is the case in my company. Production and staging is bunch of docker-composes scattered around several servers. We had to create another staging environment, I think it took several days to configure it. People run services locally, but not entire stack, that takes too much effort.

Right now I’m trying to understand and deploy Kubernetes. Scalability and availability is good, but one of the reasons that I want to make it possible to roll out entire stack locally.

Basically it’s lack of devops. We don’t have separate role and developers do the minimal job of keeping production running well enough, they got other tasks to do. There should be someone working on improving those things full time.

I'm at a company that works like this, and it's awful. There are also anti-patterns around git branching and container deployment. The staging environment is broken every other day and production deployment requires the full time effort of a very senior engineer who cherry-picks through git commits from one branch to the production branch (there are thousands of differences between these branches). I'm honestly done with them. They don't even know what sensible dev/test/deploy looks like.

In my experience, yes.

Often it's due to complexity. Systems accrete stuff (features, subsystems, dependencies, whatever) over time due to changing business requirements. The more this happens, the harder it becomes to integrate them properly without breaking other parts of the system. And the system is running the business, often 24 hours a day, which makes migrations hard to do. So you end up of something thats basically too complex to run locally.

And particularly in regulated industries like banking you have the security teams locking everything down hard. Problems accessing systems and getting realistic test data, etc.

Although each of those companies is made up of 5 to 50 mini-companies, each with more business units/products/services. You have to understand which of them you're selling to, what they need, how they will use your product. Are two completely different tiny departments gonna hem and haw over trying your product, or will the whole org use your software? You might end up supporting a dozen users or 500,000. Will you still have to support the same features, or can the tiny teams get away with less? Is it even worth supporting the enterprise features if your product fit is smaller teams and you wouldn't get a large-scale contract anyway?

>The buyer is typically a VP, a director, possibly C suite, depending on what you’re selling

If a company has VPs buying things surely they have some sort of internal vendor management that usually involves accounting/legal/compliance reviews. Being enterprise ready means you have the roles/documentation/process/procedure to answer those questions

If you're selling enterprise software then the important number is not "number of employees" but "number of employees who would use your software". In large retail, manufacturing, oil etc companies they have a lot of software of course, but each person is typically accessing 4 or 5 apps at most (email, office suite, HR, etc). In FAANG companies each employee involved in software development is often using closer to 30 apps (project management, comms, source control, bug tracking, telemetry, server management, i18n, etc). It makes sense to focus on FAANG companies if you're producing enterprise software simply because they buy more of it. That obviously doesn't mean there isn't a huge market outside of that as well.

And good luck selling enterprise software to Apple, Google and Facebook, they all already have some custom internal softwares for everything

I agree. The last three companies I worked for were large health care companies. They spend millions on SaaS products.

right like any software deployed by Zebra is probably going to get used by every retail company and warehouse in the USA.

Now I don't know how much of the software on my TC26 actually comes from Zebra but, pretty sure, it's not zero.

ASSA ABLOY software is probably everywhere there's a door these days.

I don't know who the players in HVAC are but considering that a lot of HVAC/lighting control for like Target/Walmart are remotely controlled there's definitely somebody with tens of thousands of deployments.

Not to mention EdTech shit like BlackBoard, that gets deployed to every god damned school in the country

This completely tangential, but any time I see someone talking about normalizing a database, I briefly think they are talking about doing something like:

x/|x|

Ya know, like normalizing a vector. Of course this brief confusion doesn't matter, I sort it out quickly, and it is totally clear language to other database people -- it is just a funny quirk of overloaded lingo.

But I wonder if as big data and machine learning folks (since there's some overlap with Linear Algebra there) ever get their lingo mixed up.

"To predict where the ridesharing customers will want to go, we will apply the taxicab norm to our database."

Why would a team decide to use nosql to manage "a few hundred" entities?

Nosql databases excel at simple lookups and gets by ID, rather than trying to sort and join pretty much ad hoc.

That's not a scale problem. That's a problem of database choice. Cassandra is not good at that sort of query pattern at any scale.

You are absolutely right. 95% of problems in software development are created by the software developers themselves. The fastest/smartest developers I have worked with simply make less dumb mistakes. They keep it simple. That’s all they have to do to be 10x more productive.

It's true. These types of security certifications like SOC2 are quickly becoming a minimum requirement.

I am on calls all week with our banking clients about getting SAML/SSO squared away once and for all.

We are having a lot more nervous takes around legacy LDAP/AD auth mechanisms these days.

I agree with article because you don't need to have it to "start talking" with Enterprise customers.

You probably have to prove that you have these on your roadmap and that you have engineering team that can implement these in not so distant future. That you know these things exist and have will to invest in it in future.

You actually need OAuth / SAML if you give 2 shits about your customers.

Yep, I'm working at a Series B startup and I can tell you, you absolutely need SOC 2 or ISO and/or GDPR compliance based on your market. It's not optional, you will lose business if you don't.

Yes, but.. I am a software architect. ~90% of the job offers I get are from CEOs/CTOs who want me to join their small company to help them refactor because they can’t grow anymore. Tech debt kills as well.

>With that said, you need to be prepared to scale up quickly once you have this problem.

See I interpret that to mean that you should design for scale, but not implement or invest in scale until needed.

Facebook is a great example. They did tons of things not traditionally known as highly scalable. Like one repo with the project coded in php which was largely a monolith.

Is one of the problems though is that the early audiences appreciate this kind of over-engineering? If your investors/buy-in folks are being shown early demos and you mention "and this thing is gonna scale seamlessly up to 1,000,000 users because of tech words A, B, and C" then they get excited. They don't understand A, B, or C. But what they just heard is that you're planning for Big. And that stokes their aspirations.

So how do you square they circle "Don't design to scale but scale up quickly when you need to"? You can't rewrite your stack just because you took on a new customer that is suddenly killing you.

I don't think that it's stupidly hard to get basic things set up for scaling at the start. These don't apply to everyone but:

- use a cloud provider, they let you grow

- write infrastructure as code (terraform, pulumi, cdk), it's not that hard and it makes building your 2nd, 3rd data center easier

- for the love of God think for 10 minutes about a scalable naming convention for data centers, infrastructure components, subdomains, IP ranges etc. This type of stuff causes amazing headaches if you get it wrong, and all you have to do is consider it at the start. Many AWS resources can't be renamed non-destructively, and are globally namespaced. Be aware of this. Don't paint yourself into a really stupid corner.

- even if you're not doing microservices, consider containers and an orchestrator that has scalability baked in (like Kubernetes). If you use a managed Kubernetes with managed node groups or fargate, you can basically forget about compute infrastructure from this point on

- deploy Prometheus, Grafana, Loki etc day one and build basic dashboards. With Kubernetes, you can get this installed quickly and within a day you'll at least be able to graph request/error count and parse logs.

- deploy an environment for developers to deploy to and use the product (even load test it)

The development cycle is now set up for building what you need to build. You can easily find developers at any level who have experience with containers, Kubernetes, Grafana so they can onboard quickly, be productive and troubleshoot their own stuff. They can refactor the monolith without having to invent service discovery, load balancing, ingress etc.

My personal experience with Kubernetes began at a start-up with a tiny team, none of whom had prior experience with it. We had started building a single monolithic app. But within a short time we figured out how to make a helm chart, add ingress and boom, we were "online". Every time someone was about to spend time and effort inventing something, we would quickly scan through the Kubernetes docs and CKAD material to see if we could avoid having to build it at all. We leverage cloud stuff too, where it was cheap and easy and scalable (like AWS SQS/SES/SNS/CodeArtifact/ECR/Parameter Store)

Exactly this. It sounds easy but this type of creator who can do this is very rare but usually has a high success rate. This should be the defintion of the "10X dev"

I know, this is overly-general advice to the point of silliness.

It's a little like telling a construction worker "You don't need a super-powerful 'jackhammer' to do your work. A simple claw hammer is all you actually need."

Well... sure, that's definitely true if you're building a house.

But you might have a problem ripping up a sidewalk...

There are plenty of companies building houses, and plenty who are working with concrete! Neither one of them are "the big kids."

They're not even necessarily different-size markets, either -- you can make a lot of money building houses or making sidewalks -- it's just that different problems require different solutions, and different software companies are going to have different challenges to the problem of "get bigger."

Sounds interesting. What's the DB? What framework/type of API are you exposing on the C++ size?

This is absolutely it.

Great use-case for GraphQL, unironically.

I am suprised how niche these newsql databases are. Is it because they are expensive? why wouldn't everyone use those.

I think enterprise ready here refers more to scale than to features. People try and go the microservices, distributed DBs, hyper scale route way too early rather than focussing on their product, possibly building it as a monolith with simpler SQL DBs before going all out.

I think the opposite. Prematurely building things you don't need costs $$$$, and in a tight funding environment you don't want to burn cash you don't need to.

Also compared to the 00s, the 2010s were a decade of gold plating and planning for scales you don't need.

The advice "you don't need scalable architecture" comes from the same place as "performance isn't important, code is all about readability and maintainability"

It's just insecurity and pure coping. They're not dealing with complex performance bottlenecks, so they say these efforts are wasted.