A Tale of Two Onboardings | Hacker News Reader

A Tale of Two Onboardings

convox.com

54 pointsjoeyespo47 comments

Excited to get the app running, and following the brief instructions of his coworkers, Joe types:

docker-compose up

His rails bootcamp didn't teach them anything about docker. It turns out docker-compose isn't installed. Not wanting to sound like an idiot, he googles around, and spends 3 hours trying to understand what a "container" is. Many tutorials online mention using boot2docker, but some new people seem to be using docker-machine. He asks Susan which one to use. She sighs, and mentions that the team is split between them.

Eventually he chooses docker-machine, and boots the app up. Now it's unclear where the app is running. Eventually he figures out that he has to run "docker-machine ip default" every time the docker-machine VM starts, so that he knows the IP to visit the development site at something obscure like http://192.168.99.100:5000/.

Finally, he makes a change to the code, and refreshes the app page. Nothing happens, because it turns out that his local filesystem is not synced with docker, which is linked through to docker-machine, which is running inside a VM on a virtualbox host.

Don't get me wrong, containers are great. But this sounds like more of an argument for good "getting started" documentation, and helpful coworkers, rather than proof that containers are any more simple to get up and running than your average rails app.

> Nothing happens, because it turns out that his local filesystem is not synced with docker, which is linked through to docker-machine, which is running inside a VM on a virtualbox host.

Not to mention GUI/UID issues when mounting an HFS volume.

It is not Docker vs Rails or Vagrant or some other technology, it is having that technology really work.

From my point of view having a build process where somebody can be productive within an hour of sitting down is part of doing business and it is not hard to attain if it is a goal. That Ruby shop which is so FUBAR in that article is ahead of the curve of the industry as a whole -- it is not unusual for it to take a month or so for somebody to learn to get the build system to work reliably in some places.

But this sounds like more of an argument for good "getting started" documentation, and helpful coworkers, rather than proof that containers are any more simple to get up and running than your average rails app.

This bears repeating: in the second scenario, the new hire had somebody tell them what commands to run. Without that information, they would've been stuck Googling their way to success too.

Also, I can't help but be amused at how annoying that homebrew nonsense is--aptitude on Debian is just a hell of a lot simpler.

What's "non-sense" about it?

I completely disagree with the assumption that Docker will make life simpler here.

First, question is, what are you optimizing? Are you optimizing on-boarding for new engineers? How many new engineers do you get a month?

Let's say you get 2 new engineers a month. Getting the Docker up and running, setting it up so new engineers can use it easily is also cost, that also takes time.

I don't think that evens out until you really need to set up 10-12 new engineers every month.

This post also skips the part where boot-2-docker fails to load and you need to restart VirtualMachine and a lot of other problems with Docker that you will encounter LONG after the initial `bundle install` finished and everything is set up on a single machine.

While back we had the same problem and we used the same scripts we run in production only on Vagrant and we distribute that image

http://avi.io/blog/2014/12/26/ruby-and-rails-development-env...

Docker has a lot of advantages, however, I have yet to see a single company that stuck with the initial promise of a reproducible dev environment on Docker (Please correct me if I'm wrong here).

Bumping into the same situation I would select one of the following

1. Document with all the steps (Wiki) 2. Chef (same as runs in production) on top of Vagrant

I would live Docker completely out of this, it will introduce more complexity over time than the initial setup process.

Nothing will replace good docs and great onboarding culture in my book

We run a setup that is very similar to what you describe, but we use Packer to run the provisioning system (in our case Puppet).

So, it's:

1. Documentation with the setup steps (Readme file)

2. Packer uses Puppet to provision a Vagrant box and an identical EC2 AMI.

3. Run vagrant up to get started in development.

I would tend to optimize onboarding up to a point, mainly to try to impress new engineers. But also maybe because if something hard happens, and you need to bring an external consultant in emergency that could be useful. But mostly to impress new hires.

What a shame. I hoped this would have something insightful to say about how terrible it is to be a new joiner on a development team but ultimately it's just proposing a technological solution to a people problem...

What does Janelle do when she makes her first change for her first story and something doesn't work right?

Convox engineer here. I do agree this is ultimately a people problem.

However I think there is a large technology aspect to it also.

I often think about how Rails changed problems like this.

Before Rails we were all writing bespoke PHP websites. I remember it taking ages to get up to speed with code bases on a new job.

After Rails we shared a lot of common language and understanding. I have a good idea of how to run tests and where to put changes. I also have a universe of knowledge to tap into on Stack Overflow.

In a world with more shared conventions beyond Rails, Janelle has more than her direct team to help her with problems.

She has tools that help her pinpoint the problem. For example, docker-compose offers really understandable error messages around missing environment variables, vs an app silently breaking.

She also has a community that can also help her with questions. Just today I found an excellent Docker / Django quickstart guide that has hidden away a ton of minutia about getting a Django project running on my computer.

I guarantee Joe understands a lot more about his application than Janelle does.

I'm all for hiding complexity behind abstraction layers, but it's a really useful exercise for a developer to be given an app and told "figure out how to make it work, but feel free to ask any questions you have". It causes you to really think about how the application works, and the process of figuring it out yourself is FAR more instructive than any documentation would ever be.

Not to say you shouldn't have documentation (because of course you should), but when you have to figure something out yourself, it sticks in your memory a lot better.

Remind me never to work for you.

I trust my teams to manage themselves - and part of that is learning for yourself. Engineers learn through problem solving.

There is literally decades of research showing that it is possible to learn things faster and better with appropriate teaching materials. That is why we have Universities and schools to teach people - and don't just ask people to figure out everything for themselves (ie. high fidelity simulations were found to be less educational - since their high fidelity proved to be a distraction from building the important conceptual models.)

So, while I can and have figured out complex systems on my own, it would really be much more helpful to have someone or something to help create correct mental models at the beginning (and again, there are decades of educational research showing that there is a cognitive cost to building incorrect models of systems. That cognitive cost may mean nothing to my manager - but that is exactly why I wouldn't want to work for such a person.)

I'm not saying we teach someone theoretical physics by saying "dive in!" But you're talking about things that are relatively static.

I'm not going to spend a sprint building perfect mental models of an application because if your app is anything like the ones that I've built, they change significantly over the course of a year. We rip out services and and abstract them behind new API layers all the time. Legacy interfaces are provided of course, but the way things work internally is constantly a work in progress.

Now, just because it's a good idea for a developer to learn how the system works today when they first start doesn't mean it's a good idea to carry that complexity forward to your production servers. So if the day comes when someone containerizes these applications with an actual business benefit, then let's containerize the fuck out of them. But I'm not taking on a bunch of tech debt (that will incur more tech debt as the application changes over time) just to make onboarding easier - when making onboarding push-button isn't necessarily accomplishing the goal of onboarding because the developer is going to have to learn those lessons sooner or later.

I think as long as this is made clear, it's a great idea -- onboarding a new dev is a rare opportunity to get dedicated but fresh eyes on your internal processes.

No Docs, no onboarding. No one helping and showing the new hire around environment-wise. What does this tell me about corporate-culture?

Yes. Not to work anywhere near there. Tech does not fix people or broken culture.

Seriously. Is it so hard to maintain a document that outlines the steps needed to get a standard OSX or Ubuntu install to the point where you can hack on an app?

I have to do in my personal projects often enough that I maintain such a document purely for my own reference, because I forget steps even for my own projects.

More importantly, Why was the new dev given a laptop without the crucial pieces already installed. Janelle shouldn't have had to install docker and Joe shouldn't have had to install bundler or postgres or any of the other things.

Both companies had broken onboarding processes. Docker just happened to make the effects of the broken process less annoying for Janelle.

Personally, I feel like it's the app that's badly configured if it won't work with Postgres having been installed with either homebrew or the far-better PostgresApp. Backing services should just work with the default install on development. If it needs special configuration, like enabling Hstore on Postgres, it needs to be documented, but honestly I consider it a code smell. That stuff should be auto-configured by the app.

Everything else aside, it really feels like this article purposely stretches out the first example and compresses the second:

* Joe gets told to work on his own, and only briefly consults a co-worker once. Janelle sits down with a senior dev and works through it together. How much of Joe's work could have been solved by sitting down with a developer?

* Every single command Joe writes out is included in the article, while docker steps (potentially more than one) are summarised with a sentence, e.g. "she just needs Docker on her system."

* Every time Joe has a problem, he has to figure it out on his own, google it, etc. Janelle just gets things pointed out to her by Rebecca.

Seems like a pretty biased comparison, even if the point is valid.

I started working with a company a little over a month ago. They "kind of" containerized the different parts of their stack. They don't use the dockerized versions in production. I have been the only engineer actually trying to make this work. Its been a nightmare.

Getting setup: They have about 15 parts of the application stack. When trying to get everything up and running, each layer was anywhere from 15mb to 100mb and so we're talking several gb of Docker layers just to get the basic stack in place. Then the configurations were all over the place and nothing was quite documented right.

Then it turns out there is not any clearly defined, standard way to do local development out of Dockerized applications. So I end up having to run the parts of the site I'm working on locally, and then had to configure them extra to talk to the docker-hosted parts.

So now I'm working with this hodgepodge setup that mostly works, except when it doesn't. On 2 Fridays in the last month, oddly 2 weeks apart, my Docker environment has just eaten itself and I've been forced to rebuild it. The first time some space requirement got hit and the images got corrupted. Then last Friday virtualbox was having some freakout and started pegging at 100% cpu usage before again, everything got corrupted. Now its Monday and I'm trying to put this back together again a 3rd time.

I want this to work but its just been the most frustrating onboarding experience of any company I've worked at.

    Joe asks if there are any new developer docs.  “No, unfortunately,” says ....

This is the real problem. Arguing about mitigation strategies is less important than planning for onboarding.

This is exactly what I meant. Most companies don't care about providing good documentation, and so when new people join the company and don't know what's going on they blame the platform or the package manager or something else.

There aren't very many good alternatives to writing good documentation.

The straw man here is that you can use containers poorly and do automated builds without containers.

In the end, the first 1-2 days of stumbling are a rounding error on a programmers overall productivity and getting the environment set up is only a small piece of that.

The real take away is that you need to develop in an environment that's as similar to production as possible. Virtualization, containers or no, is great for that.

My personal experience... The first 1-2 days are a rounding error. But it's very common for all internal tooling to be undocumented and fragile. This is a cost a team will pay every day in developer efficiency.

This is a people and team problem.

A sign of a team that cares about this is when the time to first deploy for an engineer is fast and safe.

Definitely. I'm a huge proponent of easy builds. The six month ramp up time for new hires doesn't exist in 2015.

1. Use foreman

2. This doesn't cover the other problems you'll have down the line with the development process under docker.

3. If you're having that level of difficulty getting a rails app up-and-running you can't be expected to be productive on your first day anyway.

Convox team-member here...

Convox is a project by the creator of Foreman, David Dollar.

In fact the entire Convox team is ex-Heroku folks building a new suite of tools that make local development and deployment awesome.

I'm not convinced there is a solution to the problem. We will always struggle to bootstrap our environment, this is a symptom of trying to reproduce production environments on different targets. In the past I've been on a team that has tried:

* Invested a lot of work getting a set of services to run on our dev desktops (a 'onebox') and it was very painful. Lots of useless configuration, adding new services is hard and it doesn't scale well. Some external dependencies can't be run locally. * Using a 'integration test' stack that you could hit from your desktop, but it was stateful and shared by 10 developers -- eventually causing us to have to block each other. * We actually built a simulation to run our tests in, it simply replaced the networking layer of our external clients with a in-memory queue; it introduced more bugs because of implicit ordering and subtle differences from production.

Production is the only easy environment, because the configuration and code is immutable there. Testing and development are too nuanced, stateful and often require lots of mutation within individual binaries/services.

From all my experiences so far, just running a test stack of services was the easiest. SSH tunnels were important too, for things like service discovery.

This is a case for automation or documentation, not specifically docker. Could use anything including a shell script to get this env setup properly. Picking docker means they also have to understand how to manage developing within a docker lifecycle.

Even a wiki page would be sufficient as in developer envs, some understanding of what is being done and why is generally useful.

If you anticipate a architecture with many micro-services, sure docker might be a good win for you. But my own experience was a win-loss scenario so I'm now back to keeping things traditional (for now). Docker made it easier to start the environment with a single command but it failed in other areas and it was a pain to maintain.

What areas did it fail in?

Reading this makes me greatful that I develop a desktop app.

Here's how I set up my development environment on a new machine:

1) Install XCode

2) Check out source code

3) Double click project file

4) click "Run"

I'm really happy I don't need to set up half a dozen servers before I can get started. I always thought web dev was simpler than native development, but modern web development is extremely complex.

I'm not sure why others are unimpressed with this article -- I thought it had one simple point to make. And my project has precisely this problem: people spend lots of time getting their environments set up and it is not reproducable.

Obviously, my project needs to be fixed. This article points out that Docker is one way of doing that.

Part of being a Ruby dev is learning how to quickly onboard. In the same way that as a C++ dev you need to learn your way around the output of configure and make, and be able to quickly figure out what error messages are telling you, you need to have that same baked-in knowledge of 1) pull the repo, 2) get the right version of Ruby installed 3) run bundle 4) install missing dependencies as bundler complains about them 5) run rails server and install backing services / configure environment variables as needed. 6) run tests and get them all passing.

If you don't already have this knowledge than you're working with an incomplete set of tools and need to acquire them. The good thing is, if you're sharp, you'll pick up the general gist of things pretty fast. That's why I think nobody's yet written a guide encapsulating this knowledge.

Docker is not the answer to the problem, it adds complexity rather than takes it away. You still have to know the ordinary way things work in case Docker breaks. I guess someone does need to write that guide, given how many people have issues with it, and the hopefulness evident in their pleas to just move to Docker already.

The problems described in this article really do hit home. I offer that anybody who sees the Joe onboarding as anything other than typical may not have been in the field long enough.

These problems have been happening since the 1990's and very little if any progress has been made in these areas.

If it is that prevalent, I also would feel safe to surmise that it has little to do with the other detractors posted here (ie it's over-engineered, needs documentation, it makes Joe a better engineer, etc.).

Coders like to code productively first and foremost, not spend their time documenting and designing processes and procedures. I say don't fight their instincts. Let them do that!

This is where DevOps comes in to play, and should be the focal point of resolution for this and related issues; letting the coders do what they already do best.

I expected this to be more of a story about actual onboarding interactions instead of a product pitch, though I certainly understand the motivation for the latter.

In every company I've started at, I try to surf through process documentation, or if there isn't any 'newbie' docs, then I keep a little live-blog of my experiences, with the idea that I can share those later. The cost of such a blog is minimal, and it helps the new hire recall parts of the process they might quickly lose track of. Additionally, with some editing work, the blog serves the greater purpose of: 1) Informing the team about the process 2) Preparing the next new-hire.

This just sounds like a vastly overengineered product.

The important difference here is not which bits of technology are in use, but that YouGram has some shared standards and all the team members are using the same toolset.

Also, everyone has issues with documentation and one good way to limit the damage is to write build scripts or vagrant/chef build files or Ansible playbooks or Dockerfiles. The principles are the important thing.

As someone who has been onboarded with docker like in the example, I can testify that this article is absurd. Docker increases complexity, not decreases. So much so that I question its value in most circumstances, and I will never on-board someone by throwing a docker image at them.

While I might be missing the point of the article, and this may indeed be a people problem rather than one solvable with software, maybe a higher-level (one that is above rails, and above bundler, in terms of generic use) build tool could be of help here?

I could imagine a command like `make fresh` performing these commands, noticing that things were missing, and informing the new dev of which way to go?

Of course, documentation is the easier/possibly better solution to this problem, but It's a lot easier to standardize with less things to standardize on, and rather than trying to get every team to write good starting documentation, maybe making a mandate like "your repo must have support a make target called 'fresh' that will take a user from 0-setup to a fully setup environment, or print a useful error"

[EDIT] - Also, some thoughts:

-Is some of this pain actually good for Joe/Janelle? Could some of these issues be constrained as learning opportunities which might benefit them long-term (i.e. have a problem with bundler, find out about how bundler works)?

-It seems like many (good?) engineering organizations with lazy (the good kind) developers will have solved this problem -- whether it's from the new dev wanting to save future devs trouble, or from the older devs wanting to save themselves time with env questions.

-Docker is great, but I feel like getting the solution without experiencing the problem can hinder technical understanding. Docker compose is a breath of fresh air BECAUSE you have an easier way to boot, manage, and isolate related (in that they are all related to your app) processes. I would fear that varying levels of understanding would be allowed to just get by with `docker-compose up`

TL;DR: One company ships with a fully configured virtual machine. The other ships with a machine that needs to be set up.

The lesson seems to me that you should be provisioning your systems with an environment that's ready to handle the code.

And then there is Otto.

In more serious tone, what was the point of this post? I don't think developers struggle as much with bundler as they might struggle with weird codebases.

Convox engineer here.

This post dances around the point. There is lots of room for improvement in getting new developers up and running with a code base.

Convox and Otto share similar goals of one command to start and application and one command to deploy it.

The differ in that Convox is a thin AWS layer. Deploying to Convox configures EC2 Container Service, ELBs, and Kinesis Stream to run your dockerized application.

Is it fair to presume the docker person is going to be waiting a lot more for tests to run due to poor io performance?

That is not a fair presumption.

If you're on a Linux laptop a Dockerized app has effectively no performance decrease.

If you're on a Mac, a VM and Docker does impact raw IO. My anecdotal experience developing like this every day is that it does not cause noticeable waiting.

There are other downsides to Docker-based development, around tool complexity and stability. In my opinion this is no worse than tools like Homebrew or rbenv.

There are also productivity upsides.

Reply on news.ycombinator.com