Working remotely with 150+ people | Hacker News Reader

Working remotely with 150+ people

medium.com

130 pointsmdeira61 comments

After completing my CS master's degree, I was hired by my advisor as a full-time software developer. My job was to coordinate the efforts of the students contributing to his main project, keeping everything moving forward.

For that job, I found that keeping things asynch was crucial. But so was keeping things lock-free.

My basic premise was to never assume that any grad student would get any particular task done. So I tried to create an environment where any time they had a useful contribution, it could be merged into the main code base. But to never put one student at the mercy of another.

I think that worked pretty well.

That sounds very interesting. Could you go into more depth what techniques you exactly used?

Generalizing this idea. Interdependences are big source of problems and delays in projects. Many people don't realize it, but you have to actively work on eliminating them. You see it in the form people waiting for other people. Every team should be organized in a way that they can we do their work without waiting for other teams.

Yes this is one reason why with large software development programs it's generally best to organize around feature teams instead of component teams. In other words, a single agile team should be able to build an entire feature on their own through the full stack instead of having one team for the database, one team for the back end, one team for the service layer, one team for the UI, etc.

This is very interesting. Can you elaborate more on such organizational structure or know of anyone implementing in real life?

To me it totally makes sense. Instead of the inevitable interdependency that arises through the component model, I can see that if teams were building features on their own than it could be completed at their leisure (in the budget & time affordable) without the multi-directional multi-disciplinary communication shitshow.

I wonder if there is an official name for this feature teams model or more guide on how to implement it.

Scaled Agile Framework features and components: http://www.scaledagileframework.com/features-and-components/

"Scaling with Feature vs. Component Teams" by Ken Rubin: http://www.innolution.com/resources/presentations/svaln-2015...

This is gold. Thank you for posting that.

The more general term for these two structures are Functional vs Divisional structures. In the case of software engineering, a feature based team is a functional structure, and a "database team" (or "API team", or any other specific technical concern) is a divisional structure.

so why aren't more shops implementing a functional organization? It's still divided into front-end and back-end. What is the hindering factor and what tools would be a good fit for functional organization? AWS Lambda where people push their own backend along with the front-end thereby managing the full stack?

I have no idea why many engineering organizations insist on maintaining the artificial distinctions of "front end" and "back end". I think we do ourselves a disservice by labelling ourselves with those distinctions, but the entire industry seems to disagree with me.

Feature teams make a lot of sense, but one of the problems is that you tend to end up with two bosses, one team boss and one functional boss.

At GitLab we're trying to see if we can do component teams with handoffs (UX => front end => back end). This is hard and we might have to reconsider. I've detailed the reasons in https://about.gitlab.com/handbook/leadership/#no-matrix-orga...

In any large engineering organization with a bunch of products each of which are large enough to need multiple teams, multiple bosses are inevitable. The issues and technology are just too complex for any single boss to manage everything. A typical agile team member might be accountable to a resource manager for HR type stuff, a Scrum Master for team tasks, a Product Owner for team priorities and requirements, and multiple component stewards for component technical integrity. The key is to align everyone to a common set of program goals and clearly delineate responsibilities between those various "bosses" to prevent conflicts and avoid excessive communication overhead.

A "no-matrix" structure might work in a small organization which has basically only a single product. In that case it's actually possible for a single competent boss to understand the customer, product, employees, and technology well enough to make good decisions.

As far as I know a matrix organization normally means a two dimensional one, with two bosses. You're totally right that all those functions exist and influence your work, but I don't think it is normal to call them bosses.

In most matrix organizations regardless of dimensionality each employee still has only one formal manager who deals with raises, career development, and disciplinary actions. The other "bosses" such as project managers aren't officially considered managers in that sense, although the employee may be accountable to them for certain things.

I agree they have one formal boss but I thought that in practise they had one other person they reported to for their day to day work and results.

That's not a silver bullet though.

In a large software system you can't expect any individual to be as effective working in any area of the system. So you're winning on the blocking front but you're losing on the productivity front as people ramp in on different pieces and you're also potentially losing a sense of ownership from the developers and instead giving them a sense they're cogs in a machine...

To me a better compromise is to ensure you have enough of a backlog that you can always pick something that is the right combination of expertise/interest/lock-free at any given time. If you plan properly and you have sized your teams properly things should move along better than trying to get everyone to work everywhere on some random feature. That's not to say you can't occasionally do that if something important comes along but to me feature teams in large scale software is to be avoided. [EDIT: realizing this is maybe too strong of a statement, a team can definitely work across multiple pieces and will ideally implement complete features, but those features need to be a match to the capabilities of the team as opposed to just looking a "feature team" as some sort of unit of production for any random feature]

There's another part to this which is that it can be difficult to see the inter-dependencies between features. So modelling your software development as a bunch of teams that can just pull the next feature from the queue and work on it may actually not solve the locking problem at all but create more subtle dependencies and less efficiency.

If you're optimizing for productivity then you're focusing on the wrong metric. In order to deliver more customer value faster focus instead on driving down the cost of delay. For a detailed quantitative economic analysis of why this works please read "The Principles of Product Development Flow" by Donald G. Reinertsen. (Trust me it's one of the 10 best books on practical software management and will recalibrate your thinking in multiple areas.)

For large complex software systems every team member can't be an expert on everything. That's why within the overall program you set up separate small agile feature teams composed of team members who each bring some specialized skills and experience. Plus you should also layer on a matrix of component stewards who help maintain technical integrity for individual components. For a detailed explanation of how that works please see Ken Rubin's presentation "Scaling with Feature vs. Component Teams". http://www.innolution.com/resources/presentations/svaln-2015...

No one ever claimed that a feature team organization is a silver bullet. But if implemented right it will usually deliver at least a minor boost.

I'll check that book out, thanks. We'll see if it can re-calibrate me ;)

In my view optimizing productivity isn't necessarily mutually exclusive with other things you're trying to achieve. If productivity is very low then most other measures will tend to fall too though that may lag. The problem is that productivity tends to have poor visibility and is difficult to measure.

What I object to with all my heart is the "feature team" as the assembly line model of software "features". I.e. the software factory where features come in one end, get assigned to a random team, and product gets delivered on the other end. This model results in low quality, expensive, bloatware at best. It may or may not at any given time make some money. It's also not the model used, AFAIK, in most successful software companies.

I've worked on teams like that before, and my biggest gripe is that it is next to impossible to get a good code review. People on your team don't have the specialized knowledge necessary for effective reviews and the people on other teams who do aren't invested enough to do it thoroughly and in a timely manner and usually won't be familiar enough with the details of the feature you are working on to do it well.

The impacted component stewards are supposed to be in the reviewers list (along with members of your own team) for any significant change.

sounds a bit like functional programming and minimizing side effects

While i'm not 100% remote (i'm part of a small regional office that has a massive head office in another part of the country, but I can work from home). Something i've found incredibly important is jumping on video chat. There's something about seeing someone in a video while you talk to them that reminds you "hey i'm working with other humans, i should respond with a level of empathy i'd expect in return"

Are these individual or group chats? What kind of solution do you have?

I work in a small office of a big company and have a 10 hour timezone difference to my most important co-workers. We chat on the phone but I don't really enjoy it. I'm not sure video chat would help, but I might be willing to try.

But it would be slightly inconvenient because of the timezones. We might have 10-20 people from 5-10 locations around the world. This means that many participants call in from home before work in the morning or after work at night.

That said, I'm quite happy working remotely mostly from home.

I feel that travelling occasionally and meeting the people in person (and having lunch, dinner, beers together) makes it much easier to work together. There's no substitute for human contact.

If you're ok with the Google machine and spending way too much money on a computer that does basically nothing, there are these from Google[0] which have been great for me and my team.

We have a Hangout open all day in the office, displayed on a 60" TV. Anyone that's working remote can jump in whenever and it's somewhat like they're in the office. Audio is my only complaint. Sometimes it is difficult to be heard

[0]: https://chromebusinessdevices.withgoogle.com/products/1677/h...

That's interesting. Are you aware of other teams using video chat like this, ie. all day long?

Do you know what the Google monthly fees are? Is the bandwidth consumed a concern?

If they're using Hangouts I presume it's free. I've used Hangouts while partially-remote (very small scale) and we just used regular google accounts.

Yeah, hangouts itself is a completely free service. I am not familiar with too many others doing this.

I've heard a couple other stories like ours. One I remember specifically was a company with a 50/50 mix, they did ALL meetings from their own desks on a hangout so that everyone had the same presence, the remote people didn't feel left out, because everyone has to fight for the same one audio track.

As for bandwidth, it hasn't been anything that we've been troubled by. Been using it for nearly 2 years now. The client on the remote machines has some hiccups from time to time and is somewhat of a resource hog. But if you keep it in the background it isn't too bad.

They are both types of chats. We use a combination of skype for business and zoom (zoom supports external people far better than skype, and actually works better in all regards).

Our time zone difference isn't as extreme as a 10 hour difference. We have offices -1 and +1 from our current time zone so it makes it easier.

A talk I attended earlier this year had a similar setup to yours and what they did was record all meetings and had a culture of always doing the meeting regardless of wether people could join. Then if you missed the time you chimed in later after you watched the video. Decisions were handled in a similar async fashion.

Even for quick conversations, video chat is much better than a phone call because it has higher quality audio and you can see the visual cues of when it's your turn to talk.

There are plenty of easy browser-based video chat applications using WebRTC like https://appear.in/. You can send people URLs to your video room and there's no software to install.

I have pretty much moved over to appear.in 100% now, I find its latency is much better when working with remote teams (I am based in Australia and the stakeholders are in the USA (East and West coast) - Appear in is flawless, have had 8 people on a call and it just works, hangouts frequently craps out unless bandwidth is perfect!

Beyond that, I'd say the ease with which a video chat can be initiated is important so that the annoyance of starting one does not become an impediment to using them when they are appropriate. My team uses Sococo which, while not without its problems, does allow for one-click video chats and screen sharing with 1 or more co-workers. It's really nice to have the option to jump into a 5-minute video chat/screen share with the click of a button to work through something that would otherwise be a 25-minute chat conversation.

FWIW: The author is VP at GitLab.

Somewhat similar to: http://asyncmanifesto.org/

It's a nice list but could be a little more specific.

I have worked in purely remote teams and I found them to be highly functional. You need a certain type of self reliant people though and really strong leadership. Somebody has to make and have the power to enforce certain decisions that affect everybody.

Can you describe in more detail what really strong leadership looks like in a remote work environment?

For example someone needs to decide how issues are tracked and define a process. That process then has to be followed by everybody. There can't be exceptions.

In general I think remote teams benefit a lot from a central person who has final say and had the ego to push things through. Somebody like Linus for Linux. You can't have people like you see in a lot of companies that need endless meetings to make decisions.

This.

I worked in a few remote teams and most had bad leadership.

No process or it changed every month or it wasn't really used by anyone etc.

No single source of truth. Some people write in slack, some in quip, some per mail, some in trello, some in github...

I think people are trying to run remote teams like "local" ones and can't understand that they are different.

"People have to sit in one room to be more productive" - "we need water cooler talk to get the word around" - blablabla

They just want to be "managers" in title, but not do the work...

I strongly agree about requiring self-reliant people. I actually wrote about this shortly after posting this on medium [0], but the TL;DR is that you (as manager) should make it extremely clear what is expected of everyone and that people should be very self-sufficient / independent.

[0]: https://about.gitlab.com/2017/01/03/set-expectations/

They have to be independent but still flexible and not stubborn. In short you need high quality people.

Sounds expensive.

In my experience having any other type of people is generally more expensive because you are wasting higher level people on actively managing lower value tasks to get done.

Sure. But if you need higher quality people in a distributed team, that just means it is harder to get things done in such a team.

I've been working remotely for 2 years now and my team has people from all over the world. People mostly talk on IRC but not a lot (about important stuff) and there is a HUGE number of periodic meetings. I can't stand them. What's worse, every time things start to appear to be going off track, the initial reaction is to always set up a new periodic meeting about topic X... this is supposed to bring people together but useful interaction never happens on meetings (too many people trying to save face). It's horrible.

When talking to external partners, our management brags about having a remote team with people from all over the world, but I don't think they understand how to make this work.

Sometimes I feel we should all be in a single office because of this, but then I remember other companies have figured this out so I don't have to feel bad about wanting to work remotely.

Periodic meetings mean the end to remote work. Avoid them whenever possible.

I try to be proactive about cancelling the periodic meetings that we do have, whenever there's nothing to talk about.

Traditional meeting strategies also work well: never start a meeting without an agenda, set a short and fixed time slot for it and always end when the time is over.

My team has a daily 15-minute scrum meeting that I haven't found overly burdensome. We're spread over a 4-hour time difference, so the time of day isn't really an imposition on anyone. It's also no big deal if someone misses from time to time but I've found it generally helpful for keeping up with what the team is doing.

We focussed a lot on async collaboration and unfortunately we get issues of team associations. You just miss out on any water cooler talk and struggle to build relationships with individual employees. This impacted our employee turnover massively even though we gave competitive salaries, flexible work hours and work-from-home.

We do a few things to alleviate this, I'm not sure if they are enough for scale:

1. Daily call with everyone where we talk about our private lives for the most part (30m call, 28m private life talk: everyone has to tell something they did for fun last week)

2. Very active chat, with many non-work channels

3. A 'watercooler' hangout, where people join to just casually talk over video chat

You're describing everything I hate, and which made me love remote to get rid of it. And you even increase the amount of time lost in those useless and horrible things.

I have the feeling this water-cooler talk thing is mostly an issue if there is no focus or the management is bad.

If everything is fine, I don't talk much "privately" with people at work and some "more social" individuals talk about their private things, when "at the cooler".

If there are problems in management, we talk about these "at the cooler".

Water-cooler talk is a way to bond with others. Some of my greatest friends were made at work, and this talk happens when we want a break. Usually getting food, coffee, or going to work out. Water-cooler talk isn't only there to bad mouth management, but to make connections/friends.

Hm, okay. Maybe I'm special, in a bad way? I don't know.

I never really got along with people that I was "forced" to be together with. School, University, Work. I mean, people liked to work with me and I'm mostly really polite and inclusive and all. But I didn't find many people I considered interesting at these places.

I got all my friends from meetups, online communities, parties, etc.

That's perfectly fine. Some people prefer to keep work at work. It is good to do though if you want to climb up and understand the politics at your office (which sucks). I'm also just a super social person, and kind of an outlier.

haha, yes.

Every time I noticed such things going on at a company, I jumped ship.

I'm working for a small (<50 employees) company, and I'm pretty lucky to have a management that cares about this.

Stuff we've got:

- weekly all hands meetings where people talk about what makes them excited - lots of non-work-related chat rooms - 2-3 meetup opportunities per year - culture of quick ad-hoc synchronous meetings

We also had a board game night via Tabletop Simulator (http://berserk-games.com/tabletop-simulator/)

It also helps massively if the company is remote-first. We've got a central office where several employees are based but you never feel like you're missing out if you're based elsewhere.

That is pretty awesome. I've only worked remotely from a place where everybody is based in the same office, but I think your place addresses the big lack of social interactions that usually accompanies remote work.

On a side note, I'm also tangentially interested to see how VR can affect the remote workspace. I've used Bigscreen Beta (shared multiplayer lounge with virtual screens) with the Vive to work on a mutual side project with some friends and being able to see each other's screens by turning your head rather than by sharing with some program has also made the experience feel closer to in person work.

I'm curious about how much the need for social interaction through work differs between Europe and the US.

In Scandinavia I feel that there's a pretty distinct border between work and friends/social life. Socialising with colleagues is usually pretty superficial like afterwork and such.

Maybe this is the reason I find these "water-cooler moments" so vague and overrated. Personally I think I derive a good and rewarding working relationship by basically just working together, even if that would be mainly through text. I do however realise that there's a lot of people - in my current on-site job as well - that really avoids communicating through text if it's possible to talk face-to-face. I'd assume these persons would probably not enjoy working remotely.

I hope remote-only companies make sure that new hires actually can, and like to, communicate through text.

Any experience what single point of truth is useful with such a large group?

In our case, we make sure that single source of truth is either a GitLab issue, for anything related to implementation or the handbook on our website for organisational matters [0].

[0]: https://about.gitlab.com/handbook/

I keep hearing the word async come in these remote working threads, what does that mean exactly in this context?

What does this single source of truth look like? A google docs spec?

Async means reducing likelihood that your work will be blocked by someone else (e.g. waiting for response from someone) as it might cause big issues when someone is 10 hours away. If you can't get any work done because you are waiting for someone who is sleeping, that's really bad.

Single source of truth can be google doc, wiki, github issues/wiki, whatever. As long as it's single. The problem is when critical information is scattered in slack history, github issues and PRs, wiki and some odd google doc. It makes it hard to find information and you will end up finding contradictory information. Lack of single source of truth or not updating one will reduce the async property of the whole operation, among other things.

Synchronous methods of communication are phone calls, conference calls, in-person meetings, and anything else that requires that two or more people attend the same "event" at the same time.

Async is everything else: email, IM, leaving voicemail, blog posts, git comments, tickets, bug reports...

Synchronous communication happens when one participant says/sends something and then waits for the other participant to reply doing nothing else. Exactly the same as with synchronous function call: you order some operation and wait patiently for it to return the result.

Asynchronous communication is when one participant sends something and leaves things as that. For a reply, either another (asynchronous) message is sent, this time in the opposite order, or the sender checks if the reply arrived.

Real-time communication, like phone calls, video conferences, and various chats and communicators are synchronous by the very nature of the activity of conversation. Everything that you may postpone checking is asynchronous; e-mail, tickets/issues, and source code commits are like that.

Reply on news.ycombinator.com