Ask HN: Is it worth getting into distributed systems after 12 years full stack? | Hacker News Reader

Ask HN: Is it worth getting into distributed systems after 12 years full stack?

77 pointssidcool67 comments

I have done 12 years Full Stack engineering, including some DevOps. Recently I have been getting very interested in Distributed Systems, and not just at a consuming level but at the theory level. I would like to get into the gritty details of distributed systems, perhaps to the level of a Masters' Degree.

Is it worth getting into the the subject? It would take a major commitment in time and efforts. Would it help with better career opportunities?

View on HN

Yes, most certainly. Best way is to look at cloud technologies used today: AwS, Google Cloud, Azure. Cloudflare's tech blog is also very good.

One growth area is cloud gaming. And I think "virtual desktops" in general may finally gain some consumer traction and scale pretty soon.

Naturally this leads to an interest in Decentralized Applications as it represents an opportunity to re-build the Web.

Some resources to get started. Best of luck ;)

MIT 6.S974, Decentralized Applications, Fall 2018

http://nil.lcs.mit.edu/6.S974/papers.html

Web 3.0 Summit Playlist

https://www.youtube.com/playlist?list=PLxVihxZC42nHlc7J-UqeS...

Virtual desktops will open up opportunities of virtual phone operating systems. If you leave your phone at home by accident pick up a "shell" at the grocery store and sign into your account. When your done with your errands drop it off at the last place you were at, or some other concept. Truly portable number.

no one really wants this. This is not a game changer in purchasing any kind of device.

Remote desktop access might be a niche feature, but people do use it (VNC, RDP, X11, etc.) to deal with problems that would be seriously inconvenient to solve otherwise. The biggest barrier to the adoption of remote desktop access apps is probably the fact that the average user can't readily conceptualise what's going on.

Until no one actually owns phones and all you have are shared shells that can be picked up anywhere.

Unfortunately, as with the idea of batteries for electric cars that can be swapped out at the charging station, there is an incentive to hold onto better units, and quickly return inferior units to the pool. It's a case of Gresham's Law.

So the utopia of interchangeable shared units of anything will only be viable if those units are completely indistinguishable. If there's any kind of distribution of quality, the incentives will cause the economy of the system to devolve into hoarding of the better units. One way to work around this is to mandate regular exchange, but that really gets you into the territory of communism. Not necessarily a bad thing in theory or in some vastly different future, but a big change.

You would have to swap out stock anyway as all physical things degrade with use. It would be a natural culling of devices that would need to be pulled from circulation regardless.This idea doesn't exclude owning your own more powerful device that may have been left at home or a friend's house. You're not interested in the physical object and keeping it for long periods of time. It's a short-term convienince proposition.

I am not much into Decentralization as much as Distributed. Thanks for the links!

I have been working in distributed systems for 6 years or so. It is a field ripe with difficult problems and active areas of research. Some things people don’t even think of as distributed systems suffer from many of the problems inherent in that field.

For example, a system using a database with asynchronous replicas - especially if latency is not extremely low - using those replicas for read scaling, is an eventually consistent system (because of replication lag) and suffers from related issues.

Multi-master databases are even worse. You get the joys of network partitioning (netsplit or split-brain) and conflicting updates.

Sure, there are ways to work around this, but all the workarounds I know of have some rather ugly tradeoffs.

Basically, there are many opportunities to do something great, and not be bored. IMHO.

100 times this. The replicated DB with master-slave and the CAP theorem was something that got me interested in Distributed systems in the first place. Google Spanner, I feel, is a marvel of modern tech. Thanks!

Do you recommend a full time degree in DS or actual projects?

Are you specifically targeting better career opportunities?

Now is as good a time as ever to be a seasoned Full Stack engineer. Pursuing an MS and shifting specialties seems very expensive.

Despite all the support other commenters are giving, I believe you should be very hesitant about making such a career shift.

Fwiw, my experience has been that many of the job openings spurring the growth in advertising for Full Stack engineers are actually glorified Front End roles, with very little work at the interface or on the back end.

I suspect that, due to bias against Front End developers (and javascript :) ), hiring managers are have trouble justifying senior front end developer positions and so are resorting to overqualifying the position and unicorn-seeking.

There's also a bias in technical interview questions, since conventional algorithmic/puzzle questions hold less relevance on the front end, so qualifying experienced engineers without asking back-end appropriate questions poses difficulties. Which means that, despite a role being a Front End position, Full Stack candidates end up looking like a better fit, since they do better in the interviews.

growth in advertising for Full Stack engineers are actually glorified Front End roles

Right. People think I’m a hater because I say “full stack engineer” is just title-inflation for webdev, but I’m just telling it how the entire rest of the world sees it.

Throughout most of the history of the web, the same people wrote the CGI scripts, and any SQL in them, as wrote the HTML; working this way is perfectly normal for a webdev for like the last 20 years. I guess we were all “full stack engineers” back then, just we didn’t have a grandiose title.

IMHO, it's got more to do with the expansion and change of the Front End developer role, than direct title inflation per se. Now, a front end developer has to maintain consistent behaviour across a panoply of platforms - each under constant change and growth, whereas the scope of actual 'web development' has remained a relatively static. It's becoming less and less realistic to expect one person to be able to effectively handle both roles any more, without making compromises on UI behaviour and compatibility.

IMHO, front end specialization is not as trivial as we think. Building websites may be simple, but building a fast front end is still quite an engineering task.

I am targeting a specialized work which hopefully would pay well. For example, someone working on the Google Spanner DB. Full stack is better paying, but after some time it feels like doing the same thing over and over again.

What downsides do you foresee in making the career shift? (Apart from a risk to salary)

I think the concept of "distributed systems" is very broad.

Do you mean service oriented architectures? That's the practical 'distributed' systems many companies are using today. You don't need a degree- just go work at Amazon for a couple years.

Do you mean decentralized systems and algorithms? Paxos, Raft, Bitcoin and TOR? Then sure, you might consider a master's degree. Or just start reading papers online and messing around with code in your spare time.

"Is it worth getting into the the subject? It would take a major commitment in time and efforts. Would it help with better career opportunities?"

I guess its an opportunity cost. What else would you be doing besides that. with 12 years of experience Id hope that you have some understanding about the subject already and arent completely in the dark. I guess with 12 years I would say it should take you no time to pick it up cause I would expect you to already be well into APIs and Oauth.

I wouldnt hurt your career opportunities. But I guess you could do something more established like get a data science degree or work with embedded systems to work with robots. Its really about what you want. But doing this is better than doing nothing.

Technically I think that a lot of people do it for a cool factor rather than for a need. You can add uncessary complexity without benefit. I dont find there to be too much deep complex ideas about it.

Basically you are performing separation of concerns across a network vs doing it by creating different class files, this allows you to offset query load to an entire set of servers which helps scaling. But your company also has to pay of another set of servers.

I vote yes for distributed systems, no for the Masters' Degree. If you already know how to code, it's easy to get into other fields and train yourself, thereby not losing money. You can also write open source code if you don't think you could find the appropriate job instantly.

I was "Full Stack" before I did an MSc in Distributed Systems at UCL and I would disagree.

Yes, the MSc taught me a lot about Distributed Systems. Yes, it cost me all of the cash I had saved up (£15,000, no debt). Yes, I could have tried to go it alone... but had I done that there is a good chance I might not have learned enough to know that I do not know enough; how to explore the challenges of designing and working with Distributed Systems; how to understand and debate the tradeoffs of different systems; how to apply the theory to the pratice; how to read, analyze and even critize white papers; I could go on...

How much is it autodidact friendly compared to Full Stack engineering? What would a "hello world" or "first learning project" of distributed systems even look like? I can only think in terms of learning via building a concrete project eg. a to-do app built on whatever tools you want to learn.

Yes, there's no simple HelloWorld for Distributed systems. One must understand a great deal of theory before committing to a role.

Learn theory at a job then?

I have been inclined to pursue this, but I won't probably get a role in a company if I don't have enough experience with DS, and to get experience I need to be good at it, it's a classic chicken & egg situation.

What open source projects do you suggest to get started with?

Yes it’s extremely interesting but just be warned: the hottest distributed systems applications are at either extremely competitive fintech firms or massive household name software companies, because at the end of the day, those are the only groups that really need and can afford the state of the art. After you get a masters, if you want to continue working on the bleeding edge, those places are your options. So yes those are great career opportunities but they may not offer what you want culturally

I work on / with what I think is pretty cutting edge distributed software but outside of academia most of the work is, at the end of the day, still engineering.

Don't you need to know some of Distributed Systems to consider yourself Full Stack engineer?

No, only a JS framework (or two if a senior)

I think it depends on the scale of apps you build for. One can deploy a simple backend API or web app to a PaaS like Heroku or Zeit, or even a PaaS-like environment managed by another team, without knowing much about distributed. I would consider someone that can do that + build a modern front end as full stack.

IMO the average CRUD web app doesn't have the scale to have distributed systems problems. Also there are managed service versions of everything these days even when one scales up.

Yeah. And the scales have to get pretty crazy before "rails on postgres with some in-memory caching" stops cutting it. Your average developer will never see it.

I believe a basic understanding is enough to be a Full Stack engineer. For example, it would be enough to understand that my horizontally scaling backend would imply stateless servers. It may not be necessary to understand how the clocks are synced across my clusters.

I did my masters in Distributed systems and I think it was definitely waste of time. I would prefer data science or something similar instead. Aside from academic positions, there is not much opportunity in the market.

> Aside from academic positions, there is not much opportunity in the market.

May I ask what you mean? I didn't formally study distributed system, but it seems like much of my daily struggle as an engineer is trying to deal with folks who are trying to implement consistent solutions in an eventually consistent world. For me, distributed computing lies beneath every non-trivial programming challenge I'm working on. Even OPs front-end JS talking to the back-end is distributed.

I would recommend getting a MS degree in /systems/ engineering. From there, you can study the distributed world but also you can access the equally fascinating world of general systems engineering. To make it better, your computer background will be a plus compared to fresh grads.

Short answer: Yes, but it depends.

Yes: I believe everyone would benefit from leaving their comfort zone and venture into more DS to expose themselves to some non-trivial work. Don't get me wrong, there are plenty of challenges on writing a working CRUD system from the bottom up but facing scalability, consistency and fault tolerance issues is a different word, and most software developers don't give a damn about performance. That's usually what happens at startups or small companies(I've been there several times). A small and focused team builds a working product, and suddenly they need to bring more experienced engineers to help then scale it once the cooking recipes from the internet aren't serving them well or because the leadership won't let the team engage these challenges. So my first advice goes here if your company has such problems try to expose yourself to them.

Practice and theory should walk side-by-side, you can read the entire library on distributed systems but still be mediocre when facing real problems because you've never encountered those problems in practice. Use the books and papers to give you a direction to where you should point and do your homework. Consistently paced challenges and exercises gives you volume(consistency) and helps you build experience. Build your glossary of concepts, nail your difficulties and once you understand the basics, go read and implement CRDTs in Rust by yourself :) - I'd also suggest you to try following a book on your own pace first before committing to any formal education. Try to find a mentor within your friends who can give you more advice's and peer review your work.

Now the sad part: Unfortunately when shifting expertise even within the same career, you give up on several years of your experience. You will look less appealing to recruiters and companies because you just turned yourself into a JR. Distributed systems engineer w/o previous experience, and those are pretty hard to find out there. If you can live with that(both checking our ego and financially) and be humble enough to accept your new position - I'd recommend you to embrace it.

Being part of a team who is designing and running a globally distributed system has been the best 5 years of my 20 year tech career. Things go wrong sure, and it’s hard, but is crazy rewarding and intellectually challenging in all the right ways. I never quite got that when I was a full stack web dev.

I wrote a post on what a distributed system engineer is, at least from our perspective at Ably Realtime. See https://blog.ably.io/what-is-a-distributed-systems-engineer-... if you’re interested in what it’s like.

Matt, tech co-founder Ably, www.ably.io

Here is starter kit for you: https://github.com/theanalyst/awesome-distributed-systems

Read, understand, repeat.

Ah and regarding career: Do Masters' degree if you want guided learning and/or target an academic career.

But if you are self learner and want industry job with hands-on distributed systems, I would recommend exposing any back-end/dev-ops/networking experience in your résumé. Also obtain AWS and GCP certificates, they will require you to learn about both theoretical and practical aspects of distributed systems.

Seek companies working on large scale distributed software, you may start as full-stack or back-end engineer and then get closer to interesting projects.

I would say yes. What's the worst that can happen, you learning something good and interesting? It's always good to have people in this field that know how to design distributed systems.

It's worth it, but probably not for better career opportunities. As almost anything else for your career would have a bigger impact and would take less time and effort.

Yes, go for it, but definitely make sure you actually build something cool during your Masters degree if you want it to improve your career opportunities afterwards. The credential by itself won't mean much, but if you list some interesting project(s) that demonstrate your new skill it'll help with future job searches.

> Java, Scala, Ruby, Python, Angular.

You should also learn C. (Isn't that a pretty important part of the "stack"? ;-))

Thanks! I am learning Rust nowadays. It's promising to replace C/C++. I just don't know it enough to list it yet.

Most core system software is still written in C though, and I think it's not reasonable to assume we are going to rewrite more than 40 years of computer history in Rust in any reasonable amount of time.

I don't believe Full Stack is a real thing. I also believe you have probably been doing distributed systems even if you think you haven't.

What do you mean regarding the full stack comment?

I mean people who claim full stack really aren't. It's a misnomer.

Well thats because you are not a full stack engineer. If you were you would understand. Its pretty simple. You have backend , you have front end , you have database. Full stack enginner has experience and implements features across all these technologies. A bit of dev ops environment knowledge is probably also good because you have to worry about the connection between all these environments in order for you to do your work.

Sure there is a quality trade off, being jack of all trades but this industry is very elastic. You can have a great frontend engineer with only 2 years experience and an ok one with 6 years experience. Time in field doesnt really gauge skill well. Its more about the challenges one takes on and the variety of problems they solve with their code.

> Full stack enginner has experience and implements features across all these technologies.

I guess most of the 90s and early 00s web devs were full stack then, including setting up the servers ("dev ops" I guess).

Well you had a tightly knit package back then, I'd argue that increased complexity in the front end was what lead to the 'full stack' developer.

As the previous poster mentioned, it's quite simple, if you can implement a feature vertically (and I'm thinking at least a 3-tier system should be considered), you could consider yourself to be a full stack developer. I've met plenty of developers in my career who refused to work outside their tier.

The fact that sales people and recruiters are making full stacks up to be some sort of mythical horses with a single horn on their head is a different story...

Thanks everyone for the comments. I am going through each one and will reply. This means a lot.

Do what interests you and stop worrying about the consequences.

> Would it help with better career opportunities?

It depends on the demand.

How many senior web devs are there? How many senior web dev positions are there?

Guestimate: 278057 senior developers and 23050 open positions [1].

How many distributed systems people are there? How many DS positions are there?

Guestimate: 1355 distributed systems people [1] and 219 open positions [2].

In other words: try to find some data that give some intuition. Sure, the 'research' (if you want to call it that, lol) is really biased but it probably is better data than your own intuition if you have no clue about it.

------- addressing some other stuff you said -------

Why do a masters in it? Do you want to research it as well? Writing a thesis is the obligatory part of it. At the Vrije Universiteit Amsterdam there is a masters in it that is spearheaded by Andrew Tanenbaum. But when you look at the courses, you see that many courses aren't about the theoretical part of distributed systems (see [3]).

To get your feet wet I'd recommend to fully follow distributed systems from Maarten van Steen. Why? First of all, his book [5] is being recommended by other people (see [6]). He also has lecture videos on it (from 2012, but since it's from a theoretical perspective they are still relevant). Edit: he deleted his videos, but I found one source here [4]. Oh yea, and he offers it all for free (which is also why I'm comfortable posting the new URL of the DS videos).

These are my 2 cents. Goodluck!

My personal opinion, don't pursue a masters. Just scan study guides of universities and figure out a curriculum, work less as a web deveoper but keep doing it (i.e. 4 or even 3 days per week) and study it independently.

[1] I searched it via LinkedIn by typing "senior web developer" and "web developer" and "distributed systems"

[2] I typed in "distributed systems" as job title on indeed.com. LinkedIn gave nonsensical results since everyone's course Distributed Systems was counted and almost no one had a title in it.

[3] https://www.vu.nl/en/study-guide/2018-2019/master/m-r/parall...

[4] https://u.cs.biu.ac.il/~ariel/download/ds590/videos/

[5] https://www.distributed-systems.net/index.php/books/distribu...

[6] https://teachyourselfcs.com/

I vote yes, but I'm probably a bit biased.

What would your reasons be? I think we are all biased and I value strong opinions.

Not everything in the world needs to be decentralized, despite what your local snake oil ICO salesman might tell you. However, there are more than enough significant institutions and processes that can and will function better with a little bit of distribution, decentralization, federation, etc properly baked in at key friction points. Building that future is a worthy and potentially lucrative cause. I know I'm speaking incredibly broadly, but frankly I had a similar question as you some years ago but without the benefits of being a full stack engineer.

Disclosure: MSc in Digital Currency holder.

Thanks. However I am not much into Crypto and Blockchain, just distributed systems (not necessarily decentralised).

Completely understood!

I always thought distributed systems weren’t necessarily decentralized systems. Is my understanding of the terminology wrong?

For example a system that uses distributed algorithms for performance improvements, or a system that is distributed by design and necessity to retain the benefits of redundancy, could be considered both centralized and distributed.

Nope. Your understanding is spot on. Mine is the one that's flawed :P.

That’s not what a distributed system is. I think OP was talking about distributed systems in the computer science sense, and the theory around it, e.g. lamport clocks, distributed synchronization mechanisms... You could argue that blockchains are also distributed systems, but only in a very limited sense.

To OP: I think it’s absolutely worth it. Any car is a distributed system, even smartphones (by themselves, not just as part of a network!) have so many independent devices with full-blown CPU cores in them that you can call them distributed systems, and as you probably know by now, it’s absolutely a non-trivial topic. So if it’s something that deeply interests you, there is a market.

Given your full stack background (by which I assume you mean web development?), you might first need to look for work on distributed systems that are strongly associated with the web, web services etc.

Which full stack?

Java, Scala, Ruby, Python, Angular.

Interesting. Some time ago I had the ongoing debate with a colleague on what it meant to be full stack.

I was under the impression that full stack meant you can do what's needed to stand up a solution at literally every OSI layer.

For example, from down at the base of infrastructure, networking, hardware/cloud, servers / platforms, integration, data / database, custom and COTs applications and services to clients, UI, desktop/web/thin/fat etc. etc.

By that measurement someone who can do all that probably already understands and in a sense, works in distributed systems - you probably had to already to cover the breadth and depth of all that knowledge.

Not picking on your choice of words here ("full stack"), just mulling it over and thinking: 1) if you can do that you're qualified to work on distributed systems and 2) there's no shortage of work advancing distributed systems and that problem isn't going away any time soon.

I'd say distributed systems is challenging (I worked on a startup doing that with database federation and it was really interesting and challenging), fun and important.

I think maybe the bigger question is, are you interested and want to dedicate the effort needed to get into it, and will that make you happy? There's so many levels of working in distributed systems - from research, inventing / building the tech, to implementing existing solutions (e.g. Apache Spark, Kubernetes etc. etc.).

If that stuff is interesting I'd first decide if I wanted to be on more on the research or practical side, and then how much money you need to live, and then choose my entry point into the field.

> Interesting. Some time ago I had the ongoing debate with a colleague on what it meant to be full stack.

Yeah, that's what makes this term a poor one IMO. It's not at all clear what it means. I've always taken it to mean "not merely Javascript."

I think most people think of a fullstack developer as someone who is experienced in everything from backend all the way up to frontend design.

Fullstack developer = someone who can do all parts of a project on their own if they wanted to.

What does "backend" mean? Just the application layer? If you're writing a Tomcat application, do you have to know how to debug Tomcat itself and submit patches for issues? Do you need to know how to write a Java compiler or an OS kernel? Do you need to know how to design microprocessors? What about how to fab them?

Calling the top two layers of a very deep stack "full stack" is a bit strange.

I usually picture a JavaScript front end engineer who’s moved into backend NodeJS and picked up some DB knowledge along the way

Not really “full stack” with no OS or DB...

Reply on news.ycombinator.com