Hacker News Reader

Comment by corytheboyd | Hacker News Reader

I'm going to come clean, I am a shitty software engineer.

I'd like to have things like CRDTs under my radar to pull from when architecting technical solutions. But I don't.

I have been coddled by the bullshit of "just-good-enough" web development for nearly a decade, and I feel like I will be haunted by it forever at this point.

I WANT to employ mathematically proven solutions, but not only has it been 10+ years since studying computer science (and with it, the relevant mathematics) but I never even graduated.

So here I am, another under-educated over-paid GED holding Rails engineer putting together shit solutions I know will eventually leak.

So, having said all of that, is there hope for adding things like this to my toolbelt? And where do I even start? I feel like going back to school full-time is out of the question, because I have a life now. Maybe online classes? I'm curious to hear thoughts from people who maybe came from a similar pit of despair, or otherwise understand what I am talking about.

It's just... sad to know enough that I produce less than optimal work but don't know enough to confidently prevent it from happening over and over and over again. Is this just the profession I have locked myself into?

View on HN

For every million bricklayers, there may be one or two material scientists devising new clay formulations to make stronger, cheaper bricks.

The irony is that the bricklayers are often paid better, and they can be certain that their profession will remain in demand for the foreseeable future.

The material scientist may hit a dead end, or their company may axe their entire department to "cut costs".

One of the smartest human beings I've ever met ended up at Google, working on something very similar to CRTDs for the Google Wave project. The outcome was a commercial failure and his team was disbanded.

I did the same classes as him at University, and I use almost none of that knowledge in my rather pedestrian job. I make nearly twice what he does and I have nearly perfect job security. So there's that.

This is exactly why I've focused on boring, mainstream tech skills. I'd rather be doing audio dsp or machine learning or fancy AR apps but I figure vanilla js/web chops are going to be in demand for the foreseeable future and pay well enough to support a decent lifestyle.

Good thing this is just a dress rehearsal for when you can spend time working on things that interest you!

Look at it this way: Other people tell us what they think we should do with our lives. What we should find interesting, or fulfilling, or is good for our career.

Meanwhile, I feel like I'm an adult and can make my own decisions. I don't care if what I like is not "cool" or "hip" or some "up and coming technology". I don't have to be in research or the cutting edge of something to make interesting personal discoveries.

In my career I've accidentally stumbled upon multidimensional OLAP cubes (in the form of SQL Server Analysis Services) and discovered that they're quite fun to work with, actually. The reputation of business analytics as something "dry and boring" is undeserved. It's a wonderful mix of human problems like "manufacturing calendars" and quirky sales bonus calculations on top of fancy optimisation tricks to make calculations over terabytes of data nearly instant.

I've also had the incredibly gratifying experience of making a bog-standard web app with only a few mildly tricky technical bits for a department of education. It helped about a million kids read more than they would have otherwise. That's something. In fact, I'm certain that my one library app made kids read more books they wouldn't have otherwise than a dozen English teachers.

My parents were educators, and I outperformed them a hundred fold without ever stepping foot in a classroom as a teacher.

I actually did take a couple of years off and try to make a living on my own selling music software with a lot of handwritten DSP code in it. I learned a lot and it was a lot of fun but I also realized that to a certain extent programming is programming no matter what the domain is. Things that seem exotic and sexy on the surface actually involve a lot of the same considerations and problem solving that building a website does.

So yeah there's no shame in working in more mundane & utilitarian areas. And, like you said, you might reach more people with your work that way.

I had the same experience doing game programming. It's not at all like gaming, and almost exactly like all other programming, except with more linear algebra.

Even simpler, if you just "know" SQL and can form a sentence, there are companies that will pay you 100k+ a year (source: I worked at one).

Google Wave used OT, though, not CRDTs. Are you saying that Wave also tried to use CRDTs, but failed?

Wave was disbanded, but the technology was apparently integrated into other things such as Google Docs, which uses OT for collaboration quite succesfully (despite the criticisms in OP's article).

Author here. I love this comment.

> So, having said all of that, is there hope for adding things like this to my toolbelt?

Yes; but not yet. Right now if you want you can use yjs or sharedb. But the APIs aren't spectacular. Eventually I want these things integrated into svelte / react / swiftui and postgres / sqlite / whatever so it just sort of all works together and we get nice tutorials taking you through what you need to know.

We aren't there yet. Automerge is slow (but this is being worked on). Yjs is hard to use well with databases, and its missing some features. We'll get there. The point of all the mathematical formalisms is that if we do it right, it should just work and you shouldn't have to understand the internals to use it.

Loved your blog - but I think the parent's question was how to gain the skills of working/designing OT/CRDTs etc on their own. (I am clarifying the question - since I'm interested in the answer as well)

I honestly think a good skill to have as a programmer is to know when something is a complex, niche problem with inherently "fragile" implementations (as in, it is really easy to make mistakes) that is therefore best solved by a few domain experts working on an open source library, and just use that library.

Based on everything I've seen about them, OT/CRDTs sound like they are that.

(Author here) I agree with that conclusion. I think they're really fun to implement and explore. I've been working in the area for a decade. And I've implemented over a dozen variants of OT systems by now. And despite all that I still consistently make mistakes that are only found with a fuzzer. Even thorough unit testing is never quite enough.

I recommend implementing them for the fun of it. I love it. But don't put anything you write into production until you can leave a randomized test suite running overnight without errors.

Example:

Random op generator: https://github.com/ottypes/text-unicode/blob/master/test/gen...

Which powers this fuzzer: https://github.com/ottypes/fuzzer/blob/master/lib/index.js

This guy just asked how to get into them because he is interested in them.

You just replied don't. Have you considered he wants to this regardless of not applying them purely commercially and wants to perform research, or in a novel industrial capacity?

I wasn't telling them not to learn this, or at least that wasn't the intent. Learning for the sake of learning should never be discouraged, and I apologize if I came across that way! I was responding to this last sentence:

> It's just... sad to know enough that I produce less than optimal work but don't know enough to confidently prevent it from happening over and over and over again.

Basically, the notion that not knowing how to implement this means the resulting product is sub-optimal. My counter-argument is that delegating work that needs expert domain knowledge and skills to the experts and use the fruits of their labor will result in a better product.

Also worth checking the new CloudFlare 10th cakeday announcement for their serverless approach to statefull objects. As I understand it, they can accomplish some of the basic CRDT use cases without the technical overhead.

[1] https://blog.cloudflare.com/introducing-workers-durable-obje...

What are you missing from Yjs?

I'm sharing the vision with you that things should just work. It literally takes you two lines of code to make any of the supported editors collaborative. The Yjs types are a minimal abstraction over the crdt model. I hope that other people build more abstractions around Yjs (for example a react store that simply syncs).

The thing is.. all of what you are talking about is already possible. A wasm port will eventually bring some performance improvements. But Yjs is already very close to optimal performance and it works in all browsers.

But I really am interested in more feature request. I'm mainly interested in building collaborative apps with CRDTs. CRDTs as a data model is an entirely new concept and we need to get experience on how to build apps with CRDTs. And also we need to find out what is still missing.

So head over to discuss.yjs.dev and share your experience.

I want yjs to have:

- Support for moving objects (as well as inserting and deleting)

- Better support for long lived documents. (I want to be able to pull historical deleted stuff out of a document so it doesn't grow unbounded).

- Presence information and markers (this is sort of supported now?)

- The gorgeous automerge API frontend ported across

- Support for interacting with yjs from rust / C / swift / etc

- The gorgeous automerge API frontend ported across

I want all of that too. Although I would add Automerge's API on-top of Yjs as an additional layer. It really does play nicely with Web Frameworks like react. Although for building editor bindings I would choose the Yjs types because they allow faster access to the data. Yjs only provides a minimal abstraction around the CRDT. I hope that other people will build nice stores/abstractions around Yjs that make interacting with shared data easier.

- Presence information and markers (this is sort of supported now?)

Presence is handled by a different CRDT (the Awareness object in y-protocols). All providers implement the Awareness protocol. In many cases you want one awareness instance for several documents, so it makes sense to separate them.

Markers is probably what I refer to as relative positions: references to specific positions in the document. It even plays nicely with undo-redo.

- Better support for long lived documents. (I want to be able to pull historical deleted stuff out of a document so it doesn't grow unbounded).

Yjs does this reasonably well. To a point that you can even build 3d applications like relm.us with Yjs. But you are right that more work could be invested in researching even better solutions.

- Support for moving objects (as well as inserting and deleting)

Moving of elements can already be modeled with Yjs [1]. Although the approaches described are not very convenient to use. Automerge implements this feature natively, but I'm a bit hesitant to implement the next best approach because I see many downsides that conflicts with features I plan to implement. There is a discussion thread where I outline a few of the tradeoffs [1].

[1]: https://discuss.yjs.dev/t/moving-elements-in-lists/92

You are not a shitty software engineer who's been "coddled." What you likely are is dissatisfied with where your career is going (and who amongst us is not in some way) and nervous about not having a name brand pedigree so you are turning it inward and trying to understand it in ways that imply the cause and effect are localized to yourself. That's understandable but I wonder if it will get you anywhere. Stop beating yourself up and start exploring what you really want.

Do you want a brand name? Go work at a FAANG. Trust me, they're hiring. You'll get there and like me you'll realize they too have their problems and maybe the "over-paid Rails shit solutions" you "know will eventually leak" maybe weren't so bad after all.

Do you want to level up as an engineer? Go work at a hyper-growth startup (probably series B) that is falling over from its scale problems. You will probably end up solving some oddly challenging and novel engineering problem along the way.

Do you actually not mind this stuff but hate feeling like you have to always keep up with the Jones and it gives you anxiety? Go find a therapist that you have good chemistry with and see if you can't work out why you feel this way and how to fix it. To be honest, the corporate rat race thrives at making people who are otherwise doing quite well with their career progression feel like they're not because that makes them easier to exploit. Life is not an exam to be min-maxed. You don't take that fancy career or lifetime earnings with you when you die, so unless you're doing it for the intrinsic joy of doing it, there's an argument to be made that becoming better at engineering has no guarantee of making your life better.

See this blog post for just what it is -- a breathless discovery of a hammer and an author who now wants to use it for everything but may not be at the point where they realize it might not be a great tool for everything. You the person are probably a lot more competent and capable than you're letting on here. And if that's the case and you're unhappy for reasons in your control, hone in on why that is and see if you can't make life moves to change it. There's never been a better time to make a career move.

Good luck. I believe in you.

  To be honest, the corporate rat race thrives at making people who are otherwise doing quite well with their career progression feel like they're not because that makes them easier to exploit. Life is not an exam to be min-maxed. You don't take that fancy career or lifetime earnings with you when you die, so unless you're doing it for the intrinsic joy of doing it, there's an argument to be made that becoming better at engineering has no guarantee of making your life better.

This. Especially that last line. I feel most of the time I'm just wanting to get more and more, both financially and challenges wise. But to what end? Your words here really calmed me down. I got me thinking. I felt like a dog running after a car.

You and me both, my friend. Perhaps now more than ever before, we are not so much dissatisfied with our careers (with a dissatisfaction that has always been present), but more able to think deeply about why that is the case. In my opinion, the answer isn't pleasant and yet, it's strangely comforting (even freeing) in a sense.

Now that I have accepted that my lasting happiness will not be found in a prestigious job or fame as an engineer, where shall I find it? Perhaps the same place other humans have for the last ten thousand years. In the center of a quiet meadow in the woods. Next to an animal that considers me a friend. In the arms of someone I love. Plucking strings on an instrument, singing my stories.

Is this response a coping mechanism, or an acting choice to minimally engage with the parts of human existence which entail naked power brokering and empire building? Maybe it's a little of both. May we both find our own peace somehow, friend.

When this guy discusses OT vs CRDT he's discussing algorithms for some very specific use cases. Namely, concurrent document editing a la Google Docs. Realistically, there's maybe 50 people in the world who are versed in these kinds of algorithms at a really deep level. 99% of us are users. If I was pressed into it, I maybe have the chops to implement a reference implementation for something like this, but I've never come with a country mile of an organization that has the resources to invest in something like that. You can be a great programmer with a very successful career as someone who only knows how to read an API and use it. And unless you are tasked with creating a concurrent document editor, you will probably never need to understand the API for a CRDT.

generally agree, but I bet there is more like 500-5000 people that understand this kind of stuff. Most modern web tools support at least some basic functionality of synchronously working on the same document.

Concurrency is really broad problem space with a million applications, but not real-time editing of an indeterminate data object. Concurrent modification of a row in a database is a perfect example of deeply complex problem that 99% of us don't really need to understand beyond how to use the API. If you know how to commit a transaction, you don't need to read the thousands of dissertations on how to process them optimally.

I wrote an article for folks like you (because I also felt like this was my life at one point).

In this article I explain the basics of the simplest CRDTS the way a web developer without a deep knowledge of CS could relate to it: https://statagroup.com/articles/editing-shared-resources-crd... . I don't present the concept 100% fully, but I give a starting point for someone who "only knows rails". I found most online tutorials dive directly into the theory.

I included all the resources I used to learn the concept on the page too.

It's part of a series I wrote about editing shared resources ( https://statagroup.com/articles/editing-shared-resources ).

I feel ya, I did not graduate either, and didn't go for CS. After having Linux as a hobby, I made my way into a web dev job. The "good enough" work is killing me. I feel like I am pushing and pulling my team to the light, even simple things like DRY.

I think one thing I'm learning is where to spend my effort. Its definitely a balance, but I'm putting more time/passion into my personal projects/learning, whereas before I saw more opportunities for overlap with my passion and work. When your work doesn't recognize that effort or actively fights against those efforts, it's draining and frustrating.

I've also been thinking about dedicating some serious time to learning. I keep thinking about the idea of taking a month or two off from work. Just treat myself to my own mini-semester of homeschool. I'm somewhat confident I could learn enough to land a better job, but more importantly, one I enjoy more.

You guys are overestimating the degree to which concepts from CS curricula are relevant to the average engineer's work. "Pushing and pulling your team to the light" is normal, can't be solved by getting a CS degree, and is more about your team dynamics than anything.

No, you have not locked yourself into anything, you can always grow. Comfort can breed complacency, so my initial advice is to quit your job and try to get a new one that's more demanding. It forces you to kick your own butt into learning new stuff.

I am an under-educated over-paid GED holding Glorified Sysadmin, and while a lot of the shit solutions I work on will eventually leak, I do spend a non-trivial amount of time looking for better solutions. I also try to keep my ears open for what other people are working on and read up on concepts I'm unfamiliar with, whether they crop up on HN or through work.

I learned about CDRTs several years ago at a former employer where a team mate was talking about how it was the future. However, we didn't end up using them. It turns out the more advanced your system gets, the more work goes into it, and the more of a pain in the ass it becomes to run [until it reaches a certain level of operational stability]. So I wouldn't say that using the most bleeding edge research is a good idea even if you know about it. Most people are still fucking up the basic stuff we were doing 10 years ago, and the neckbeards here probably have loads of stories about how much simpler and more stable distributed systems used to be.

> So, having said all of that, is there hope for adding things like this to my toolbelt? And where do I even start?

HN is a decent place to start. Now that you've read this article, you're aware of the general landscape of concurrent document editing algorithms.

Ultimately, all you really need is the language to a) describe your problem and b) compare potential solutions. I have a CS degree, and it's not like we spent semesters learning every algorithm under the sun. But what we did learn was the language to describe and compare the performance of algorithms (computational complexity theory). Beyond that, it's just about knowing enough terminology to be able to describe your problem, which you could learn just by skimming through an algorithms textbook (or random Wikipedia articles, honestly).

For example: a friend of mine was recently trying to devise an algorithm for ordering buff applications for a collectible card game. Once I was able to find the right language to describe the problem (topological sorting), finding an algorithm to use was easy.

The article mentions Martin Kleppmann. Go buy a copy of his "Designing Data-Intensive Applications". It may well have been titled "Practical Distributed Systems for the Everyday Developer". It is an absolutely fantastic book with a perfect ratio of theory/practice.

Extra points for buying a dead tree copy and reading it without a thousand alerts and internet temptations vying for your attention :)

Sounds like you've got an awareness of CRDTs now. That's a win and a differentiator from many other engineers.

I already knew about them, but there will be a stack of other stuff I don't know about. Just get Googling when you hit different problem areas.

I found out about CRDTs because I was working in a niche where a security device was used that breaks TCP between two hosting regions for an application. So I searched for issues to do with that and refreshed myself on the two general's problem, Byzantine fault tolerance, 2 and 3 phase commits and at some point OTs and CRDTs popped up. You'll find this stuff if you keep looking at a problem for long enough.

That was a while ago though and despite still working in that area I've still not implemented anything with them because I'm a contractor and I'm under the direction of people who don't have CRDTs in their toolbelt and don't want ideas from others! I guess my cheerful message is keep at it but know you can only do so much and there's plenty of other pits of despair to fall into even when you're out of the current one. Yay!

I never graduated, but that was back in the 80s. The important part is not getting "official" education, but learning how to self-learn. You don't need to necessarily understand the underlying maths, if you can understand the general theory.

And you're not a "shitty software engineer" if you are self-aware enough to know what you don't know. Read, start with Wikipedia, look up the references, play with stuff, explore.

If you need to understand certain things at a deeper level, then explore the classes and other guided tuition.

I've been doing this stuff for 30 years and I still learn every day. You will produce less than optimal work for ever. As a master craftsman (which is what you are, not an "engineer") you are always learning and improving your craft.

You can be satisfied when completing some work if you did your best to produce it and it works within the constraints you had. There will always be things you would do different or better, so do that next time.

CRDTs are just applied partial order theory. Partial orders are just mazes where you can drag your finger in a certain vague direcion, taking arbitrary turns in that vague direction, and still find the exit.

Get your intuition down really good and 10+ years is not required.

I can give a philosophical answer rather than a technical one. Our lives are filled with learning various things. And we all realize there is no end to it. But truly what makes you feel better about what you do is not this endless learning. It is when you can drop what you learnt and think freely about a problem that you can solve. Until you reach there, you are in a boat tossed by the waves trying to grab whatever you can to keep yourself afloat. Please disregard this answer if it is not of any help.

I might suggest starting with one thing at a time, pulling in more and more. No one learns it all at once.

I might suggest Martin Kleppman's Making Sense of Stream Processing freely available from Confluent [0].

[0] https://www.confluent.io/stream-processing/

Picking up Elixir is a decent way to get into backend distributed systems from a Ruby/Rails background.

I think this is surprising levels of true. Elixir really works as an on-ramp from web to a lot of other stuff.

I love the candor in your post but you're being too hard on yourself.

Learn functional programming, gradually. You'll get there. I like to think I reverted a decade of brain damage inflicted by enterprise java development in three years.

I've worked for multiple Rails shops with high quality engineers. The trick is to interview them too before accepting any offer. The best teams I've been on have been small. Past a team/project size, maintaining high quality code becomes much more difficult.

Check out http://www.roomservice.dev

Turns your front end app multiplayer with a library.

They manage the hard part and give you a simple API to get the benefits of realtime/syncing/diffing.

> The major downside of having an “always syncable” data structure is that it gets really big, really quick

Perhaps it would be prudent to read the actual article before tossing off this sort of comment since it's indeed the whole point?

Normally I would downvote a comment that amounts to "read the article", except that this one is entirely on point.

One of the classic problems was size and performance. However both problems have now disappeared. Operations are `O(log(n))` and the size overhead is a factor of 1.5-2.0. That's quite workable.

Mind elaborating more on how’s is it better than CRDT?

This is my daily struggle routine. I cannot get rid of the idea that proceeding my career only accumulate how much I earn, instead of making myself a better programmer.

> And where do I even start?

There are so many resources available to learn CS today. Are you really baffled about how to learn outside of school? Read. Practice. Repeat.

Hey are you me? Cause this sounds like me.

I am in this photo and I don't like it.

Reply on news.ycombinator.com