Hacker News Reader

Comment by tptacek | Hacker News Reader

This thread has (at least at the moment) serious Bob Martin Sudoku Solver energy to it. Tailscale has solved an infamously complicated problem using, for the most part, simple tools. They're not just successful; they're remarkably successful, spookily successful, upsettingly successful.

Consider whether the secret sauce here might not be au courant database choices, but rather something much harder for random teams to pick up and run with: just straight up good design and programming.

This is a thing that happens in other fields, too; some of the hardest dishes to cook seem extremely simple and have few ingredients. They're hard because there's no place to hide: your technique has to be flawless or they don't work. A famous entrance exam for new cooks at fine dining restaurants is "make an omelet". Tailscale is, thankfully (and, to me, irritatingly), more like an omelet than a truffled hash.

View on HN

> This thread has (at least at the moment) serious Bob Martin Sudoku Solver energy to it.

To the forum's credit, the top comment (by u/judofyr) the last time tailscale blogged about their unusal db setup, got it exactly right: https://news.ycombinator.com/item?id=25768042

> Interesting choice of technology, but [tailscale] didn't completely convince me to why this is better than just using SQLite or PostgreSQL with a lagging replica.

> In particular [tailscale has] designed a very complicated system: Operationally you need an etcd cluster and a tailetc cluster. Code-wise you now have to maintain your own transaction-aware caching layer on top of etcd. That's quite a brave task considering how many databases fail at Jepsen.

> Considering that high-availability was not a requirement and that the main problem with the previous solution was performance ("writes went from nearly a second (sometimes worse!) to milliseconds") it looks like a simple server with SQLite + some indexes could have gotten [tailscale] quite far.

I did commit the cardinal sin of HN metacommentary and referred to the state of the thread at a point in time as if it was a durable property of the thread, which was a mistake I own and take full responsibility for.

If anything, this ^^^ is how you win the long game. History will surface itself when you make assertions. Gracefully revocability scales.

At least he's not the Dropbox guy

What does that mean? Can you explain?

This comment trivializing Dropbox: https://news.ycombinator.com/item?id=9224

I for one do not think the comments trivialized anything. I see these as inevitable questions that a leader like Drew Houston should be prepared to answer at some point. Think of it this way, would you disqualify me if you were hiring and I asked a similar question at the interview?

I think the underlying actual question for Dropbox to ponder (but perhaps not answer publicly) was "what is your moat?" without the business development language and at least for me that remains an open question.

I remember I was in college when Dropbox started and I thought Dropbox was silly to offer more storage for inviting more users. It was easy to approach people and ask them to try something in college. I think I almost maxed out my invites. I remember feeling I was taking Dropbox to the cleaners...

One activity question for HN: why do you think Dropbox continues to exist today while copycats like Barracuda Copy have failed?

> One activity question for HN: why do you think Dropbox continues to exist today while copycats like Barracuda Copy have failed?

FWIW, i've been using Dropbox since practically the day they went online, and have been a subscriber for all but the first few months of that time. They've never once failed me, never been offline when i needed them, never lost anything, they created a Linux sync client out of the box (whereas Google first claimed the would make GDrive for Linux but never did), and it's easy to manage via multiple interfaces (mainly their own web UI and https://rclone.org, which can proxy your dropbox behind an add-hoc FTP server running in your LAN).

They're extremely pricey compared to the same storage in GDrive, but Dropbox has never let me down so i'm in no hurry to drop them.

That comment is unfairly judged, taking into account how much they used AWS instead of their own solution.

"...Half-a-billion people stored files on Dropbox. Well, sort of. Really, the files were in Amazon’s cloud. .." [1]

[1] "The Epic Story of Dropbox's Exodus From the Amazon Cloud Empire"

https://www.wired.com/2016/03/epic-story-dropboxs-exodus-ama...

Why can't a company use cloud? It's not like there's Amazonbox client I could install now and get the same service. Dropbox never was about the storage itself, IMHO - it was about the ease of usage. There were enough options to get storage - cloud or non-cloud - when it came out, but none of them had the easy folder-based sync and sharing (public folder) of Dropbox.

It's not if the company uses cloud or not. It's if they are solving the hard problems or instead, outsourcing the technical challenges to their cloud provider while charging the user a non justifiable premium for the intuitive user interface :-) So it's about the eventual value add and the business case.

The challenging part is to provide the back-end. Get massive storage, with proper checksum integrity, at a cost effective value and with flawless security.

A single command (and minor variations around it):

aws s3 sync <name_source_dir> s3://<name_target_bucket>

get's you all that. It can be easily mapped to an intuitive user interface by a software development boutique in weeks.

And because that is the difficult part, it seems for the first 8 years of their history that was solved for them by AWS, according to their own reported history.

Sharing a folder or file from Dropbox is no more intuitive than using the S3 console. Looking at the very colorful history of security issues around Dropbox [1] it's certainly not safer.

So why would a user pay more for storage costs than with a cloud provider, with notoriously less security? Does a tiny layer on top one or more cloud providers justify the add-on value for the user or the current valuation of 9 Billion dollars?

[1] "Criticism of Dropbox": https://en.wikipedia.org/wiki/Criticism_of_Dropbox

And for what its worth, systems like minio have emerged due to that commercial innovation. AWS has made a lot of investment in that system and will earn perpetual annuities from it, but by creating a defacto protocol we now get many choices that use its API. Its interesting because AWS never was benevolent with their S3 code, they just created a novel and rigorous offering that became worthy of emulation. Dropbox really hasn't done any equivalent contribution. If anything, the open source projects that have emerged to offer self-hosted alternatives to it are more feature-complete and extensible- such as nextcloud/owncloud, pydio, syncthing, etc.

side note: I'm filled with regret for every document that I authored on Dropbox Paper. It didn't last for long, but there was a time that it was the official wiki for a company I worked at. Its not impossible to export, but never with full-fidelity.

> Sharing a folder or file from Dropbox is no more intuitive than using the S3 console.

I've not really used the S3 console but, at least on macOS, sharing a file from Dropbox is as simple as "right-click on the file in Finder, Share..." then enter an email address or create a share link from the popup. Or I can right-click and "Copy Dropbox Link" if the direct link is good enough.

Is the S3 console simpler than a right-click?

Even easier and safer, create a signed url that expires after 5 min...Send it to the user...

aws s3 presign s3://awsexamplebucket/test2.txt --expires-in 300

...are we deliberately reenacting the original Dropbox thread here?

It usually ends up happening when the Dropbox thread comes up, it’s really interesting seeing both sides rehash the same points.

But now you have to know which bucket the file is in, have the AWS CLI tools installed, use Terminal, copy the URL to messages / email, etc.

How is this easier than "right-click, pick a menu option"?

(If I want an expiring Dropbox link, it's there under 'link settings' in the pop-up. Admittedly only has date granularity but still there as an option.)

Example from the console: "Setting up an Amazon S3 bucket to share files"

https://gist.github.com/pjkelly/be2cd3881e766620a411

Again, how is this easier than a single right-click in Finder? Also don't those files need to be public to create a link? You can't create a link to a non-public file which grants access either directly via the link or with a password?

Their fundament is UX, not technology. Their technology could be whatever and users wouldn't care - the users care about syncing their files easily. Few geeks might flame about it on a tech forum, but nothing more.

Not that I like Dropbox too much. I'm going to switch to my self hosted solution. But it's certainly not as easy as using Dropbox, even as someone who uses AWS since 2012 frequently.

The median Dropbox user doesn’t know what s3 or is what ‘console’ means.

This and the “no Wi-Fi less space than a Nomad” comments on slash dot about the iPod are my favourite examples of “you don’t get consumers” comments.

to quote u/InGoodFaith (news.ycombinator.com/item?id=29555814):

> Wouldn't also be HN without the misinterpretation of BrandonM's response. Here is dang's comment about that topic/meme: https://news.ycombinator.com/item?id=29178442

As someone who was there to witness it when it happened, that is not the impression I got at all, and I think Dang may have taken the more humble interpretation in that remark.

You don't need to have been there, though, as anyone can read the perfectly-preserved exchange even now.

And some things that should not have been forgotten were lost. History became legend. Legend became myth...

Lmao Stavros we should share a laptop or something or just speak over the phone the way we run in to each other so often.

Its different reading it at the time because it factors in the context of the community at the time as well as the lived experience of what it was like at the time to be a member or participant in given discussions. With that contrast, you can then get an idea of the intent of those comments beyond just the words, as they are often (and in this case) reinterpreted through the lens of current day contexts.

Example: ask what people think of the hit TV show The Apprentice today in the present, then hop in a delorean and ask someone before the trump presidency or candidacy what they think of the show.

I guess that's true, the cultural context is different now.

You can call me anytime <3

The Dropbox guy admitted he was wrong too, but we won't let him live it down. At least we don't remember his name.

People seem really stressed that they've changed databases twice before. Their service never went down and they didn't stop delivering features while all this was happening, so I don't see the big deal. You pick what works at the time, and incrementally evolve as problems and solutions become apparent.

This article could have easily been "why we were down for a day while some auto-vacuum setting was broken." Then everyone would be like "why didn't you just scp a json file between machines" ;)

Another comment on this thread tried to dunk on them, saying that they'd done this so many times before (JSON to etcd, etcd to sqlite) that they must be getting really good at it. That comment struck me as so close to understanding what they're doing right here, it was painful.

Yeah. Maybe people are overestimating how expensive this kind of change is. We can all imagine being part of a larger org, where one day an All Hands appears on your calendar, and some exec you've never heard of proclaims the beginning of a 2 year multi-team database migration project. So now instead of making nifty software, you are just in 40 hours of meetings a week fighting over small details on this migration nobody wants to do, and everyone is miserable about it. Features stopped getting released, people leave the company, the weekly status meeting never makes the project seem anywhere near completion.

Say "switch databases on a successful project" and I bet people's minds go there. That is indeed expensive and painful, but if you have a small team, it's just being agile.

There is also the "punching up" / sour grapes aspect that can't be ignored. "I would have picked the right database from the start, but nobody's funding my crazy idea." Life just isn't fair, and HN is where you can vent, I guess.

I have the odd feeling that this kind of overestimation comes from the experience that folks tend to have in writing more traditional N-tier applications, especially post-Rails. A certain kind of database-first, ORM-first design philosophy really took hold after that, and it became somewhat common to see incredulity in response to telling someone that you weren't using something like MySQL or Postgres in production.

The kinds of projects that I more generally tend to see make these migrations successfully are the ones that design with in-memory data structures first and then sort out how best to persist that kind of thing to disk after their domain model has solidified a little bit.

Tailscale smells a lot like that to me.

When I used dynamic languages (Node.js and Ruby) I leaned on the database as the structural source of truth for types—it was the rigor a successful project needs.

But as I’ve been using only statically typed languages the last few years (on the backend), I’ve moved to designing the domains in the code, like you said.

Yup.

When Tailscale migrates databases, it is a few engineers working for a few weeks solving problems that already exist.

When a company with 50k employees does it, it takes two years and only makes things worse.

+1 on good design and programming.

My go to language of choice is PHP/MySQL. I've never not been able to scale a project. At one point while working at Comcast I prototyped a system for every error code from every cable set top box (150 million) in the country on a MySQL server instance running on a MacBook Pro pulling data from Splunk in real time. All so I could generate some png charts and embed them in Slack.

I know there are limits to MySQL but so far in my career I haven't had any projects hit those limits because hardware has been getting better faster.

But my Kubernetes Cluster on AWS can handles this for only 30.000$ a month

That Comcast prototype sounds amazing. PHP and MySQL can really bring you very far, i use it for nearly everything and it is so incredible easy to develop and “deploy” new versions. I use propel as ORM and it is very close to the DB and I can change schemes fast and just let the autoupdate handle the rest.

> But my Kubernetes Cluster on AWS can handles this for only 30.000$ a month

A k8s cluster for something like that wouldn't cost anywhere that amount, plus you would get the advantage of being able to deploy your app in multiple availability zones. If you don't need k8s features, there are definitely simpler options that will give you redundancy and won't break the bank.

I only used a MacBook cause it was faster to ask IT for one than to have my ticket processed and approved by the cloud and security team which both required a write up on why I need a vm and what justification I have for needing it. Meanwhile IT is like "here you go".

"know there are limits to MySQL"

I have gone past mysql limits a few times. Will you go beyond its capability is a simple measure. Can your hardware write fast enough to keep up? If your hardware maxes out writing to disk 200 megs per second, welp mysql maxes out at 200 megs per second. If you need 400 megs per second of writing, well you need two instances that can write independently. That is when you are fucked. At that point you get into sharding* and doing whacky things mysql isn't meant todo. Reading things from disk, mysql's replication scales just fine.

That's what I mean though. I've never hit those limits. These days it's easy to put together a raid 5 with NVMe disks and you'll quickly reach gigabyte per second write speeds.

What cloud service can you do that on? Seriously asking.

This comment makes me unreasonably giddy.

The replies are focused on MySQL (and to a lesser extent, PHP). None of that matters.

The hero here is the logging system serving as seekable stream of collected events. (I’m trying hard here to not say K-fka).

By consuming events from that source and being able to perform time-based seeking, you’ve offloaded (or dare I say, eliminated) your scaling pressures.

Even if you commit the “sins” of no indexes on heavy queries, using MyISAM, or building a giant, memory-intensive associative array… none of it matters.

You can choose to wipe your local dataset, start again in a different language, switch to another data store model, and none of your downstream consumers will know nor care (unless you blog about it).

*This* type of design, that prioritizes consumer-facing resiliency and continuity, is what I look for in systems that scale.

One of the largest forums online in 2003 was was ezboard. A billion page views per month. Back then - insane - well that was beyond insane. I started work there as a young idiot. "What db you guys using, mysql?...." long pause... "We use a file system." I dug in. Yep. A fucking file system. Files allowed: simple backups, impossible speed, obvious cache systems, partitioning.

Later on i met the roomate of the author of memcache. Few hundred lines of C. Completely changed the game how caching systems worked. Years later a super senior FB engineer explained to me over a beer - "we basically use mysql as a file store, everything else is in memcache." Good ideas are stupid simple.

The guy who wrote memcache is on this thread, I think?

bradfitz lol he is.

And he even works at Tailscale now.

nobody goes to these restaurants for their omelets, though at the same time nobody should care how tailscale actually implements things so long as it works well.

imo they’re not solving a notoriously hard problem, they’re realizing that they have the tiniest version of this problem (and that will most likely continue to be true). when that’s the case and you want to prioritize stuff that moves the needle like dev velocity, why do anything less simple? getting hung up on building scalable solutions far beyond what you need is somewhere between a fool and a big co employees game. far too often people introduce complexity way before it’s needed, and that almost always slows down your devs ability to actually fix bugs and make the product more reliable.

They don't, but there are other "finesse" dishes that people do in fact go to restaurants for that are notable for their simplicity, and the fact that you can only pull them off if you have flawless technique. Japanese, in particular, is notorious for this.

There’s a manga about a food critic (Oishinbou [1]), where he takes another food critic to a hole in the wall Tokyo restaurant as an example of elite cooking. The chef-owner brings him his meal.

Which is just a piece of cooked fish, rice, and miso soup. The critic is first almost insulted by such a simple meal. But then realizes the fish, rice and soup are prepared to perfection, and is amazed at the chef’s skill.

[1] https://en.wikipedia.org/wiki/Oishinbo

Which is basically also the ending of the Pixar film Ratatouille. The protagonist wows a critic with a simple dish prepared to perfection. It’s a common trope, but also true. “Simple” food prepared well can be much better than a complex dish with many flavors.

Similarly, I’ve also heard (probably from a show like Top Chef, so YMMV) things like a salad or soup are also quick ways to differentiate chefs. In this case, it’s less about technique and more about identifying and mixing flavors.

It works because there is no place to hide. You either have good flavor or not.

Please could someone explain the "Bob Martin Sudoku Solver energy" reference?

All I found: https://news.ycombinator.com/item?id=15441323

my interpretation is that there is a certain class of people who think "if i just apply the process then i will get a satisfactory result", as bob martin does with using TDD to implement a sudoku solver instead of just thinking through the problem and designing a solution up front. i don't think its a particularly great analogy because the bob martin example is about trying to use a "mindless" process to achieve a desired design, not picking an overkill option.

> serious Bob Martin Sudoku Solver energy to it

You mean Ron Jeffries?

I might!

The Bob Martin Sudoku Solver is the same thing but twice as expensive and replaced what little self-awareness it had with unnecessary sexual innuendo.

(Not to namedrop, but, uh) I know apenwarr glancingly IRL and he's amazing. A force of nature. When people talk about "10x" programmers I think of him.

Reply on news.ycombinator.com