Hacker News Reader

Comment by kentonv | Hacker News Reader

Hi all, I'm not sure how responsive I'll be able to be on this thread since I'm in orientation today. :)

But, here's the things I expect to be repeating a lot:

- Sandstorm is still an independent company, under control of Jade and myself.

- Sandstorm is "my baby" and I'm not going to stop working on it just because I have a day job. Yes, it will slow down a bit -- but on the bright side, we were previously spending the vast majority of our time trying to figure out enterprise sales (rather than actually building cool features), and we don't have to do that anymore. So, the difference may not be as big as you'd think. See: https://sandstorm.io/news/2017-02-06-sandstorm-returning-to-...

- Nevertheless I'm pretty excited about the work I'll be doing at Cloudflare as well -- which includes Cap'n Proto and other fascinating infrastructure tech.

View on HN

I'm pretty stoked about future cap'n proto developments. In the scenarios where I've brought up Cap'n Proto, all but the most senior/prolific developers have a near impossible time of ascertaining why one might want to use capnp over competing approaches (e.g. Protobuf). I think, despite a pretty decent description on the website and my own skills in explaining/teaching, such protocols are nearly impossible for the majority of developers to intuit -- so they're all perceived as being nearly identical, save for financial backing and other forms of social proof.

With some of the rough corners filed smooth (e.g. Windows support), a bit more marketing and promotion, and eventually more big names advertising their use of capnp, my job getting buy in from my peers would be much easier.

Performance might be one reason to use Cap'n Proto over Protobuf.

http://dbeck.github.io/5-lessons-learnt-from-choosing-zeromq...

Can you provide a 'top 3' list of reasons to use Cap'n Proto over Protobufs?

1) Cap'n Proto doesn't encode/decode messages thus it's nuch cheaper for processing and memory management

2) protobuf in the proto3 design doesn't cary default values. So if you have a bool field and want to explicitly send false, well you have to change it to some other type or use the default values all the time

3) protobuf generates incredibly large serialization/deserialization support coce for each template. For some languages like Python in can be in hundreds of kilobytes. Cap'n proto messages are significantly smaller

There is more for CnP but Protobuf has much better support and is by default used in projects like gRPC. Also new CnP is lacking speed in new development in comparison to Protobuf.

But I'm using in one of my side projects and I'm very happy with it

Thanks. Those are pretty significant advantages. These things should be on the home page of the project. That's how smaller projects gain share.

The first one is, but as the obviously hyperbolic "infinitely faster" claim.

re: "Doesn't encode/decode messages"

Does this mean that individual fields are encoded/decoded lazily?

Well, you could look at it that way. But the "encoding" is the same encoding that an "int" field in a struct uses, and "decoding" it is a trivial load instruction.

At what point is byte ordering handled?

The format is little-endian, so it's necessary to use a little-endian load instruction, which almost all architectures (including most big-endian architectures) have.

(But every widely-used arch these days is LE.)

(2) is explicitly one of the reasons to use proto3 over proto2, though :) It's weird seeing it listed as a disadvantage instead of the other way around.

Depends on the use case. For instance if you're mapping the data to database, you can't send null variables. Or you store the defaults (waste of space) or you send it as another type.

In our case we've now over 1T of rows. Storing default variables would be incredibly inefficient and expensive.

I'm not understanding your case much. Proto3 does not serialize zeros/false, precisely because it'd be a waste without a benefit. After deserialization, the value is as equal to zero as if the zero had been written on the wire or on disk. Why would you want to store that?

Is nullability of primitive types what you're missing? As in, having a boolean value that can be true, false, or null. If you need that, defining your field as google.protobuf.BoolValue instead of bool gives you that (same with the other primitive types: https://github.com/google/protobuf/blob/master/src/google/pr... ). But making primitive types never null aligns proto3 with most programming languages, making the generated code more idiomatic and performant.

I use protobuf-generated RTTI for the purpose of doing endpoint SQL storage of the messages. (tables, columns, foreign keys, insert/select statements are all auto-generated from the protobuf message) using C++.

Does capnproto provide similar RTTI information to walk through properties or recurse into other messages?

Absolutely. I was the one who originally designed the Protobuf RTTI (aka "reflection") interface, after all... :)

It's actually considerably easier to implement for Cap'n Proto since the in-memory objects are actually backed by byte buffers containing the wire format. No need to compute offsets of class members (which technically violates the C++ standard though it works on all compilers).

> Sandstorm is still an independent company, under control of Jade and myself.

Whenever I hear that I'm subvocalizing 'for now'. How do you see this in the longer term?

(Isn't everything in our life "for now", given a sufficiently long now?)

What kind of hostile commercial interest there might be, especially when the technology is protected by open-source licenses?

If there were a non-hostile commercial interest, akin to Citus in Postgres, won't it be a win for all?

That such independence is - historically at least - usually the opposite. There are very few examples of where post acquisition such independence turned out to be the real deal.

As far as I can tell, Sandstorm did not get acquired or acquihired - Kenton, Jade, and others (?) as individuals all accepted job offers from Cloudflare.

I think some others of the team work for stripe now.

That's correct.

Ah ok, so the team has joined cloudflare but the corporate entity continues with the original owners.

So in some ways better than an acquihire and in some ways worse.

It seems almost universally better? I can't think of any acquirer that would be more successful with Sandstorm than the founders themselves. Nobody I can think of is aligned with "users own their data" and big enough / cares enough to help Sandstorm gain adoption.

Even Sandstorm not gaining traction with their B2B strategy is likely a blessing in disguise: with enterprise money coming in they would have felt a hard pull to do more things that enterprises like. No matter how much conviction you have, making payroll often trumps that.

Now, they've proven that the enterprise route doesn't work. My hope is that the project can continue to improve and gain traction to the point where it sets up or joins a foundation, then potentially try to restart a complementary commercial venture on top of that.

> Now, they've proven that the enterprise route doesn't work.

Well, no... I don't think we proved it didn't work, only that we lacked the expertise to make it work. With more time and some good hiring I think we could have fixed the problem, but the investor money ran out, so...

Ok, that might have been an oversimplification on my part, maybe "isn't a natural fit" would have been better. I'm sure there's enough value there that you could have eventually made a business out of it.

But (as I interpret the blog post, at least) this isn't an acquisition. Cloudflare didn't buy Sandstorm, it just hired all the Sandstorm developers. Those developers still retain ownership of Sandstorm-the-legal-entity and any intellectual property it owns. It's not an acqui-hire, in other words, just a hire.

Which, if that's the case, means that Sandstorm still is legally independent of Cloudflare, and the original Sandstorm devs can pretty much continue to do whatever they want with it. There would be some gray areas, like what will happen to future code or other IP they create for Sandstorm while employed by Cloudflare, or while using Cloudflare systems/resources; and of course their continued employment with Cloudflare may be contingent on their doing things with Sandstorm that Cloudflare likes, or at least doesn't disapprove of too strongly. But if the Sandstorm devs are just selling their future labor and not the company itself, they'll have more independence than they'd have gotten in an acqui-hire scenario.

Correct. The CF people are cool and have no problem with us continuing to operate Sandstorm, as long as we're getting our Cloudflare work done.

Cap'n Proto will presumably end up with mixed copyright, but it's MIT license anyway so who cares?

I'm sure you have already but look over that employment contract carefully.

CF being "cool" shouldn't put your project and its users (me!) at risk.

Yes, we have made sure to add appropriate language to the appropriate paperwork (in addition to being protected by California law).

We've worked closely with Kenton on this. In fact, our lawyers wanted us to add language to ensure that it was clear that we don't own Kenton's IP around this.

Contrast that with the problem for Sandstorm if they couldn't figure out how to fund the effort.

Hey Kenton, you replied previously (in another thread) that he will keep running Oasis since it's your baby. So, let's say Oasis goes down right now and right now you are in the middle of your orientation... does it mean that nobody will look into it until later tonight (that too if you have time) ? What is the agreement with Cloudflare?

If I get paged I'll step out to deal with it. Other team members can also handle pages, though I'm primary on-call.

Note that Oasis is extremely stable. We haven't had an incident requiring human response in weeks -- and the only kind of issue we regularly had before that is now automated.

Congrats Jade and Kenton.

Personal thanks to you for both Cap'n Proto and Sandstorm.

I guess I'm confused by the carefully chosen wording of this post. Did Cloudflare acquire Sandstorm? Was this an acqui-hire? Appreciate the clarity and I understand if there are elements that can't be shared publicly.

No, Cloudflare did not acquire Sandstorm, but four of the seven team members (including both founders) are joining as individuals. This was our preference -- I love Cloudflare but I'd also like Sandstorm to remain independent.

Sorry, I meant for that to be clear in the post but I guess I failed.

It's ok. That's why I asked. I've seen acquisition announcements worded the same way so I just wanted to make sure I wasn't misunderstanding. Thanks for the clarity!

I suspect they intend to keep working on Sandstorm in their spare time (like most of us on here and our OSS side projects), but my have explicitly had their work contracts worded in such a way as that's allowable (or maybe Cloudflare is giving them x% of time to dedicate to it?)

I'm currently in the process of leaving my full time job to work on my OSS stuff full time again. It's a tough move because you do eventually run out of money if you can't get enough grants, donations or a following on Patreon. That's how I ended up back in a full time gig myself.

If you're running a founder-team-only private company without any real assets, the company only really "exists" for as long as the founders can be considered to be its employees. So, if another company comes along and makes an offer to the people to hire them as individual employees—not a company, not even a gelled team, just individual offers—then the company-that-was just sort of ceases to exist, because it now has nobody working for it. I think that's more what happened here.

If the company sells off all its other assets, yes. But in this case, Sandstorm Development Group, Inc. owns copyright in the Sandstorm open source project and (I'd assume) continues to operate the hosted service.

This matters because Sandstorm Development Group, Inc. (and not Cloudflare!) has the ability to grant proprietary licenses to the Sandstorm codebase.

Happy to see work on Cap'n Proto continue!

Hey Kenton!

Congratulations, I'm really happy for you, Jade, and the rest of the team.

I am sad about the projected lower development activity in Sandstorm in the coming months. You had a tremendous community/brand built up, which is very hard to do for decentralized web technology initiatives -- several independent chat groups I'm in are lamenting the news today. If you're aware of any groups discussing further development in the decentralization space, please let me know (email's in profile, or I will check this comment every hour or so).

Seriously great work so far.

The community (including me) is continuing to organize on the sandstorm-dev mailing list (https://groups.google.com/group/sandstorm-dev) and #sandstorm on Freenode IRC. Definitely drop by and say hello!

https://sandstorm.io/community

I've often found that my motivation to work on something decreases if I'm being paid to do it. Side projects are often much more fun and energizing than your day job. Hope you feel the same!

Back when I worked at Google, my favorite weekend activity honestly was working on open source side projects. I'm a workaholic like that.

Even while working full-time on Sandstorm, on weekends I'd work on "things I want to code" rather than "things I need to code".

But where do you fit the LAN parties or that FF playthrough? Do you sleep?

LAN parties are every third Saturday. :D

I usually take a week vacation when there's a new FF to play. :P

Are you going to integrate any of the LANGSEC principles or stuff like formally-verified parsers at INRIA into Cap n Proto? Or have you already? I've been especially curious to see what a combination of your excellent scheme with verified parsing/protocol tech would be like.

Congrats! Since your LAN House is in Palo Alto do you take the Caltrain up there? What do you think of the commute? I myself am between jobs now and trying to settle on a place - currently located in SF but thinking of moving to Palo Alto.

Yes, Jade and I are taking Caltrain. CF is right next to the SF station so it works. I have yet to find out if spending 1.5 to 2 hours on the train every day will drive me crazy. :)

I started commuting on Caltrain (Mountain View <-> SF) 3 months ago. For me it's quite easy to read a book on the onward trip and work on the return trip. The only time I'm annoyed is when I board in the morning, people never stand in a line (like BART). They just approach the coach from all directions and pile inside. But it's empty enough that I can wait till everybody gets in, and I will still get a free seat/comfortable corner.

I did Santa Clara -> 22nd a couple years ago when I had a cofounder who lived in the city. I found I ended up writing a significant amount of code for that startup on the train. It actually worked out pretty well; I'd put the finishing touches on features on the way in, then discuss with my cofounder, figure out what we needed to do next, maybe talk to some potential users in the city, and write the code for new features on the way back. If you've got most of the reference docs for the frameworks you're using downloaded or memorized, or are working on stuff that's mostly thinking and not looking stuff up (I'd imagine capnp fits in that category), you can get a significant amount of work done.

That's a very good point. Working from train often means spotty internet connection. So it's hard to instantly look up something in Google. A good offline documentation helps. For older languages and IDEs like C and C++ and Visual Studio 2005 like 10 years there were CHM help files for everything. Nowadays almost everything is available almost only as web help, if there is even an official full documentation at all. Also like 10 years ago, books came with an CHM file on CD/download. Nowadays books are already outdated when the get published, things like App and Web development is moving very fast. So starting a new project on train is not a good idea, fixing bugs from error/warning messages isn't a good idea, but doing some maintenance/refactoring work is well suited for boring train rides.

Honestly it's absurd that the Caltrain doesn't have wifi yet. So many non-techy cities have it on their commuter rail lines...

It's not absurd to me anymore, I "get" how CA, and the bay area especially, operate now :) I've heard good things about Verizon, and I personally use a Project Fi phone as a hotspot. It's decent enough for hackernews/reddit/google/stackoverflow, except for the tunnels right outside the city.

Unrelated - I just read your `Owner of a LAN-party optimized house` and it seemed quite interesting, have you made any updates since to the architecture or the hardware? The last blog post was in 2011!

A couple months ago I updated the graphics cards to GeForce 1060s (from 560s) and doubled the RAM to 16 GB, which seems sufficient to run modern games at high graphics settings. Otherwise, no changes.

are you planning to have a pure-python implementation at some point ? I understand about the performance... but there are many usecases where a pure python implementation is far more practical.

I don't have specific plans regarding language support -- mostly that's been up to contributors so far, with me focusing on the C++ "reference" implementation. Hopefully, though, at CF we can find more resources. (The author of pycapnp, Jason Paryani, also joined CF with us, FWIW.)

Reply on news.ycombinator.com