Of Money, Responsibility, and Pride | Hacker News Reader

Of Money, Responsibility, and Pride

veridicalsystems.com

112 pointssilenteh39 comments

This is a mistake. The Apache foundation doesn't sell $250/hour consulting gigs for its primary source of revenue. Neither does the Linux Foundation, the SQLite Consortium, or other massive, mission-critical open source products.

This is the wrong funding model. It keeps money in OpenSSL developer's pockets, but there is no financial incentive for any OpenSSL developer to work on foundational improvements to OpenSSL. He said himself: there is over $100,000 in open contracts for competent developers to work on non-foundational improvements to the project. If you are an enterprising developer with good C skills and a knack for crypto projects and you apply to work for the OpenSSL foundation, are you going to start servicing that $100,000 pool of contracts or are you going to pretend that money doesn't exist and live on ramen?

If nearly all of OpenSSL's revenue comes from clients that want OpenSSL to meet their particular needs, then none of that money is going to developers to strengthen OpenSSL's foundation. This is why OpenSSL looks like a hodgepodge of hacks upon hacks in order to accomplish narrow goals with limited impact testing. It should be no surprise to anyone else: clients are literally paying OpenSSL developers for this, and nothing else.

Who is paying OpenSSL for developers to clean up the code base and remove ancient #IFDEFs? Who is paying OpenSSL for developers to analyze code paths and do case analysis? Who is paying OpenSSL for developers to write unit tests or even have a test harness at all?

No one will pay an hourly rate to accomplish these tasks. Google is not going to pay by the hour for a developer to stare at a function until they grok it; they want a feature. Joe Company will not pay for developers to write unit tests, they want OpenSSL to handle $QUIRK from a vendor's system, or to know how to make their code handle it.

This model needs to go away. Competent OpenSSL developers time is too valuable to waste on client asks. Their project is too important, and as long as the money is flowing only for novel features and not structural improvement, then that money will dictate that only new features are developed.

This is one of the better comments I have seen on OpenSSL in the past week. Well said.

"This is why OpenSSL looks like a hodgepodge of hacks upon hacks in order to accomplish narrow goals with limited impact testing."

It doesn't just look like a hodgepodege of accumulated hacks, it is a hodgepodge of accumulated hacks. :)

"It should be no surprise to anyone else: clients are literally paying OpenSSL developers for this, and nothing else."

One could say this with respect to many popular open source projects, including ones with corporate sponsorship. The complexity just keeps building over time and there is no such thing as "finished, accepting bug fixes only".

"Who is paying OpenSSL for developers to clean up the code base and remove ancient #IFDEFs? Who is paying OpenSSL for developers to analyze code paths and do case analysis? Who is paying OpenSSL for developers to write unit tests or even have a test harness at all?"

Those are rhetorical questions. We know the answers. Alas, when the people who pay for (open source) software and consulting pay to have "features" removed instead of added, "pigs will fly".

Doug McIllroy is quoted as saying, "The hero is the negative coder".

(Just in case this need explanation: Prof. McIllroy is the mind behind UNIX pipes and one of computer science's most prominant contributors. "Negative coder" means someone who removes code instead of constantly adding, or "committing", new code.)

We could really use some more heros. And as we switch away from OpenSSL there will be a lot of links to libssl to remove.

Meanwhile some people have been writing and testing small, auditable and usable open source crypto, more or less for "free".

http://tweetnacl.cr.yp.to

My guess (and hope) is that pathological requests for "features" to be added would be met with heavy scrutiny. The authors already have day jobs in academia.

> Meanwhile some people have been writing and testing small, auditable and usable open source crypto, more or less for "free".

With all due respect that is complete bullshit. I do not care that you put quotes around free. Writing "free" will never be considered to include sums in the hundreds of thousands of dollars. More importantly blatant lies like this muddy the debate and set outrageous expectations. The Nacl project gives the following description of funding:

  NaCl was initiated by the CACE (Computer Aided Cryptography Engineering)
  project funded by the European Commission\'s Seventh Framework Programme
  (FP7), contract number ICT-  2008-216499. CACE activities were organized
  into several  Work Packages (WPs). NaCl was  the main task of  CACE WP2,
  \"Accelerating Secure  Networking,\" led  by Tanja Lange  (at Technische
  Universiteit Eindhoven)  and Daniel  J. Bernstein (at the  University of
  Illinois at  Chicago, currently visiting Eindhoven). CACE  nished at the
  end of 2010 but NaCl is a continuing project.

  ...Many  of  the  algorithms  used  in  NaCl  were  developed as part of
  Daniel  J. Bernstein\'s   High-Speed  Cryptography  project   funded  by 
  the  U.S. National  Science  Foundation, grant  number  ITR-0716498.

I found the funding information for ITR-0716498. djb is listed as the PI for the project.[^1] I could only find the high level funding of ICT-2008-216499.[^2] (wtf EU?) CACE WP2 is only one component of the project. I would love it if someone with better knowledge of EU funding can find the funding for the WP2 line item. The figures are:

  NSF ITR-0716498 funding: (USD)     400,000.00
  EU  2008-216499 funding: (EUR)   4,733,078.00 ***NEED WP2 line item***

The tweetnacl implementation lists two more funding sources. As above it was easy to locate the NSF funding but I totally struck out for the nwo funding:

  NSF 1018836 funding: (USD)        $436,203.00[^3] 
  NWO grant 639.073.005 funding:    ???????????

Don't get me wrong, I have a lot of respect for djb and I think he and his coworkers deserve every fractional euro/dollar of funding that they received but they did not work for free. Most importantly they should not be expected to work for free.

[^1]: http://www.nsf.gov/awardsearch/showAward?AWD_ID=0716498

[^2]: http://cordis.europa.eu/projects/rcn/85344_en.html

[^3]: http://www.nsf.gov/awardsearch/showAward?AWD_ID=1018836

NB: This is the nwo funding site: http://www.nwo.nl/en/funding I think the english version may have a reduced set of features. I can not find the this grant information on the site.

Wow. I guess "more or less" was not strong enough wording for you?

The point is that using something like NaCl costs you, the developer/user, nothing more than if you are using OpenSSL.

Do you agree?

No, "more or less for free" is not close to hundreds of thousands of dollars plus whatever funds came from the EU and NWO.

I have to say I am confused about your reply in the first sentence you seem to acknowledge that the wordingwas related to the cost of "writing and testing" crypto software. However in the second sentence you seem to indicate that your thesis was about the switching costs users face. Which is it? You did not say I get to use nacl "more or less for free" you said that "people have been writing and testing small, auditable and usable open source crypto, more or less for 'free'." That quote seems to be about the cost of creation not the switching costs.

Do you think djb et al produced nacl "more or less for free?"

I think you misunderstood what I meant.

I mentioned "free" only to point out that there is no financial cost to switching to it. I guess I did not type the sentence with enough care; words are missing. My apologies.

I imagine people would be willing (and are accustomed) to paying for software of similar quality.

But I'm also wondering why this bothered you so much.

Does it make a difference that grants were received?

Is the funding not transparent enough?

The blog article on OpenSSL mentions payments for consulting and "features" to be added to OpenSSL.

Should I be concerned about what those features are, and who is paying for them? Are you concerned?

I'm just nterested in cleaner code than OpenSSL's. NaCl looks cleaner to me.

Maybe I'm wrong. But I'd rather be compiling programs that use libnacl or some other simpler alternative than ones that use libssl.

We all have to make decisions about what software we choose to use, even if we are not cryptographers.

I see nothing wrong with discussing alternatives to OpenSSL. This bug has been a real PITA.

  > I mentioned "free" only to point out that there is no financial cost
  > to switching to it. I guess I did not type the sentence with enough
  > care; words are missing. My apologies.

It speaks highly of your character that you say this to the jerk on the internet said you were full of shit.

  > But I'm also wondering why this bothered you so much.

Because crypto is important. A lot of harmful attitudes/mindsets are reinforced if people think NaCl was created in the authors spare time and did not require funding:

- "Why should I donate to GnuPG/OpenSSL/Tor/Mozilla(NSS)? Those NaCl devs wrote NaCl for free."

- "How hard could it be to implement a crypto library? Nacl was a side project. The Nacl devs 'have day jobs in academia' and created nacl in their spare time. They did it for free, so they obviously didn't need to spend money on testing environment, research material or hire/consult experts. On the other hand look at SelfiesMadeEa.sy they raised serious cash and had to quit their jobs because they tackle hard problems."

- "Obama and the rest of gubmint are taxing me to death. Government should be pay for the military and maybe some roads; not waste money on liberal academics in ivory towers, maplethorpe and those pinkos from NEA or some stupid robot/telescope that cant do metric conversions."

- "OMG NSA is evil. USA does nothing but invade countries and privacy."

  > Does it make a difference that grants were received?

No it does not make a (negative/harmful) difference that grants were received. I think it is a shining example of modern civil society; you have the US, NL and the EU teaming up to fund strong crypto by top notch folks from a number of countries. Governments should fund research, applied and basic, and they should be encouraged to fund more of it.

Somewhat tangential: Knowledge of the grants also seeks to eliminate the idiocy in the latter two examples above. People need to be reminded that big government is not always an evil force, governments can be a force for good. I do not know if you saw my other comment about tor funding but tor had revenue of \$2+ million in 2012 and 60% came from US government. I don't know where you are from but I bet you have met a simple minded moron wearing a tea party costume or trendy European threads that will not stop complaining about the evil Obama surveillance administration. Blow their minds and ask them to wrap their heads around the:

- $800k from DoD for "Basic and Applied Research and Development in Areas Relating to the Navy Command, Control, Communications, Computers, Intelligence, Surveillance, and Reconnaissance"

- $350k from State for "Programs to Support Democracy, Human Rights and Labor" and "New America Foundation: International Programs to Support Democracy, Human Rights"

  > Is the funding not transparent enough?

If this is in regards to the lack of numbers from NWO or the EU I am sure that I am at fault. (I also think one of djb's EU grant numbers might have a digit transposed) I imagine that the dutch version of nwo.nl is easier to use.

  > The blog article on OpenSSL mentions payments for consulting and
  > "features" to be added to OpenSSL.

  > Should I be concerned about what those features are, and who is paying
  > for them? Are you concerned?

I think we should be concerned that OSF is not doing a better job highlighting sponsors and attracting new ones. It should be easier for someone with check writing authority at big.corp.com to stumble across the sponsors information and think to themselves "hey, we should drop some petty cash on these folks. We use the product and I bet the marketing folks would appreciate the bump in visibility for a fraction of the cost of our latest failed social network branding efforts." If I was OSF I would look at the \$2 million tor brought in and ask myself "maybe we could do a better job of sponsor outreach? Tor is important to these people that wrote checks and tor uses libssl-dev, I wonder if there is an opportunity?"

The budget for NWO 639.073.005 is stated to be €1,500,00.00 at http://www.nwo.nl/onderzoek-en-resultaten/onderzoeksprojecte... .

Dank u.

> "This is why OpenSSL looks like a hodgepodge of hacks upon hacks in order to accomplish narrow goals with limited impact testing."

Can you point out specific examples that you view as hacks upon hacks?

Maybe I've spent too many years in the code base, but I've also seen worse.

OpenSSL does a lot. Maybe smaller modules would be better and more testing certainly. Organizations using it should also be contributing back more.

Hacks upon hacks seems like a stretch to me.

As a side note, the NaCL library you mention does only a fraction of the things OpenSSL does. OpenSSL could certainly stand to be broken into smaller components, but trying to compare it with a very small library that does mostly primitive operations is...an improper comparison.

I like Dan's work and have used in in projects, I just think your comparison and analysis are quite off base.

It takes 500 extra lines of C to write a secure channel abstraction with a server and client on top of NaCl. This is the curvecp implementation.

And something capable of doing an SSL/TLS connection with cipher sweet negotiation and client certificate validation?

Maybe throw in hardware token/PKCS11 support?

You are entitled to your opinion and your preferences.

As I am to mine.

From the tweetnacl.cr.yp.to paper:

"OpenSSL is the space shuttle of crypto libraries. It will get you to space, provided you have a team of people to push the ten thousand buttons required to do so. NaCL is more like an elevator -- you just press a button and it takes you there. No frills or options.

I like elevators." - Matthew D. Green, 2012

Yes, it is improper to compare a space shuttle to an elevator.

It's also absurd to use a space shuttle when all you need is an elevator.

Use whatever you want. Not everyone's needs are the same.

I like small components that are independent. The OpenSSL binary is feature for feature one fo the most complex I have ever used.

I prefer simplicity. That's just me.

Not for everybody. But some might desire it.

You have my sincere apologies for daring to mention an OpenSSL alternative.

The fact that this NaCl is so small and limited is the whole point.

I guess that point was missed.

I think you should reread what I said -- I think it needs to be componentized, because OpenSSL does a lot. Plus has a bunch of utilities to do things.

Comparing it to a library that is mostly crypto primitives is not a fair comparison.

Also - I'm still curious of examples of "hacks upon hacks" for my own curiosity. I've been using OpenSSL in a number of projects for 15+ years, so maybe I am used to certain things.

Reminds me on an article about how short-term, market-driven university research is killing innovation in abstract fields with no apparent financial ROI (e.g. mathematics, physics, chemistry and biology).

If research was market-driven in 1859 Riemann might not had time to play with zeta functions because at the time no one knew what to do with them. Same for Bayes and so many others. It's not that those man had a better environment around them than current researchers, it's mostly that we're getting results slower than we might as a human kind.

> This is the wrong funding model.

It's only the wrong funding model if there is a superior alternative. It's a non-ideal funding model, but as is the main point of the article, they are and have been actively looking for alternative sources of funding without success. I hope your gripe is not directed at OpenSSL but at those who could be supporting it financially.

It's true that their funding model is imperfect, but then other funding models are imperfect in different ways, and their model should be perfectly workable: spend six months of the year working on contract jobs and the other six months making general improvements to the code.

The reason it's not working is that their rates are far too low. This is, for whatever reason, a classic mistake of technical people. If you're turning down work for lack of time, you need to increase your rates until this stops happening.

They say they have 100 pending contracts... I believe their funding strategy could be fixed by hiring people to do the contract hours for a salary (those don't need to be OpenSSL core hackers, just good enough with the core of OpenSSL to help customers and provide viable work when a modification is requested). With the money you earn you also pay the core team members to just work at OpenSSL with the right priorities: this is where you improve security, do refactoring, and trow away #ifdefs.

One approach, if you're coordinating development through the foundation, might be for the foundation to require some percentage of the funds go to foundational improvements.

Could someone involved with the OpenSSL Foundation and the OpenSSL project maybe pitch in with a quick description of how the project is managed?

* Who owns which subsystems?

* Is there a board of governors or a BDFL or something like that effectively overseeing the whole project?

* What is the process for screening commits from people new to the project?

This whole post seems to be tinged with a bit of defensiveness on behalf of the most active committers to the project. But it wasn't the active committers who introduced this most recent bug.

The tor project is a great example of how open source software (OSS) projects can work with sponsors. Trying to find more information on Qualys and PSW Groups sponsorship of openssl is a nightmare compared to tor project sponsors.[^1][^2] Without the tor project's emphasis on transparency and professionalism I doubt they could post numbers like this:

  Since meeting the revenue milestones of $1,253,241 in 2009,
  $1,574,119 in 2010 and $1,681,101 in 2011, Tor has reached new
  heights in 2012 with over $2 million in revenue (unaudited).[^3]

My comment should not be read as a criticism of OpenSSL, it should be interpreted as cause for optimism. The tor project has demonstrated that OSS projects can get sponsored to solve complicated security problems that are difficult to explain to the general public.

[^1]: List of sponsors/projects: https://trac.torproject.org/projects/tor/wiki/org/sponsors

[^2]: Example monthly report for SponsorF's project: https://lists.torproject.org/pipermail/tor-reports/2014-Marc...

[^3]: Tor Project Annual Report 2012, pg 8, https://www.torproject.org/about/findoc/2012-TorProject-Annu...

It's well and fine that Stephen lives very cheaply, but all of this is an attempt to distract from the OpenSSL project's very real issues by wearing a cilice then bitching about it.

The fundamental facts are these: openssl contains a large quantity of code that, if I where to check into my company's repo, I would have at best a rough conversation with the cto and at worst I'd get fired. Plus a lack of good tests. These combine to create more than hypothetical problems; we've seen some severe security holes and there's almost certainly more to come.

The question that should be discussed is if openssl is, ala sendmail, unsuitable for purpose and, if so, what should it be replaced with.

Google Cache version (site is 404ing for me)

http://webcache.googleusercontent.com/search?q=cache:http://...

OpenBSD has had two holes in a heck of a long time. By contrast OpenSSL has had a remote execute in 2010, and another 4 in 2002, and is regularly patching DOS's resulting from memory corruption that turns out not to be exploitable.

It is 453,000 or so lines, more than five times the size of xv6. It is ten times as big as PolarSSL. Documentation and internal structuring is wildly inconsistent. Features that make static analysis annoying are widely used. The API is far too low level.

Do you believe this is acceptable in a security library? Do believe that aspiring to the security of qmail or OpenSSH is a reasonable goal, even at the cost of features? Why should I use OpenSSL for TLS termination when formally validated alternatives exist?

> Why should I use OpenSSL for TLS termination when formally validated alternatives exist?

Oh please do share! (spoiler: alternatives which enable side channels because the pretty compiled-optimized-code (that is generated from source code that itself may feature immutable data etc. with no side effects) is ripe with leaks via cpu branching, caching, etc etc do not count.) This is not a spiteful rhetorical inquiry, by the way!

Those side-channel attacks are largely theoretical. OpenSSL and RSA in general are vulnerable to side channel attacks, because many of the fundamental operations are not constant-time. That can change eventually, but RSA is difficult to write in a constant-time manner. OpenSSL certainly has had its fair of lab-demonstrated side-channel attacks, but I don't think anyone has been able to demonstrate their use in virtualized hosting environments against arbitrary other tenants.

Side channel attacks once you've already got code running on the same operating system as the target are much easier. But if you can get arbitrary code to execute on the same machine running OpenSSL, you probably already have their keys.

So, I think the criticism to OpenSSL is valid. Why use it when it seems there are less bad alternatives? A lot of it comes down to network effects, inertia, and licensing. That's a much better answer than side-channel attack surface area, which all RSA implementations share :)

Nope: you can exploit the timing attacks from across a network link in AES. DJB did this long, long ago. Furthermore, I believe OpenSSL has constant-time RSA, but you can always use ECDSA for which constant time implementations in C exist. (I wrote one, but I still need to switch the hash function to SHA256 and add the unnecessary encoding steps to the output, so consider it a PoC).

Are you sure? Searching for "side channel OpenSSL" reveals a majority of the attacks are against ECDSA. Of course, searches aren't the best measure of vulnerability, it's just an indication of popularity.

http://cr.yp.to/antiforgery/cachetiming-20050414.pdf ECDSA is nice because you can use partial nonce recovery+lattice reduction. Furthermore recent Intel chips have AES instructions.

> Those side-channel attacks are largely theoretical.

Before you write secure software, you have to consider who your adversaries are, and what they are capable of doing. Side-channel attacks over the network are definitely practical [1] [2] [3]. If you're making even a basic SaaS product, you should assume your adversaries can carry out the above. You should assume that by now, even lowly script kiddies have side-channel exploits in his arsenal.

[1] Relevant paper (2010) http://www.informatics.indiana.edu/xw7/papers/WebAppSideChan...

[2] Ed Felton's analysis on [1]

[3] Bruce Schneier's analysis on [1]

PolarSSL has been validated with FRAMA-C by a company. Unfortunately you have to pay to see the results, and they use blinding for bignum arithmetic, not constant-time. (But they do have a decent curve implementation). http://trust-in-soft.com/news/

To remove side channels and preserve validation, you need to be slightly clever. You validate that a C implementation of one of the functions does the same thing as a function in the validated package, then swap them in the build output. This can be done for miTLS. Your validation of the C part is much more limited in scope than the whole thing.

> To remove side channels and preserve validation, you need to be slightly clever. You validate that a C implementation of one of the functions does the same thing as a function in the validated package, then swap them in the build output. This can be done for miTLS.

Aha, this is interesting! Thanks for the link.

I believe the formally validated alternatives have licenses that don't make them acceptable for a large part of the possible users.

SSL is important. Proprietary users should buy proprietary licenses

I am not defending OpenSSL but I am not sure your comparisons are very informative and quite frankly some of your sloccounts seem to wander far away from fact.

  > OpenBSD has had two holes in a heck of a long time

Two remote holes in the default install. The default install is configured in such a way as to minimize the attack surface area. Do you know what services are configured to respond to public internet traffic in the default install? OpenSSL on the other hand essentially is always interacting with public internet traffic. Do you use OpenBSD on your desktop and laptop?

  > 453,000 or so lines, more than five times the size of xv6

Are you referring to Xv6, "the simple Unix-like teaching operating system" a project designed for education and not commercial service offerings? How does this comparison of the size of xv6/OpenSSL inform the debate? The Linux kernel has 12 million lines of code. What should I conclude from this number compared to OpenSSL? Do you think some of PolarSSL's failures with frankencerts can be explained by how small the code base is? Please see the addendum for more sloccount discussion.

  > Do believe that aspiring to the security...even at the cost of features?

I don't mean to be difficult but I have no idea what you are asking here. Which features are you referring to? Without some specificity of "cost of features" this seems to be more about showmanship and rhetorical style than fostering meaningful debate. Do you run OpenBSD on your desktop/laptop?

  > Why should I use OpenSSL for TLS termination when formally validated alternatives exist?

What are these alternatives? I did not realize there were multiple formally validated alternatives to OpenSSL. Maybe we have different opinions about "alternatives" versus "implementations."

SLOCCOUNT Addendum:

I did not want to inject a pile of sloccount data in the main body of my comment but I do have some questions about your counts. After a little research it seems that your numbers might be somewhat hand wavy which is unfortunate because you gave no indication of this in your comment.

What versions of OpenSSL and PolarSSL are you referring to? My count for OpenSSL 1.0.1g puts it at 6.5 times the size of PolarSSL and not the 10 times that you stated in your comment. I could not connect to OpenSSL.org so I do not know if they have released a version since April 7th but I must say I am surprised that that the OpenSSL dev team added 91,344 lines (25% increase) in the last seven days. And even if they did that still puts them 90,000 lines short of "ten times the size of PolarSSL."

My sloccount data:

  # OpenSSL 1.0.1g:
     Totals grouped by language (dominant language first):
     ansic:       273820 (75.71%)
     perl:         69192 (19.13%)
     asm:          11015 (3.05%) 
     cpp:           4367 (1.21%)
     sh:            3238 (0.90%)
     lisp:            24 (0.01%)
     Total Physical Source Lines of Code (SLOC) = 361,656

  # PolarSSL 1.3.6-gpl
     Totals grouped by language (dominant language first):
     ansic:        52212 (94.71%)
     sh:            2165 (3.93%)
     perl:           748 (1.36%)
     tcl:              4 (0.01%)
     Total Physical Source Lines of Code (SLOC) = 55,129

  # Linux kernel 3.14
     Totals grouped by language (dominant language first):
     ansic:     11828087 (97.00%)
     asm:         275047 (2.26%)
     !!!TRIMMED counts < 0.5%!!!
     Total Physical Source Lines of Code (SLOC) = 12,193,312

  # Xv6
     Totals grouped by language (dominant language first):
     ansic:         7022 (87.81%)
     sh:             481 (6.01%)
     asm:            301 (3.76%)
     perl:           189 (2.36%)
     lisp:             4 (0.05%)
     Total Physical Source Lines of Code (SLOC) = 7,997

You are correct: I meant 10x the size of xv6 and 5 times that of PolarSSL. I used wc, so comments and blanks count. xv6 implements a multiuser operating system. This is not an easy undertaking by any means, and far more complex than TLS in terms of conditions to satisfy.

As for the rest of your comment, I have used OpenBSD on the desktop. I have never found the lack of a webserver running, or anything but sshd to be a hindrance to making a machine on which with a browser I can read papers, bank, listen to music, and program. sshd is of course attackable by anyone: good luck getting in though. Other OS's have followed OpenBSD, as do most hardening guides.

One of the features OpenSSL has is support for ciphers that long ago have been disabled, together with out-of-date TLS versions such as SSLv2(!). TLS is a specification, with multiple implementations. The specification is of poor quality, the protocol worse, but OpenSSL is in a class by itself, including implementations of national ciphers no one uses.

While Frankencerts were an issue, there was no realistic way to turn the two incorrect "OKAY" and the one "NOT OKAY" result of PolarSSL into an attack. You would need a CA to issue a cert with a start time in the future. It also does not take an enormous amount of code to fix this issue.

Reading about Dr. Stephen Henson reminded me of the article written about Tarn Adams, the creator of Dwarf Fortress.

http://www.nytimes.com/2011/07/24/magazine/the-brilliance-of...

Not hard. Just change the license and require commercial use to be paid

Reply on news.ycombinator.com