Hacker News Reader

dbfclarkon A polynomial autoencoder beats PCA on transformer embeddings

I think your per-axis std normalization is likely doing a big pile of the work —- it’s fairly well-known that “wrong” PCA, setting sigma=Id or just taking a square root, gives better embeddings than the un-normalized version. It would be worth showing a comparison to similarly-normalized PCA I think, if it’s not too hard?

dbfclarkon If you're going to vibe code, why not do it in C?

I did a goodly chunk of vibe coding over the summer and I found that the best language for me was Rust implementations with Python bindings for interface. A few reasons:

- Everything about rust enforcing correctness catches lots of bugs

- Using a high-level API means I can easily hand-check things in a repl

- In addition to tests, I required a full “demo notebook” with any PR — I should be able to read through it and confirm that all the functionality I wanted has actually been implemented

If the philosophy is (and it should be) “loc is free”, it’s worth thinking about how we can make LLMs produce more loc to give us additional comfort with correctness. Language choice is very much a way.

dbfclarkon Phase behavior of Cacio and Pepe sauce

Another completely viable solution (other than adding extra starch) I’ve found is to sprinkle a bit of sodium citrate (the sodium salt of citric acid, a common food additive and cheap on Amazon) over the cheese before adding to the pan. This improves the melting qualities of the cheese and avoids the starch issue altogether. You’re basically using pecorino velveeta.

dbfclarkon How does cosine similarity work?

A good way to understand why cosine similarity is so common in NLP is to think in terms of a keyword search. A bag-of-words vector represents a document as a sparse vector of its word counts; counting the number of occurrences of some set of query words is the dot product of the query vector with the document vector; normalizing for length gives you cosine similarity. If you have word embedding vectors instead of discrete words, you can think of the same game, just now the “count” of a word with another word is the similarity of the word embeddings instead of a 0/1. Finally, LLMs give sentence embeddings as weighted sums of contextual word vectors, so it’s all just fuzzy word counting again.

dbfclarkon Turing's topological proof that every written alphabet is finite (2010)

Convergent sequences are always Cauchy; for metric spaces, compactness and sequential compactness are the same.

dbfclarkon GPT-4 Outperforms Humans in Pitch Deck Effectiveness

The key question here isn’t so much whether GPT-4 beats the actual human decks as much as what it had to fabricate to do so. The humans are probably constrained by things like “reality” and “what their business has done in it” while GPT-4 could make up anything it wanted. A fair comparison would be to humans given the same prompt and told to invent whatever facts they wanted.

dbfclarkon Pompeian Group Therapy

Very let down by the title -- why no ancient Roman abstract algebra?

dbfclarkon What are your company's anti-values?

Not my company, but Zocdoc has the best values I’ve ever seen exactly because they all have anti-values:

Patients First

Important, not Immediate

Learners before Masters

Together, not Alone

Progress before Perfection

Adaptable, not Comfortable

dbfclarkon Cheating on a string theory exam

Sure. Moonshine I know a lot about, relative to physics at least. Given that I now work somewhere that specializes in term embeddings, it kind of feels like I've spent half my life injecting stuff into vector spaces...

dbfclarkon Cheating on a string theory exam

Hey, I'm a mathematician, not a physicist. We tend only to get as far as "oh yeah, here's all this pretty math we care about. I hear it has some applications to physics..."

dbfclarkon Cheating on a string theory exam

Presumably the extra credit is that the binary Golay code is closely related to the Leech Lattice, and thus to the entire moonshine situation, which gets you to string theory. See:

https://en.wikipedia.org/wiki/Leech_lattice https://en.wikipedia.org/wiki/Monstrous_moonshine http://motls.blogspot.com/2015/03/umbral-moonshine-and-golay...

dbfclarkon Ask HN: What is the skillset for programmers at AI startups?

Very company dependent. At Luminoso (http://luminoso.com), we live in Python, with varying doses of Javascript for frontend, very little R (but some of us know it), Java and C++ are pluses (for working with other people's software), and we have at least some Haskell in production (mostly for preprocessing). And plenty of ops tools for deployment. We have used a few neural networks frameworks and machine learning packages; definitely the Python machine learning ecosystem is big for us but we haven't found our One True neural network framework yet. Other companies (say in image recognition, Ditto Labs for instance) are definitely more CNN-oriented, yet others are certainly doing work in R.

If you have a company that is doing genuinely interesting AI work, it's likely that they are somehow on the forefront of research, pushing existing tools to do things that they can only barely do. If you find yourself in an AI role (typically only some roles actually do AI, of course -- systems need to stay up, great UIs need to be built, etc.), I would guess that you'll need to familiarize yourself with their particular toolsets and methods rather than assuming that there are universally correct things to go learn.

dbfclarkon Wesley A. Clark, legendary computer engineer, dies at 88

Indeed -- it's struck me recently how Wes was probably among the first (if not necessarily number one, if only because of ill-definedness) people to make computers smaller. Looking back at the sixty years since he began, it's been quite something.

dbfclarkon Wesley A. Clark, legendary computer engineer, dies at 88

Quite. Minsky was actually advisor to my company, which is based in part on his notion of common sense computing. There is much to mourn recently.

dbfclarkon Wesley A. Clark, legendary computer engineer, dies at 88

He'd talk about computers for fun, at least. But try to convince him that the internet actually worked and you'd have a whole evening's conversation on your hands...

dbfclarkon Wesley A. Clark, legendary computer engineer, dies at 88

It's a reference to a story about Ramanujan, who once (remarkably) told G. H. Hardy that 1729 wasn't a boring number because it was the smallest number that could be expressed as the sum of two cubes in more than one way. Sometimes trivial but always erudite in the best way, that was Wes.

dbfclarkon Wesley A. Clark, legendary computer engineer, dies at 88

Everyone has stories, most better than me. I find my most vivid memory of experiencing him as an adult was that every time you spoke with him he'd have had some completely flabbergasting new life experience you'd never imagined. He was a guest on something like the second trip to China after Nixon, took seminars with Oppenheimer at Berkeley (he tended to call him Oppy), and may have been invited to help create an early ATM. The list goes on and on.

One Christmas, we were talking about how difficult it is to assess the accuracy of an algorithm when you only get a few tries because running it is expensive; he promptly told us about his experience with this problem, which he encountered driving plutonium nuggets across Richland, Washington. After the video from the ACM conference on personal workstations got published here, I learned for the first time he'd been fired from MIT three times for insubordination, and I never got a chance to ask him what the story there was...

Oh! And he actually made a physical turing machine good enough to do fairly sophisticated computations on. It ended up in my father's intro CS class at Princeton for many years.

And HN will of course appreciate one of his favorite one-liners:

Did you know that 49 is the lowest number that can be expressed as the square of 7 in only one way?

dbfclarkon Wesley A. Clark, legendary computer engineer, dies at 88

I'm his grandson, not his colleague, so far be it from me to say he was a great man, but he was a great grandfather and an inspiration. He taught me an enormous amount, from logic to just how hard it can be to get people to adopt new technologies -- he eschewed word processing, writing every document directly in PostScript even up to the present -- but most importantly he showed me how important it is to love what you do, and for that I owe him a ton of my happiness and career.

Fare you the hell well, Gramps.

dbfclarkon 16 Things

I couldn't possibly agree more in general, though of course I'd quibble with many of the specifics...

As to stodginess, I'd say there's a big difference between personal/small commercial lines and the big ticket enterprise-type stuff -- the underwriting process goes from something data-driven to something relationship-driven very quickly indeed. The bigger the commercial line, the more likely it's all about who throws the best yacht parties (reinsurance in particular suffers from this massively).

Crowdsourced insurance is indeed a terrible idea, though you could imagine 'web of trust' insurance that almost made sense -- say my ten thousand best friends and I know that we're all great actuarial risks, perhaps because we have some kind of information on which it's illegal to select (that we're all in the same gym, say). We could then try to write ourselves health insurance for cheap, because our plan would select only us gym members. You can sort of make it work, as long as you're prepared to make the regulators hate you.

Which is the real problem, of course -- most people buy insurance because they have to, not because they want to. Auto insurance that wouldn't pass muster with the police, or home insurance that wouldn't satisfy the bank holding your mortgage, doesn't solve the problem.

Do good problems exist? Sure. A web-based Managing General Agency, for instance, could do very well for itself, but the expressed ideas in this section are pretty terrible.

dbfclarkon Python-ftfy: Given Unicode text, make its representation possibly less broken

This is an impact of the default Unicode normalization, which is set to NFKC. This normalization is lossy for things like the ordinal indicator and trademark symbol; if you'd like to keep the ordinal indicator unchanged, use NFC normalization:

  >>> print ftfy.fix_text(u'ordinal indicator º to o in addresses.')
  ordinal indicator o to o in addresses.

  >>> print ftfy.fix_text(u'ordinal indicator º to o in addresses.',normalization='NFC')
  ordinal indicator º to o in addresses.

dbfclarkon Python-ftfy: Given Unicode text, make its representation possibly less broken

Luminoso cofounder here (we make ftfy, among other things). Our use case is fairly specific: a customer uploads text documents, often as a spreadsheet originally exported from someone else's tool, but doesn't think hard about the encodings involved, so in order to serve them well we have to fix whatever happened. Accordingly, we put the most effort into solving problems that happen for our English-centric, US-centric customer base; they're the most important problems for us (though Russian has also gotten some love, as you can see from the commit history). On top of that, making Asian language exports from other tools work correctly usually requires enough encoding-awareness to mitigate a bunch of the problem[1], so we see those problems less frequently.

That said, if you have examples where ftfy fails in any language, please submit them! We want this tool to work well, because anything that we can't fix will cause us to have egg on our faces with a customer someday...

[1]No peeking: how many format options in Excel's "Save As" dialog, excluding Excel formats, produce a document from which Unicode can be recreated accurately?

dbfclarkon SxSW – Lost That Loving Feeling for Early Stage Entrepreneurs

One meaningful point: we're early-stage and we bought a booth and we seem to be getting great value out of it, but we have a very specific target audience not mentioned by the original poster: digital marketers. At least the interactive part of SXSW is now way more a marketing conference than a tech conference and if you regard it in this light you can do very well. If you need to pitch social media managers, it's awesome, if you just want to talk about something consumer-y, not so much.

dbfclarkon Windows Phone Is Failing Because It’s Great

The provocative headline isn't really where the article is going, but it does make a good strategic point: windows phone is failing because they've decided to split the difference between their competitors' strategies, requiring enough control to allow the software to be good, but not enough to ensure that the hardware is also good, thus ensuring that their products aren't so great that consumers demand them but also requiring enough control that carriers don't like them and thus don't push them. Optimal points often live at the extremes rather than in the middle.

dbfclarkon Ten years later, insiders press a different view of Enron

For me, the key problem here is that these are very peripheral insiders indeed. Here's my company's study of the Enron corpus:

http://blog.lumino.so/2011/11/07/enron-evil-versus-football/

Just look at all that orange at the top of the first image: that's traders talking to each other about trades. Energy trading is what the company was about, not the tiny internet company that they may have spawned or the HR department. Those people don't deserve to have had their reputations ruined by the fraudulent hedge fund blowup and C-suite malfeasance, but that doesn't mean their parochial views are the correct overall view.

dbfclarkon Economics in One Lesson

The real problem Hazlett ignores is the existence of market failures in the actual world. The labor market in particular is rife with failures, in areas from search costs to market power, to the point where many labor economists model the labor market as a monopsony, i.e. a single-buyer market, which is one of the classic situations where government intervention to raise prices can increase total surplus. There's a whole discipline of labor economics that deals with the fact that the labor market is imperfect in excitingly complex ways.

In general a real market is imperfect in any case where firms make profit above their cost of capital -- i.e. most of it -- since perfect competition is supposed to drive profits to zero everywhere. The employees of a firm making profits certainly can be paid more without imperiling production, since the profits could simply be reduced.

Almost throughout, Hazlett makes exactly this (very common, for those with a political axe to grind) fallacy, supposing implicitly everywhere that no company is profitable despite the fact that we are manifestly not living in that world.

dbfclarkon The Great Cul-De-Sac Problem And How To Alleviate It.

Not necessarily -- people may want to live there in spite of the density, since not everybody is like you in turning down a 30% salary bump to avoid the big city. But it's definitely true that dense expensive places are dense and expensive because people want to live there. The regulatory problem being pointed out is that it's borderline-illegal to increase densities almost everywhere, despite increasing prices. It is all but certain that if there were no regulatory incentives for one level of density over another that there would be a lot more density than there is now.

dbfclarkon It pays well to be a (failed) CEO

Contrariwise, I'm actually in favor of CEO severance packages. A CEO can do enormous damage to a company and if the board thinks that they're doing something wrong it's very difficult to oust them on any kind of reasonable time period. Getting failures out now can easily be worth tens or even hundreds of millions to a company, so handing Mr. Apotheker a few million bucks to go away is probably well worth it.

Of course, CEO's shouldn't be paid that much in the first place (I do tend to think in almost all cases that the bad a poor CEO can do outweighs by several times the good a good CEO can), which would tend to lessen the value of severance packages overall. But if any Fortune 500 CEO deserves anything close to their current compensation package, spending only $10mm to fire one sooner rather than later is an excellent deal.

dbfclarkon Trader: I dream of another recession (and Goldman Sachs rules the world)

http://www.forbes.com/sites/emilylambert/2011/09/27/trader-o...

Is some actual reporting. He's an individual day trader, not an institutional trader of any kind. So I'd say you're pretty right.

dbfclarkon Understanding Groupon Means Understanding ACSOI

I'm afraid that doesn't fly. If a company wants to use any kind of non-GAAP financial measures in their marketing to investors they have to be disclosed and explained in the prospectus by law. That's how the prospectus works: it has to include everything you use in the marketing materials. If you think it's invalid it doesn't matter, but if the metric itself is fine it must go in the prospectus.

dbfclarkon The MassChallenge Startups Raise $90M+ & Create 500 Jobs

Just FYI, MassChallenge is already the Massachusetts partner of Startup America, the White House's initiative to promote entrepreneurship. I think they know.