Hacker News Reader

nikhilsimhaon Chronon, Airbnb's ML feature platform, is now open source

We did and gave up due to scalability limitations.

Fundamentally most of the computation needs to happen before the read request is sent.

nikhilsimhaon Chronon, Airbnb's ML feature platform, is now open source

It opens for me in incognito mode - albeit with a large popup that I had to close.

nikhilsimhaon Chronon, Airbnb's ML feature platform, is now open source

There were two main drivers -

- inability to back-test new real-time features. People were forced to log-and-wait to create training sets for months. Chronon reduces this to hours or days.

- the difficulty of creating the lambda system (batch pipeline, streaming pipeline, index, serving endpoint) for every feature group. In chronon, you simply set a flag on your feature definition to spin up the lambda system.

nikhilsimhaon Chronon, Airbnb's ML feature platform, is now open source

I agree with you - about the sentiment around the GenAI megaphone.

FWIW, Chronon does serve context within prompts to personalize LLM responses. It is also used to time-travel new prompts for evaluation.

nikhilsimhaon Chronon, Airbnb's ML feature platform, is now open source

Feathr from linkedin is the closest. But there doesn't seem to be much recent activity on the project.

nikhilsimhaon Chronon, Airbnb's ML feature platform, is now open source

By mutable data do you mean - change data coming from OLTP databases? If yes, we do this via the EntitySource api.

https://www.chronon.ai/authoring_features/Source.html#stream...

nikhilsimhaon Chronon, Airbnb's ML feature platform, is now open source

I am with you on this one.

nikhilsimhaon Chronon, Airbnb's ML feature platform, is now open source

Offline is pretty easy to get started with. It should take less than a week to set it up for new use-cases across the company. (You can begin building training-sets if offline is setup)

Online is a bit more involved - you need a month or more to test that your KV store scales against traffic coming from chronon for reads and writes.

nikhilsimhaon Chronon, Airbnb's ML feature platform, is now open source

Bighead is the model training and inference platform.

Chronon is a full re-write of zipline with 1) a different underlying algorithm for time-travel to address scalability concerns. 2) a different serde and fetching strategy to address latency concerns.

nikhilsimhaon Chronon, Airbnb's ML feature platform, is now open source

Airflow is the current implementation since it is the paved path at airbnb. But we are open to accepting contributions for other orchestrators.

Someone mentioned they wanted to add cadence support.

nikhilsimhaon Chronon, Airbnb's ML feature platform, is now open source

we call this chaining.

In the API itself - you could specify the chain links by specifying the source.

To be precise - a GroupBy(aggregation primitive) can have a Join(enrichment primitive) as a source. To rephrase, you can enrich first and then aggregate and continue this chain indefinitely.

> Graph DBs are kind of scaling poorly

That makes sense. Since you scaling these on the read side it is much much harder than pre-computing on the write side. (That is what Chronon allows you to do)

nikhilsimhaon Chronon, Airbnb's ML feature platform, is now open source

re: scalability benchmarks - we plan to publish more benchmark information against publicly available datasets in the near future.

re: human-in-the-loop workflows - do you mean labeling?

nikhilsimhaon Chronon, Airbnb's ML feature platform, is now open source

the ability generate training sets against historical inferences to back-test new features

another one is the focus on pushing as much compute to the write-side as possible (within Chronon) - specially joins and aggregations.

OLAP databases and even graph databases don't scale well to high read traffic. Even when they do, the latencies are very high.

nikhilsimhaon Chronon, Airbnb's ML feature platform, is now open source

Author. Happy to answer any questions.

nikhilsimhaon Ask HN: What Are You Learning?

I like this translation: a.co/d/gt8dwbT

nikhilsimhaon Ask HN: What Are You Learning?

Avadhuta Gita - a treatise on the non-dual nature of reality and consciousness.

The allegation is that - the boundary between the “I” and “else”, that the “I” so strongly attaches itself to, is an illusion.

The other claim is that, with meditation, one can experience this truth and there is no greater joy.

nikhilsimhaon New studies suggest millions with mild cognitive impairment go undiagnosed

Brutal

nikhilsimhaon ML Engineering Online Book

Llama 2 models were trained on slurm

nikhilsimhaon Let's talk about joins

Never heard a union be called a “vertical join” before.

nikhilsimhaon Ask HN: Good books on philosophy of engineering?

cannot recommend sicp enough: https://mitp-content-server.mit.edu/books/content/sectbyfn/b...

uses scheme to illustrate how to think about computation abstractly.

nikhilsimhaon Japan's 18-year-olds at record-low 1.06 million

/s ?

nikhilsimhaon Nuclear power gets new push in U.S., winning converts (2022)

France?

nikhilsimhaon 37, the median value for the second prime factor of an integer

This is one of the best articles I have read in a long long time!

nikhilsimhaon Apple might have to pay that €13B EU tax bill after all

/s in the end? (genuine question)

nikhilsimhaon Learning Nim: Write an Interpreter [video]

Very well done!

nikhilsimhaon How to Reduce Memory Requirements with Product Quantization

And they worry about LLMs hallucinating

nikhilsimhaon MTIA v1: Meta’s first-generation AI inference accelerator

I hope they take comfort in the fact that this is open-source.

nikhilsimhaon Our climate may already have reached an irreversible tipping point

You could vote for the candidates (senators, state and national, governors and presidents) that push for nuclear, building-heating-tech regulations, carbon taxes on food production and tell your friends and family about it.

If you have the means to, you can even donate.

nikhilsimhaon Longevity study across 5 species found a new pathway to reverse aging

It does get easier with repetition (mentally and socially).

I do a water fast on thursdays (~36 hrs). I find that to be much easier than intermittent fasting mentally somehow.