talolard

I work on inference at a neocloud, but opinions are my own .

The economics and thus tools you can throw at inference change at various scales . As a “blunt” contrived example , on a gb300 the GPUs communicate super fast over nvlink, and the cards can offload kv cache to dram and then disk, “fast enough “ for these tool heavy agentic workloads.

Which come together to mean that at high enough scale and in the right scenario, we can work with wild ttls on the kv cache and still comfortably hit SLAs and tokenomics.

I’ve been vibe coding so much I didn’t even remember how useful this could be

I used DallE and Anki to teach my son to read. I'm not sure the HN crowd will be wowed by old tech, but this post leaves techs to the agents, and deals with the emotional aspects of learning I learned.

Yes, I wrote the clickbaity title with AI, but the rest of the post by hand.

I haven’t had major issues with sccs yet. The linter enforces forward references so the cycle pain we do have is with dynamic/deffered imports, and it’s usually solved by splitting a module.

If you look at the pyrefly repo (metas new type checker), there are some deep thoughts about sccs, but I didn’t fully grok them.

Not IAC, but I’ve been doing a similar trick to sequence adding type annotations to python code,

Eg take the module graph, break the SCCs in a similar manner , then take a reverese topological sort of the imports (now a dag by construction).

Hi HN,

I’m wrapping up a role where I spent a significant amount of time writing Triton kernels. It’s a fantastic tool, but the learning curve has some sharp edges. I wanted to share a few practical "notes from the field" for anyone moving beyond the very opaque docs.

You can do both ways but the latter is the more useful one. Duckdb is designed to read the data very fast and to operate on it fast. So you load a csv/json/parquet and then “create table” and Duckdb lays out the data in a way that makes it fast to read.

But you(I) wouldn’t use it like a standard db where stuff gets constantly written in, rather like a tool to effectively analyze data that’s already somewhere

Oh just took a look at the release notes, the new ability to write hive partitioned data with the partition by clause makes etl stuff much easier

I use Duckdb as a data scientist / analyst. It’s amazing for working with large data locally, because it is very fast and there is almost 0 overhead for use.

For example, I helped an Israeli ngo analyze retailer pricing data (supermarkets must publish prices every day by law). Pandas chokes on data that large, Postgres can handle it but aggregations are very slow. Duckdb is lightning fast.

The traditional alternative I’m familiar with is spark, but it’s such a hassle to setup, expensive to run and not as fast on these kinds of use cases.

I will note that familiarity with Parquet and how columnar engines work is helpful. I have gotten tremendous performance increases when storing the data in a sorted manner in a parquet file, which is ETL overhead.

Still, it’s a very powerful and convenient tool for working with large datasets locally

Postgres implements this[0] as well, and it's really wonderful. It doesn't give a human the search experience they are used to, but for the superhuman who can write regex , this becomes a very cheap way to search data at scale.

[0]https://www.postgresql.org/docs/current/pgtrgm.html

I'm a native Hebrew speaker, where we have wild word morphology, and also worked on NLP at a large bank where we trained models to anlyze Bloomberg chats (that are almost english but behave like a DSL).

In both cases, "token-free" has been the way to go for extractive tasks like entity recognition.

It's unfortunately a technique that is off the beaten path, so a lot of extra engineering effort needs to be paid down. For example, when dealing with spans but embedding the text at the byte level, one needs to account for multi-byte codepoints that can throw off the alignment between "encoded" and raw bytes.

As the article alludes, the increase in sequence length can be very expensive. In extractive tasks, I've found it effective to use much lighter models with limited (but large) context length like ByteNet[0].

Thinking out loud, as always, there is no one-tool for everything. Often the summarization/few-shot capabilities of an off-the-shelf transformer are so far ahead that it's not worth building a model from ±scratch to solve the subtlties of tokenization. Other times, you don't need the unbounded context or have a simpler task and can forgo the power of off-the-shelf models for the specificity of a token free one.

[0] Neural Machine Translation in Linear Time (https://arxiv.org/abs/1610.10099)

I used this channel and it was so frustrating it might not as well have existed . While the stripe team is always nice and the tech support is great, when I had a much less sever case of “stripe wants to destroy my business” I could get no straight answer or help from the support team

I built LightTag and eventually sold it. In my experience , at lower price points (<100 month) it’s a none issue. If you can build a self serve funnel, get people in it and convert them no one will care if your 1 person or 100.

Once you go up in price point it becomes a bigger issue, I had many deals die when they realized I was a 1 man show.

That shouldn’t stop you. If you close a 50k recurring deal once a year, that still adds up to a great income.

I came to the same conclusion on business models. I wish there was a way to know if that’s it, or is my imagination/knowledge restricted.

I keep thinking of this quote I heard “When the car was invented someone could predict parking lots but no one could predict Walmart”

It’s been a long time since I touched c++, so pardon my naïveté. I’d have assumed that optimized thread pools were a done thing. What’s new here and why was there a gap ?

I think you should clarify your goal. If you want to learn coding and specifically Django then this is a good project. If you just want a blog there are other tools that are easier like Jekyl and Hugo

My knee-jerk reaction is "cookie banners"

I used to use redux-orm which offered a similar promise. This library looks seems more mature and thought out, kudos!

My app is still alive and kicking and I ended up implementing many of the ideas here myself, in a less reusable fashion. A primary motivator was that as the use cases evolve, a storage model optimized for application specific access patterns becomes compelling.

The convenience of having properly implemented relational semantics and performance enhancements of Indices is huge, but the risk of hitting assumptions that don't fit your use case is also non-trivial

Keeping with the theme of insider experience, you might want to reach out for companies doing auditing, fraud detection etc. A domain expert who knows ML is a hot find

I remember windows 95 made a sound when it started. Maybe now that boots are fast there is no need to notify a user who may have focused on something else while waiting ?

I worked at Citi for a while and this pain sounds familiar. It once took me 6 months of hounding an MD to get approval for my app to connect to a particular database.

It sounds like you're very motivated and presumably talented, but the organizational friction doesn't really allow for your blessings to shine.

On the other hand, this is the reality for many many people and many many dollars. It's a great opportunity to both develop empathy for a kind of user you never imagined (all those people trapped in these processes for life) and to learn new ways to excel, within the context you're currently in.

When I was at Citi I learned a lot about communicating with stakeholders and fostering internal relationships which has been very useful in my career. Maybe you can find the same there ?

Could you elaborate, im interested in understanding your perspective but couldn’t follow it. In my mind , uptick in coal usage was directly caused by a decrease in wind power production. I understand that if Germany had chosen nuclear over wind, that decline would not happen and thus the usage of coal would not increase. Is that not true ?

This looks useful, I like the pricing of free/$9.99 a month.

No one asked, but it's HN so I'll say it anyway. I think it's a questionable "vc business" but a great business for 1-2 people. The road from this to an enterprise sales motion, or even a 10K/year contract is hard for me to imagine. At some point, it becomes cost effective for my org to build this functionality in house.

However, as a hobbyist /single dev / small team, $120/Year is a no brainer after the first 2-3K I spend on GPU by mistake. As you know, setting up spots when I just want to get shit done is a pain and I'll gladly pay you (a little) to make that go away.

Still no one asked but... One thing that plays to your advantage is that the current price point is something anyone in your user group can buy on their own, and there are a lot of us / enough to make a nice business out of.

Good luck!

Add a user agent to the request

See here: https://www.sec.gov/os/accessing-edgar-data

There are two answers to this. First, empirically we see that the more parameters we add the better the model performs ==> Weights continue to contribute (and aren't dead) .

Second, there is a very popular paper called "The lottery ticket hypothesis" [1] that in any network you can find subnetworks that work just as well. e.g. The parameters are redundant. This was written in 2018, which is a long time ago in big NN world, so I'm not sure how it holds up to current insanity sized models.

[1]https://arxiv.org/abs/1803.03635

I don’t know if it’s correct , but I often think of a classification model as learning the parameters of a dirchlet distribution with the final softmax layer being a sample from it

A quick summary/translation for those of us who don't speak ML.

We keep hearing about these giant models like GPT3 with 1.5 billion paramaters. Parameters are the things that change when we train a model, you can think about them as degrees of freedom. If you have a lot of parameters, theory made us believe that the model would just "overfit" the training data, e.g. memorize it. That's bad, because when new data comes in in production we'd expect the model to not be able to "generalize" to it, e.g. make accurate predictions on data it hasn't seen before, because it's just memorized training data instead of uncovering the "guiding principles" of the data so to speak.

In practice, these huge models are, in laymans terms, fucking awesome and work really well e.g. they generalize and work in production. No one understands why.

This paper is a survey or overview of what "too many paramaters" are, and all the research into why these models work even though they shouldn't.

I think your last paragraph is the sales pitch for AWS. Hiring that level of expertise doesn’t scale. Easier and cheaper to hire 10x as many “developers” and pay the AWS bill than headhunt performance gurus that understand hardware and retain them .

I think AWS Aurora is the most prominent thing that comes to mind. The key value proposition is the separation of storage from compute. That unlocks many promising features, “serverless” , a better parallelization story of OLAP queries etc

A non ml way to approach this is to use phonetic distance, e.g. qanoon and kanun sound the same so they are close.

There is an algorithm called Soundex with python implementations you can try.