yunyu

Prevents infosys/wipro slop from overwhelming the system, and filters down the incoming roles to only those that can't be filled by a US citizen (i.e. specialist technical jobs, top engineers commanding $500k/yr)

Never understood this take. Just because it doesn’t address the systemic inequities in society doesn’t mean it’s not useful.

You're absolutely right! This isn't just X, it's Y...

This is just advertorial marketing slop?

We solved this at Ramp on the expenses/AP side with an agentic RAG implementation and a custom embedding model, backed by D&B/Google/user-submitted corrections.

If curious, details here:

https://engineering.ramp.com/post/transaction-embeddings

https://engineering.ramp.com/post/fixing-merchant-classifica...

To be clear, this is what we ended up doing

Hey all, member of the benchmark team here! The goal for this project was to see how LLMs well could do bookkeeping without an overly opinionated scaffold. We gave them access to processed transaction records and code execution tools, but it was up to them to choose exactly how to use those.

Claude and Grok 4 did reasonably well (within CPA baselines) for the first few months, but tended to degrade as more data came in. Interestingly, the failures aren’t exclusively a context length problem, as we reset the context monthly (with past decisions, accruals/deferrals, and comments available via tool calls) and the types of errors appear to be more reward hacking vs pure hallucinations.

Accounting is very interesting in an RL-first world as it is pretty easy to develop intermediate rewards for training models. We are pretty sure that we can juice the performance more with a far more rigid scaffold, but that’s less relevant from a capabilities research perspective. We’re pushing down this research direction and will see how it goes.

Let us know if you have any questions!

"Your fancy AI scaffolds will be washed away by scale." - Noam Brown

A certain level of errors is tolerable/inevitable. But the accountants need to be able to correct for them once they build up

No comment, the good news is that accounting and taxes are verifiable - so in principle it is possible to RL models to do them correctly

This was true up until they started training them using Reinforcement Learning from Verifier Feedback (started with O1). By sticking a calculator in the training loop, they seem to have gotten out of the arithmetic error regime. That said, the ChatGPT default is 4o which is still susceptible to these issues.

LLMs are on the verge of replacing data scientists and investment bankers. But can they perform simple accounting tasks for a real business?

We built AccountingBench, a test where LLMs must "close the books" for a real SaaS business using 1 year of Stripe/Ramp/Rippling/Mercury data.

Claude 4 and Grok 4 start strong - within 1% of human CPA baselines in month 1.

But as time progresses, all models inevitably accumulate compounding errors and exhibit erratic behavior, causing significant deviations.

That said, the early accuracy here is promising. With targeted post-training, models may be able to replace humans for this kind of work.

Maybe put the warning below the UI, so it doesn't cause the layout to change?

If you are a gamer, you are no longer NVIDIA's most important customer.

4o mini is a weak, old model. I’m pretty sure even regular o1 (non-pro) would be able to crack a simple vuln

On the flip side, this is extremely fast and works pretty well on my M1 Pro Macbook running Chrome. The only downside is that it's a little blurry (as it's rendering at 1x pixel scale).

Score of 6. Started and sold a YC startup

Congrats on launching – we just spent quite a bit of time replicating/transforming our primary database into clickhouse for OLAP use cases, and it would have been way easier if there were a postgres-native solution. Hoping the managed hosting providers catch on

The above is a common anthropocentric take that has been repeatedly disproven by the last decade of deep learning research: http://www.incompleteideas.net/IncIdeas/BitterLesson.html

Understanding audio with inputs in the frequency domain isn’t required for understanding frequencies in audio.

A large enough system with sufficient training data would definitely be able to come up with a Fourier transform (or something resembling one), if encoding it helped the loss go down.

> In computer vision, there has been a similar pattern. Early methods conceived of vision as searching for edges, or generalized cylinders, or in terms of SIFT features. But today all this is discarded. Modern deep-learning neural networks use only the notions of convolution and certain kinds of invariances, and perform much better.

Today’s diffusion models learn representations from raw pixels, without even the concept of convolutions.

Ditto for language - as long as the architecture is 1) capable of modeling long range dependencies and 2) can be scaled reasonably, whether you pass in tokens, individual characters, or raw ASCII bytes is irrelevant. Character based models perform just as well (or better than) token/word level models at a given parameter count/training corpus size - the main reason they aren’t common (yet) is due to memory limitations, not anything fundamental.

For further reading, I’d recommend literature on transformer circuits for learning arithmetic without axioms: https://www.lesswrong.com/posts/CJsxd8ofLjGFxkmAP/explaining...

I agree with your point at the highest (pretrained model architect) level, but tokenization/encoding things into the frequency domain are decisions that typically aren’t made (or thought of) by the model consumer. They’re also not strictly theoretically necessary and are artifacts of current compute limitations. Btw E5 != E5 Mistral, the latter achieves SOTA performance without any labeled data - all you need is a prompt to generate synthetic data for your particular similarity metric.

> Unlike existing methods that often depend on multi-stage intermediate pre-training with billions of weakly-supervised text pairs, followed by fine-tuning with a few labeled datasets, our method does not require building complex training pipelines or relying on manually collected datasets… We leverage proprietary LLMs to generate diverse synthetic data for hundreds of thousands of text embedding tasks across nearly 100 languages.

It’s true that ultimately there’s a judgement call (what does “distance” mean?), but I think the original post far overcomplicates what’s standard practice today.

Vector databases are oriented around search and retrieval. The usual approach of generating vectors is to fine tune a large pretrained model and extract the inner representations. Because the dataset contains successful queries and retrieval results, all you need to do is optimize a loss function (usually contrastively or approximated versions of common ranking functions) using raw inputs on the similarity objective supported by the vector database. For common modalities like tabular/text/image/audio data, there is basically no human judgement involved in feature selection - just apply attention.

Note: current state of the art text to vector models like E5-Mistral (note: very different from original E5) don’t even require human curation in the dataset

I got SMILE instead of LASIK for that exact reason (avoiding dry eye), and have perfect vision with moist eyes. Highly recommend avoiding LASIK if you qualify for newer procedures.

Ryan literally just made the previous CEO leave?

Jsonformer - https://github.com/1rgs/jsonformer-claude, https://github.com/1rgs/jsonformer

Extracting data from raw unstructured text/images and converting them into formatted fields

What's the plan/timeline for offering cosine similarity support, given that most OSS embedding models are fine tuned on a contrastive cosine distance objective?

You simply sample tokens starting with the allowed characters and truncate if needed. It’s pretty efficient, there’s an implementation here: https://github.com/1rgs/jsonformer

Most legitimate CS programs like Stanford or CMU CS are not ABET accredited.

With ever growing context windows, you can upload your entire codebase (or at least the public API) in the prompt and have the model reason off of that.

It turns out that accurately predicting text using a model that's a couple orders of magnitude smaller than the inputs requires understanding what the words mean.