25 karma · joined March 20, 2020
We did experiment with a Bradley-Terry loss function (https://hackmd.io/eOwlF7O_Q1K4hj7WZcYFiw), but we found that even better was to calculate Elo scores, do cross-query bias adjustment, and then MSE loss to predict the Elo score itself.
In our training pipeline, we had to convert the fixed point iteration to be on _ directly for numerical stability. I have a post on that here!: https://hackmd.io/x3_EkXGKRdeq-rNHo_RpZA
Bradley-Terry also very cleanly turns into a loss function that you can do gradient descent on, which will cause your model to efficiently learn Elo scores! Our calculations are at: https://hackmd.io/eOwlF7O_Q1K4hj7WZcYFiw
But, we did need to work on numeric stability!
I have our calculations here: - https://hackmd.io/@-Gjw1zWMSH6lMPRlziQFEw/B15B4Rsleg
tldr; wikipedia iterates on <e^elo>, but that can go to zero or infinity. Iterating on <elo> stays between -4 and 4 in all of our observed pairwise matrices, so it's very well-bounded.
We found that MSE after elo-adjustment worked equally well. And, MSE lets you shuffle (q, d) across the dataset which has good statistical properties (Versus contrastive, which makes you sample the same query many times within a single minibatch)
In this case "InfoNCE" isn't applicable because the reranker's output is a scalar, not a vector. So that's why we checked both bradley-terry and MSE.
It's definitely the best if not only way to get extremely high signal, and a score assignment that actually converges the more you sample.
In terms of the "F" in 2AFC, we actually have this amusing snippet from our prompt:
> Do NOT output a score of 0.0, ensure to focus on which document is superior, and provide a negative or positive float between -1.0 and 1.0.
When we were exploring the mathematical foundations, we considered ELO scoring against a "Universal Corpus" based on the natural entropy of human language (Obviously that's intractable, but sometimes this term cancels out like in the DPO proof).
But eventually we figured out a method using cross-query comparisons to assign an "ELO bias" to all document ELOs within a given query's candidate list. This normalizes it correctly such that when a candidate list is all bad, the ELOs shift low. And when the candidate list is all good, the ELOs shift high. Even when the relative ELOs are all the same.
For each document, there is a secret hidden score "s" which is the "fundamental relevance according to the LLM". Then, when we sample (q, d1, d2) from the LLM, the LLM follows the statistical property that:
- The "fundamental hidden preference" is `pref = s_{d1} - s_{d2}`, usually ranging between -4 and 4.
- The LLM will sample a normal distribution around the `pref` with stddev ~0.2, which is some "inner noise" that the LLM experiences before coming to a judgement.
- The preference will pass through the sigmoid to get a sampled_score \in [0, 1].
- There is an additional 2% noise. i.e., 0.98 * sampled_score + 0.02 * random.random()
When we use Maximum Likelihood Estimation to find the most likely predicted "hidden scores" \hat{s} associated with each document, then we go ahead and sample pairwise matrices according to `0.98 * sigmoid( \hat{s}_1 - \hat{s}_2 + N(0, 0.02) ) + Uniform(0.02)`, then we get a pairwise matrix with virtually identical statistical properties to the observed pairwise matrices.
It all depends on the use-case, sometimes you get a query that you couldn't have predicted the filter beforehand. In those cases, usually what you have to do is open up a spreadsheet and then manually categorize every document by hand. LLMs and modern AI are great ways to automate this.
A really good solution might be to have a system that computes these filters on-the-fly, but also caches them for later reuse if a query asks for that filter again.
For long documents we have a rolling window strategy. So, we cut the document into 5,000 token groupings for use in inference. There's also a 400 token overlap, and we prefer the earlier chunk for overlap tokens.
For example, if Group #0 overlaps with Group #1 at index 5,200, then we use the logprob from Group #0, because it had more context. Group #1 gets the benefit of context for indices 5,000-5,400, even though we toss out the logprobs for that range.
Details about the groupings are here: https://github.com/ZeroEntropy-AI/llama-chunk/blob/0cac10a34...
--
No need to keep track of chunks that it's already made, we just want heatmap values and then we use those heatmaps to split at the hottest character that's around our target chunk length (Or use a threshold value and binary search the threshold for our target # Chunks or average chunk size).
It scored better than LlamaIndex's recursive character text splitter and that was including some custom regex work to improve it. If you put enough effort into the regex you could probably get there, but the whole point of the agentic chunking is for it to be automatic and contextual.
I've written a lot of RAG pipelines over the last year, and one consistent pain-point is writing regex to chunk the documents correctly.
Right now, the most common chunking algorithms are:
- Split every 1000 characters
- Split on whitespace
- Recursively split on: (many newlines, then one newline, then periods, then spaces)
The best is recursive character text splitter, but regex is super brittle and when it fails to match it ends up creating huge chunks. Worse, this solution also has the overhead of needing to maintain regexes for every single filetype.
Here we propose LlamaChunk, an inference-efficient method of LLM-powered chunking. Using this method, it only requires a single LLM inference over your document in order to provide the most optimal recursive character text splitting, without needing to hope that a bunch of hard-coded rules work on your unstructured data.
We're hoping to build a community for state-of-the-art RAG research @ https://discord.com/invite/yv2hQQytne , come join!
It's not travelling... If you're staying at home...
Simple as that, it's decentralized, that's the whole point, you fundamentally can't tell people which fork to believe, people use whatever fork they want. And the people mostly wanted the fork with their money in the DAO preserved. People who wanted the unaltered chain stayed there, no big deal.
(I'm not counting the mathematicians of the 1700s that did amazing work, I'm referring to modern Europe and the US, 20th century and beyond)
It's kind of sad that a huge market for businessmen is taking an American business, and just doing the same thing in Europe or Latin America. It's, it's sad. There's uh, absolutely no foreign competition in most situations that we have to meet. The ratio is clearly wrong, and we compete with more companies than we can found so Europe should statistically be represented in the other direction. It's not even that profitable, American consumers are a gold mine they're so price insensitive. And the modern world is mostly what was invented in the roaring 20s in the US, and computing technology that was developed almost entirely in the US.
(The Japanese were trailing along in the 80s I guess, but they were mostly our China, they were just a factory line that we abandoned when China caught up and their GDP has stagnated for 20 years. They had a few great companies, and they still do. South Korea is the only actual one that goes toe-to-toe but they're too small to profit off 300 million people like we can, per capita I do think they are more entrepreneurial and inventive than us though)
There's not even VC money in Europe it's almost impossible to get funded, you need American or Chinese money which means they own most of the business anyway. Like in the US, take the money away, you can get started again. But in Europe? You better buy a plane ticket. No one will fund you. The non-tech companies are mostly incumbents. Or brands like WeWork that are much more expensive than other co-working spaces but they have the brand to hold them up.
You always have to be humble unless you're a billionaire because there's always someone vastly more powerful and rich than you are, by orders of magnitude. Your network's wealth and power as a normal distribution always ends up trailing behind you as you try to meet new people above you, but there's always the right tail end where you know a few people who could spend your net worth in a night because they want to.
https://www.youtube.com/watch?v=vrl5PFB35Ec
Look at how humble the CEO of the Maverick's has to be as he meets Mark Cuban. It's somewhere in there, I'm not gonna find the time but it's in there.
I don't know if you've ever seen the Black Mirror dystopia about social media and kissing up to people. There's a lot of that. It sucks. That's why most business events involve drinking until tispy, it loosens everything up.
https://www.nbcnews.com/news/us-news/former-boeing-manager-s...
Notably: "For the first time in my life, I’m sorry to say that I’m hesitant about putting my family on a Boeing airplane," Ed Pierson wrote to a company executive before the first tragedy.
This would never happen if the executives were engineers. Every time we, humanity, eventually gets stuck in a rut. The Romans invent the most unbelievably complex and well structured government of their time, while the gauls hunt like animals, and they conquer the world, they develop all of the technology (Well, the greeks were even better with their near idealistic Athens, but like the PhDs it never gets implemented or spreads). So the Romans do this, they win. Then...., nothing happens. It goes to shit. The Patricians murder the Plebians who protest, they become corrupt, the people have less and less control, it becomes awful. Then, dictatorship. Then, emperor. Who were the people who drafted the first setup of the roman government? They're 600 years dead, and those who replaced them made it shit. Took 1000 years to recover. The founding fathers. Compare George Washington, and his morality, to our modern politicians. No one had any reason to do anything other than create the perfect government, everyone helped, it was collaborative. Now they're cutthroat. It always goes to shit. Corporations are the new target. The founding fathers tried so hard to put every single failsafe they could, because they knew. They knew they knew. They knew those who would come after them would make it shit. While the founding fathers intentionally put failsafes - the entire article is a big ass fail safe you just read it (It just says exactly what the restrictions are - if people acted just like the founded fathers did they would never need restrictions in the first place), our modern politicians research every possible exploit to the system as if they're in infosec. Gerrymandering, oops! Jefferson forgot about that one, it took 300 years to discover the buffer overflow, but if it's discovered it's abused. Every metric becomes a target. The target of "leading the revolution" gave people who were intelligent and passionate about saving their country. The target of "becoming a politician" in 2020 is https://en.wikipedia.org/wiki/Roger_Stone I highly suggest you read just the paragraph about high school. It shows what you need to be, what the selection criterion is. The only selection criterion that works is: Coming up with something new, a new business, a new idea, a new government. After that, they hunt. They get their MBAs to overthrow you with credentials so that they can squeeze themselves in, even though they didn't found Boeing. The patricians bribe who they had to to squeeze even, even though they never came up with the original government. The politicians cheat what they must to squeeze into what the founding fathers tried to prevent. Someone has their eyes on the prize.
Mind you, the French are just as intelligent as the Italians, they're nearly identical at birth, only culture separates them. A french man who grew up in italy is an italian man. It's the system that is what separated the gauls from the romans, and it's survival of the fittest, you're the first to have an opposable thumb then you win. Whoever figures out how to create a self propagating government wins, a couple people did, got a few thousand followers, and it propagated. Business is the new self propagating system, until stability is reached. Only the new people have what it takes. It's not really too genetic either, it is in some way have 4 kids one of them will work, what of the other three? Kahn couldn't stop it, Alexander the great couldn't stop it, Romans worked for 300 years because of adoption, but it was still bleeding. Go through the history. Each half century involved more failures by the plebians and more success from the corrupt patricians, it was slow but sped up, more and more restrictions were put on the people by the incumbents. You always need fresh people. Carnegie? Vanderbilt? Compare to the modern CEO of BP...., of Boeing, stability kills. In hackernews, for people in IT, no one knows shitty bosses and executives more than they do.
(1) Yes, Uber should have trained the driver better to look at the road. By trained, I mean there should have been a sticky note on the wheel saying "PAY ATTENTION OR YOU WILL KILL SOMEONE"
(2) Yes, The driver absolutely should have to pay some penalty for this, if Uber told him that he should have been paying attention (Which they most certainly did). Watching a video while in a self-driving car is IDENTICAL to watching a video in a normal car. Modern self-driving cars are NOT fully autonomous, and they SHOULD be viewed as IDENTICAL to cruise control for all legal considerations, and thought experiments. Most of the top comments, which are blindly attacking self-driving cars, are not making this analysis.
However, (1) and (2) do not justify the lack of logic displayed by most of the top comments here. There seems to be violations of
(a) There is no logical difference between a person accidentally killing someone, and the self-driving car accidentally killing someone. Actually, because the car is already known to not be fully autonomous, this already is a case of the person accidentally killing someone. However, even if the car is fully autonomous, we MUST be considering the ODDS of an accident. None of the other comments are doing this. There is always an odds of an accident, so a specific accident means absolute bullshit. Literally nothing. This post doesn't even mean anything. What SHOULD be posted is "Self-driving cars with humans at the wheel kill X people per road-hour. Human-only cars kill Y people per road-hour". If X > Y, then yes we have a fking problem. But without that information, we literally have nothing to even think about, or process.
(b) Disabling the safety feature of the car is not a fk'ing concern, at all. Almost no cars have these features - only expensive ones do. Turning an expensive car into a normal car is NOT something you can be sued for, or is even something anyone should care about. Who is at blame for the situation is simply not dependent on this fact. I don't understand why people are discussing this aspect.
(c) Do you see the image? Maybe it's not showing it all. But as far as I can see there isn't a light there. Wtf is she doing crossing the road if there's no light there, without waiting for the cars? As a citydweller that is constantly found in the middle of streets trying to cross parts of the road that don't have stop lights, I just can't fathom ever being in her position. Maybe my city has less considerate drivers than her city, but if I tried to cross roads without a red light blocking cars, and didn't consciously give right of way to the traffic, I would die sometime this week.
(d) Many people seem to quote "self-driving system classified the pedestrian as an unknown object, then as a vehicle, and then as a bicycle" as "Oh this self-driving system is complete sh*t it's all Uber's fault". Makes no damn sense. As a side note, if you've ever worked with a neural net, especially with video as opposed to still photos, you already understand that the given sentence means nothing. That's just how they work, and there will always be milliseconds in-between frames where it reassigns the object's identification. But, anyway, this is not relevant. The self-driving part could have been completely off, or disabled. The car is SUPPOSED to be driven by the driver, and any deviation from that is the fault of either the driver, or Uber's training of the driver. Whether it's the former, or the latter, is exactly where and how suing should be directed and handled. Nothing else matters, despite most comments putting much emphasis on many other aspects of the situation.