prats226

Sorry I meant instead of having one big piece of metal which would have restriction in terms of length, having 2 pieces of lenses seperated by just vaccum? You would have lesser material, easier launch and higher magnification because of larger focal length etc?

Why aren't telescopes built this way?

An issue would be as soon as you make questions public, even by letting hosted LLMs predict on them, they are tainted. You can't use them anymore. So would it be a one time test dataset?

For LLMs, programming languages are basically additional languages that we speak. So how it handles low-resource programming languages is same as how it handles speaking languages with less contribution in training data?

DSL's would be even harder for LLM's to get right in that case compared to the low-resource language itself

I think the choice mainly stems from how you want to use the output. If the output is going to get fed to another LLM, then you want to select markup language where 1) the grammer would not cause too many issues with tokenization 2) which LLM has seen a lot in past 3) generates minimal number of tokens. I think markdown fits it much better compared to other markup languages.

If goal is to parse this output programmatically, then I agree a more structured markup language is better choice.

Given that input is image and not raw pdf, its not completely unexpected

I think author has taken a very long term view of what's happened so far without getting too politically specific.

I found it very informative to get a sense of how many abstractions are there in the system. So I atleast know where to deep dive on.

Reminded my of this article: https://ampcode.com/how-to-build-an-agent

Some of the things are just more natural in python being a dynamic language. Eg decorator to quickly convert methods into tool calls, iterating over tool functions to create list of tools, packages to quickly convert them into json schema etc.

Consuming many incoming triggers, eg from user input as well as incoming emails from gmail, or messages from slack which would trigger new agent run was lot more natural in go with channels and switch for loop vs in python where had to create many queues and threading etc

So far bigger bottleneck I have found in writing agents is in scaling integrations and not the for loop for agent. Lack of libraries for go is a really big challenge.

What feels missing is cheap wrt what. You have to analyze value created for a service and value captured by company. Eg if you use LLM's to do multiplication of two numbers, its doing billions of computations to get one computation incorrectly predicted majority of the time.

I don't think model sizes increased suddenly, there might not be emergent properties for certain tasks at smaller scales but there was improvement at slower rate for sure. Competition to improve that metric albeit at lower pace led to slow increase in model sizes and by chance led to emergence the way its defined in paper?

If we use some metric as proxy for intelligence, emergence simply means a non-linear sudden change in that metric?

Is there a technical paper released about the model architecture? Great resolution points to diffusion style generation rather than just token based?

Content creation might shift from web to conversations with LLM? Eg if you are not completely satisfied with the output?

Also so far we haven't run out of things to train LLM on, eg images etc are still underutilized, lot of high value content in enterprises not even on public internet.

This seems like intended usage? The server actually executes the moves and interacts with the environment, the core orchestration or reasoning is offloaded to claude?

SEO and UX feedback: All the headers and H1/H2 tags are black in color with dark blue background making text unreadable

This is CLIP. Here if you would see, the model is pretrained to ingest both images and text. However if you would see the prediction mode, its basically text generation only. None of the multi-modal architectures think of both image and text as first class citizens

I feel like success of LLM's have been combination of multiple factors coming together favourably: 1) Hardware becoming cheap enough to train models beyond a size where we could see emergent properties. Which is going to become cheaper and cheaper. 2) Model architecture which can in computationally less expensive manner being able to look at all inputs at the same time. CNN's, RNN's all succeded at smaller scale becuase they added inductive bias in architecture favourable to the input modality, but also became less generic. Attention is simpler in computation to scale it and also has lower inductive bias. 3) Unsupervised text on internet being source of data which requires light pre-processing hence almost no efforts wrt annotations etc reaching scale wrt scaling laws corrosponding to large size models. Also text data being diverse enough to be generic to encompass variety of topics, thoughts vs imagenet etc which is highly specific and costly to produce.

Assuming that text only models will hit a bottleneck, then to have next generation models, in addition to a new architecture, we also have to find rich dataset which is even more generic and much richer in modalities and the architecture being able to natively ingest it?

However something that is not predictible is how well the emergent properties can scale with model size further. Maybe few more unlocks like model being able to retain information well in spite of really large context length, ability to SFT on super complex reasoning tasks without disrupting weights enough to loose unsupervised learning might take us much further?

That's an interesting point. The bias might or might not be intentional. From the benchmarks I have seen, lot of tools solve slightly different problems altogether and also target slightly different data distribution and in the end have to build best solution around it.

Which is why publishing open benchmarks is first step where there is public scrutiny around whether then benchmark itself irrespective of the results is fair. In the end, the end user will choose the benchmark that's best fit for their usecase or mostly will create a variation of their own, do their own unbiased evaluations.

Can share link? Maybe some kind of mistake.

Bias wrt ordering is a great point. What we consider structured information in this benchmark is irrespective of how its presentation (Order, format etc), it should be directly comparable. So the benchmark does that it into account.

Example is if you are only converting lets say an invoice into markdown, you can introduce bias wrt ordering etc. But if the task is to find out invoice number, total amount, number of line items with headers like price, amount, description, in that case you can compare two outputs without a lot of bias. Eg even if columns are interchanged, you will still get the same metric.

Don't think you can. And also there is big difference in plain old OCR, which is just getting all text out from image and document processing which is can you only get the relevant information in a good structure that can be directly pushed into a database.

Automation is combination of both, accuracy and accuracy of confidence scores.

Good way to think about automation is recall at high precision which is what you need for true automation where you don't worry about documents that are very likely to have correct results and focus on manually correcting documents likely to have errors.

The reason accuracies are tighly grouped but not the automation is because these models are trained to be accurate but not necessarily predictable, where there is no real way to get confidence score calibrated.

Couldn't find the benchmark mistral used as well

Yeah this new benchmark is kind of inspired by these existing benchmarks, things that are missing wrt automation

Traditional OCR's usually have detection + recognition pipeline. So they will detect every word and then try to predict the text for every word. Errors obviously can happen in both parts, eg some words not detected which will get missed from output. Or word recognized incorrectly which is also common and more comparable to hallucination. However give its trained to work only on a small patch, accuracy is often higher. Comparing this to VLM's, they are looking at entire image/context and auto-regressively generating tokens/text which can also have lot of language bias, hence hallicinations.

In most multimodal archiectures, images and text is fed as input and they live in same latent space, however major difference is most networks are trained to decode latent vector into text but never an image (Atleast in same network). This makes the network more biased towards text I feel and treats image embeddings as second class citizen?

None. I just got curious about it. For my startup, we have full time employees on board.

"CNNs versus GCNs" is not necessarily correct? You will need to apply GCN on top of CNN to get the structure out of otherwise unstructured text?

To get this personal information, asking for government-issued ID gives them one more data point on you and reinforcement of identity. So a win for them as well I guess?

Yeah nowadays with post-training quantization techniques as well as things like squeezenet which is quantization aware training technique, models are becoming fairly small to be able to run on phone smoothly