gillesjacobs

I really want to see some evaluation benchmark comparisons on in-chunk augmentation approaches like this (and question, title, header-generation) and the hybrid retrieval approach where you match at multiple levels: first retrieve/filter on a higher-level summary, title or header, then match the related chunks.

The pure vector approach of in-chunk text augmentation is much simpler of course, but my hypothesis is that the resulting vector will cause too much false positives in retrieval.

In my experience retrieval precision is most commonly the problem not recall with vector similarity. This method will indeed improve recall for out-of-context chunks, but for me recall has not been a problem very often.

Yeah exactly, existing benchmark datasets available are underutilized (eg KILT, Natural questions, etc.).

But it is only natural that different QA use cases require different strategies. I built 3 production RAG systems / virtual assistant now, and 4 that didn't make it past PoC and what advanced techniques works really depends on document type, text content and genre, use case, source knowledgebase structure and metadata to exploit etc.

Current go-to is semantic similarity chunking (with overlap) + title or question generation > retriever with fusion on bienc vector sim + classic bm25 + condensed question reformulated QA agent. If you don't get some decent results with that setup there is no hope.

For every project we start the creation of a use-case eval set immediately in parallel with the actual RAG agent, but sometimes the client doesn't think this is priority. We convinced them all it's highly important though, because it is.

Having an evaluation set is doubly important in GenAI projects: a generative system will do unexpected things and an objective measure is needed. Your client will run into weird behaviour when testing and they will get hung up on a 1-in-100 undesirable generation.

Source: https://github.com/MangoLion/bgbye

Poetry worked best in several Python-first companies and clients I have worked for.

It's not perfect but the best bet for complex and collaboration-heavy projects.

ML engineer, I am up to 3 irreversibly broken environments after simply adding a package. Conda is inexcusably slow and but even mamba can save it. Every conda project I have ever was later switched to poetry.

Here's a fun challenge: try to determine channel priority in your conda env, go ahead try.

Conda is the single worst packaging tool I have ever used.

The cited reason given is

> An investigation by the publisher found a number of articles, including this one, with a number of concerns, including but not limited to compromised editorial handling and peer review process, inappropriate or irrelevant references or not being in scope of the journal or guest-edited issue.

It's quite a leap to arrive at AI generated content.

It is all speculation, but it sounds more likely the guest editor did not properly handle peer review (not blind, befriended or non-expert reviewer) and/or misapplied submission selection criteria (maybe out of scope papers by friendly colleagues were not rejected).

Exactly, BloombergGPT performed worse on financial sentiment analysis then much smaller fine-tuned Bert-based models.

For many extractive tasks BloombergGPT was quite disappointing. A 5-10% performance hit with much larger inference cost compared to smaller models is not desirable.

But the research investment for Bloomberg makes sense to take the risk: a do-it-all generative model can mean significant complexity reduction in maintenance and deployment overhead.

It didn't directly pay off for many extractive tasks, but I bet they're iterating. Bloomberg has the data moat and the business needs in their core products to make it worthwhile.

This is indeed often the case, however, my university reviews each thesis, and deemed it can only change to open access in 2026 (+5 years from defense).

I think this is default policy for thesis based on publication agreements here.

In any case, I am not too worried.

Thanks! The fine-tunes where done in 2019-21 on a 4xV100 server with hyperparameter search, so thousands of individual fine-tuned models were trained in the end. I used weights and biased for experiment dashboarding the hyperparam search, but the hardware was our own GPU server (no cloud service used).

I doubt you can fine-tune BERT-large on a phone. A quantized, inference optimised pipeline can be leaps and bounds more efficient and is not comparable with the huggingface training pipelines on full models I did at the time. For non-adapter based training you're going to need GPUs ideally.

I use LLM information extraction for financial news articles with OpenAI Azure and it is a huge problem for me.

404 Content moderation response in 4% of articles. This is just financial news text.

It is a prime reason we are considering open models.

rovr beat me to it below. Here are more links: https://jacobsgill.es/phdobtained (fun fact: because my thesis contains published papers, I am in breach of a few journal's copyright by uploading my own thesis pdf, but fuck'em).

LLM approaches were evaluated on my own time and but published (I left research after obtaining my PhD).

This is entirely unsurprising and in-line with the finding that even small specialized models do better in information extraction and text classification. So no wonder finetuned large LMs do good too.

Personally, my PhD did fine grained ACE-like event and sentiment extraction and "small" specialized finetuned transformers outperformed prompting LLMs like BERT and Roberta-large. Would love to see an inclusion of small model scores with some sota pipelines.

This is great work anyway even if it replicates known results!

Facebook/Meta has been doing open research in ML and NLP for far longer than the current LLM era: Convolutional Neural Networks for Sentence Classification (2014), FastText (2016), PyTorch (2016), fairseq (2017), LASER (2018), RoBERTa (2019), XLM-R (2019), and BART (2019),

They were always present at ACL with decent open research as far as I have been studying/working in NLP (2014).

It's part of a strategy to attract top talent in the field. If you want top researchers you have to let them publish, which in turn hones a reputation of solid research, attracting more talent.

I stand corrected. Guess all that posturing about privacy was all empty election promises then.

Seems like only the Greens, the left and within ID the AFD voted against.

There are many different types and implementations of representative democracy. And the supranational technocracy-driven particracy of the EU is only representative in name. It is on the oligarchic end of the democracy-oligarchy spectrum.

Edit: well like someone pointed out instead of party programmes and declarations, we should look at past voting behaviour on the 2021 vote: https://mepwatch.eu/9/vote.html?v=134463&eugroup=ID

Seems like only the Greens, The Left and Afd within ID were against that first version of Chat Control.

___

This isn't the case though: the right-wing Identity and Democracy faction is explicitly pro-privacy, digital rights and against Chat Control.

- https://id-party.eu/program/ (ID Party Official Site)

- https://idgroup.eu/news/online-censorship-is-a-threat-to-eur... (ID Group News)

- https://en.wikipedia.org/wiki/Identity_and_Democracy (Wikipedia Overview)

- https://id-party.eu/declaration-of-antwerp/ (ID Party Official Site)

The ID group is opposed to EU-wide surveillance measures, and promises to protecting individual privacy and national sovereignty.

Now ECR is also sceptical and have expressed concerns, though their voting record on privacy related legislation is more inconsistent, I believe.

Supranational/federalist "representative" particracy does not work.

Representative democracy works better with increased locality, where policy and politicians are directly beholden to constituents.

This isn't necessarily true: the Identity and Democracy faction is explicitly pro-privacy, digital rights and against Chat Control.

- https://id-party.eu/program/ (ID Party Official Site)

- https://idgroup.eu/news/online-censorship-is-a-threat-to-eur... (ID Group News)

- https://en.wikipedia.org/wiki/Identity_and_Democracy (Wikipedia Overview)

- https://id-party.eu/declaration-of-antwerp/ (ID Party Official Site)

The ID group is opposed to EU-wide surveillance measures, and promises to protecting individual privacy and national sovereignty.

Because it is exactly the EU governance that lacks democratic representation, enabling unfavoured politicians to remain in significant power. You can only elect MEPs (via party lists) locally. The majority of legalislation is drafted by the technocratic European Commission and legislation is mostly passed through parlaiment unchanged from there.

There are many nationally disgraced politicians seating in the EP, EC or council. Only the ones in the EP were electible.

The people have correctly identified that a massive supranational unaccountable government is the problem, enabling corrupt people to keep ruling which undermines the core functioning of representative democracy.

The field of study is called linguistic typology.

During my linguistic studies, the course material was comprised of the Oxford and Cambridge Handbook of Linguistic Typology (and a bunch of papers on specific linguistic features). But the Oxford handbook is a very good start.

It's a very cool field of study that will allow you to identify commonalities across languages and the differences between them. A good typologist should be able the be dropped in a completely unknown language context and learn the features from use (which was basically our exam).

Dat het verschil in gemiddelde passieve woordenschat tussen Duitse en Nederlandse moedertaalsprekers zo groot is, twijfel ik als linguïst toch wel een beetje aan.

Are you sure that there study/studies are sound and use directly comparable measures for passive vocabulary items?

Ah this takes me back a decade when I used Praat to annotate the prosody of "no wonder" mirative statements for my Computational Linguistics master thesis. Hours spent in Praat and that horrible scripting language...

It did lead to my first publication as co-author and some interesting insights: https://kuleuven.limo.libis.be/discovery/fulldisplay?docid=l...

Always beware these type of Social Science studies that make for great popsci headlines.

The kind of result that makes careers: seemingly elegant and simple study design, common sense hypothesis and overall great marketability.

Diederik Stapel was one of the largest science fraudster in history and this kind of stuff was his bread and butter. More recently Francesca Gino was caught falsifying the data of the "honesty pledge at the top of form" that is highly similar in concept to this.

Now for the career-ending question: Will it replicate?

Used this library quite a bit. I still have no idea if there is a good reason this API is not packaged within huggingface/transformers.

Probably historic reasons, anyway solid 9/10 API.

Back in 2020, so that's 4 years of active development on both tools since then. Both tools have different scopes though (cf my other comment in the thread).

You need to upload your time series (loss, performance metrics) to the cloud. Not your weights or models.

Reliance on cloud services is a legitimate worry though for privacy, IP, process control, reliability, etc.

The comparison between Optuna and WandB was not apples to apples. Optuna is completely self-hosted and local. It also focuses on hyperopt narrowly with flexible design unlike WandB that now assumes to be capture a large part of cloud-based MLOPS workflow.

It would be more fair to compare Optuna to Hyperopt. And I think Optuna was the better choice there, but I did simple PoCing and have no strong opinions.

I have extensively tested Optuna and Weights and Biases (WandB) for hyperparameter tuning on multiple task-specific transformer models back in 2020.

Optuna lost out by a long mile back then in feature parity and dashboarding. Optuna did not have hyperband optimization which was and still is one of the best search algos for hyperopt. It looks like it is possible to implement hyperband yourself now, but in the loosely coupled architecture between Sampler and Pruner it's a bit baroque [1].

Anyway back then it was clear WandB was the far superior choice for features, ease of use, experiment tracking and dashboarding. We went with WandB for our lab.

Could be Optuna caught up, but WandB has seen significant development too. Looking at their dashboard docs, it looks meagre compared to what you can do with WandB.

1. https://tech.preferred.jp/en/blog/how-we-implement-hyperband...

Everything is an engineering trade-off between classification performance, data requirements (quality and quantity for supervised fine-tuning/training), dev time and inference speed/resources.

You don't always have to do full document extraction. For a pdf form information extraction task, I once implemented a pipelined approach on images that extracts handwritten fields by using a trained segmentation model and a extraction model. You get error percolation but performance was between 85-90 F1, and we had human-in-the-loop for uncertain preds.

However, if top performance is a concern (and you're not dealing with handwritten stuff here) I would look into SotA document extraction models for extracting text, tables, graphs etc like LayoutLMv3 [1]. It's probably something like LayoutLM that OpenAI is using themselves. Every major cloud provider has its own document AI services that have similar performance, so that might get you going faster.

On the extracted text of scientific papers, you will run in to context size issues, so take it into account if using LLM zero-shot approach.

You could go with a chunked topic model approach for this too (BerTopic is a nice starting point).

In any case, it all depends on the specific task (amount of topics to classify), and the engineering trade-offs. Only you have all information to figure that one out.

1. https://huggingface.co/docs/transformers/model_doc/layoutlmv...

2. https://maartengr.github.io/BERTopic/index.html

The yes-or-no reference answer test is a really bad way to go about this. Maybe take a note out of RAGAS evaluation templates and use an LLM to iteratively summarise the nuanced category.

They have invested the most in preference alignment with special attention to DEI (for better or for worse).