Tranformer-based Denoising AutoEncoder for ST Unsupervised pre-training

A new PyPI package for training sentence embedding models in just 2 lines.

The acquisition of sentence embeddings often necessitates a substantial volume of labeled data. However, in many cases and fields, labeled data is rarely accessible, and the procurement of such data is costly. In this project, we employ an unsupervised process grounded in pre-trained Transformers-based Sequential Denoising Auto-Encoder (TSDAE), introduced by the Ubiquitous Knowledge Processing Lab of Darmstadt, which can realize a performance level reaching 93.1% of in-domain supervised methodologies.

The TSDAE schema comprises two components: an encoder and a decoder. Throughout the training process, TSDAE translates tainted sentences into uniform-sized vectors, necessitating the decoder to reconstruct the original sentences utilizing this sentence embedding. For good reconstruction quality, the semantics must be captured well in the sentence embeddings from the encoder. Subsequently, during inference, the encoder is solely utilized to form sentence embeddings.

GitHub : https://github.com/louisbrulenaudet/tsdae

Installation : pip3 install tsdae nltk datasets sentence-transformers torch

Python code : from tsdae import TSDAE

# Initialize an instance of TSDAE instance = TSDAE()

# Load a dataset train_dataset = instance.load_dataset_from_hf( dataset="louisbrulenaudet/cgi" )

# Train the model with the dataset model = instance.train( train_dataset=train_dataset, model_name="bert-base-multilingual-uncased", column="output", output_path="output/tsdae-lemon-mbert-base" )