Hacker News Reader

txtaion Topic modeling with semantic graphs: a different approach

Currently, topic categories are used to label the generated topics, hence the single additional level.

Expanding functionality to include something like this is possible though. The community detection algorithms kind of do this in a way. Each iteration builds smaller and smaller communities.

txtaion Topic modeling with semantic graphs: a different approach

With the method discussed here you can add a list of labels to categorize generated topics. This isn't quite what you're looking for but it does create a two level hierarchy.

Reference: https://neuml.github.io/txtai/embeddings/configuration/#topi...

txtaion Topic modeling with semantic graphs: a different approach

Dimensionality reduction with UMAP combined with HDBSCAN is a popular topic modeling method found in a number of libraries. txtai takes a different approach with a semantic graph.

When enabled, txtai builds a semantic graph at index time as it's vectorizing data. These vector embeddings are then used to create relationships in the graph. Finally, community detection algorithms build topic clusters.

This approach has the advantage of only having to vectorize data once. It also has the advantage of better topic precision given there isn't a dimensionality reduction operation (UMAP).

txtaion The Overflow Offline project

There was a recent HN Post for codequestion which builds an offline semantic index (using https://github.com/neuml/txtai) on the archive.org Stack Overflow dumps - https://news.ycombinator.com/item?id=33110219

GitHub: https://github.com/neuml/codequestion

Article: https://medium.com/neuml/find-answers-with-codequestion-2-0-...

txtaion Txtai 5.1: better machine translation, Whisper transcription and ARM support

txtai executes machine-learning workflows to transform data and build AI-powered semantic search applications.

5.1 adds new model support for the translation pipeline, OpenAI Whisper support in the transcription pipeline and ARM Docker images. Topic modeling was also updated with improvements, including how to use BM25/TF-IDF indexes to drive topic models.

Release notes: https://github.com/neuml/txtai/releases/tag/v5.1.0

txtaion Txtai 5.1 released: new translation models, Whisper transcription and ARM images

txtai 5.1 adds new model support for the translation pipeline, OpenAI Whisper support in the transcription pipeline and ARM Docker images. Topic modeling was also updated with improvements, including how to use BM25/TF-IDF indexes to drive topic models.

Release notes: https://github.com/neuml/txtai/releases/tag/v5.1.0

txtaion Locally-hosted alternative to cloud machine translation

With the rapid pace of development in machine learning, the quality of seq2seq translation models are fast approaching what's available on cloud platforms. Convenience, cost and privacy are all good reasons to consider. In the case of low-resource languages, cloud translation may not exist.

Couple models to highlight (available on the Hugging Face Hub):

facebook/mbart-large-50: Supports translating between 50 different languages. https://huggingface.co/facebook/mbart-large-50

facebook/m2m100_418M: Supports translating between 100 different languages. https://huggingface.co/facebook/m2m100_418M

facebook/nllb-200-distilled-600M: Translations between 200 languages, including low-resource languages. https://huggingface.co/facebook/nllb-200-distilled-600M

txtaion Topic Modeling with BM25 and Graphs

Also available as an article: https://neuml.hashnode.dev/classic-topic-modeling-with-bm25

txtaion Show HN: Semantic search of Stack Overflow with codequestion

Another good point. Once you open a web browser, you can get lost in the internet.

txtaion Show HN: Semantic search of Stack Overflow with codequestion

No problem with the question. codequestion builds a fully local index that can run in VS Code. It also has features for topic exploration and graph path traversal.

txtaion Show HN: Semantic search of Stack Overflow with codequestion

codequestion 2.0 is a major new release. This release migrated from word vector to sentence transformer models. It also integrates semantic graph functionality (topic modeling, path traversal) from txtai 5.0.

Release Announcement - https://medium.com/neuml/find-answers-with-codequestion-2-0-...

Release Notes - https://github.com/neuml/codequestion/releases/tag/v2.0.0

txtaion [dead]

txtai 5.0 is a major new release. This release adds the semantic graph along with enabling external integrations. It also adds a number of improvements and bug fixes.

Semantic graphs, also known as knowledge graphs or semantic networks, build a graph network with semantic relationships connecting the nodes. Semantic graphs in txtai can be used for topic modeling, graph traversal and analysis.

Release Announcement - https://medium.com/neuml/whats-new-in-txtai-5-0-e5c75a13b101

Release Notes - https://github.com/neuml/txtai/releases/tag/v5.0.0

txtaion [dead]

txtai 5.0 is a major new release. This release adds the semantic graph along with enabling external integrations. It also adds a number of improvements and bug fixes.

Release Announcement - https://medium.com/neuml/whats-new-in-txtai-5-0-e5c75a13b101

Release Notes - https://github.com/neuml/txtai/releases/tag/v5.0.0

txtaion Txtai 5.0 released – build semantic search applications and workflows

txtai 5.0 is a major new release. This release adds the semantic graph along with enabling external integrations. It also adds a number of improvements and bug fixes.

Release Announcement - https://medium.com/neuml/whats-new-in-txtai-5-0-e5c75a13b101

Release Notes - https://github.com/neuml/txtai/releases/tag/v5.0.0

txtaion Whisper – open source speech recognition by OpenAI

Check out this notebook for an example on how to run Whisper as a txtai pipeline in Python or as an API service: https://colab.research.google.com/github/neuml/txtai/blob/ma...

txtaion Measuring the popularity of different vector databases

Thank you for the kinds words!

Best documentation on training models is the trainer pipeline - https://neuml.github.io/txtai/pipeline/train/trainer. There are a lot of great pretrained models available: https://huggingface.co/models?pipeline_tag=sentence-similari...

Funny on Python, never thought of how that section read, definitely a good clarification to make.

txtaion Measuring the popularity of different vector databases

https://github.com/neuml/txtai

txtai can build vector indexes with Faiss/HNSW/Annoy and supports running SQL statements against them. External vector databases can also be plugged in.

txtaion Show HN: The Semantic Graph

Also available here: https://neuml.hashnode.dev/introducing-the-semantic-graph

txtaion Shakespeare Semantic Search

When I enter the text and scroll down on left panel to click "Search" it works at least on my machine.

txtaion Shakespeare Semantic Search

On mobile the left panel won't automatically show up, you have to expand it, enter text and click Search button under the "Ignore Search Words" check box.

txtaion Shakespeare Semantic Search

You have to click search

txtaion Show HN: Build simple yet powerful open-source semantic applications

GitHub Project Link: https://github.com/neuml/txtai

txtaion Show HN: Generate webpage summary images with DALL-E mini

That is a good point if you're looking for an existing image. This post is looking towards DALL-E mini as a start in generating descriptive images for any text.

txtaion Show HN: Generate webpage summary images with DALL-E mini

You can also change the min/max summary length, which is defaulted to 0 and 60 respectively.

txtaion Show HN: Generate webpage summary images with DALL-E mini

You can also change the prefix (or remove it) to change how the drawings are created. Right now it's set to "Illustration of ". You can try different things like "Sketch of ", "Oil painting of ", "3d animation of " and "Mosaic of ".

txtaion Show HN: Generate webpage summary images with DALL-E mini

You can change the seed. If you search the notebook for 1024, you can replace that argument with a different seed to get different results.

txtaion Show HN: Generate webpage summary images with DALL-E mini

Agree, that one stood out to me too. Though they are all fascinating in a way.

For example, if you read the Wikipedia article on the War of 1812 and then look at the image generated, it makes sense why it chose frigates at sea. If you've been to the Epcot, the image looks similar to the world showcase around Germany/Italy, but it's original in it's own way.

txtaion Show HN: Generate webpage summary images with DALL-E mini

Thanks, good to know.

txtaion Show HN: Generate webpage summary images with DALL-E mini

Was this a cost just for the DALL-E piece or the full text extraction, summarization and image generation workflow?

txtaion Show HN: Generate webpage summary images with DALL-E mini

OK, I misunderstood there. Running the code, which generates 15 images, on a standard GPU Colab environment takes about 2 minutes. It may be possible to submit a single batch of text summaries to the DALL-E mini model, which would improve performance a good deal.