nutanc

Hey, no problem on the language. Always happy for any thoughts :) We will open source next week.

A couple of corrections: I never meant to say our method was superior. We have not done any benchmarks etc enough to write a paper. We have just been using this approach in our RAG pipeline and we have been happy with it. Sorry, will add a disclaimer in the post.

Agreed. For semantic chunking, only read and compare works best. Not sure how to scale that to a bench mark. Guess for the PG essay, only PG can answer if the topics align with his thoughts :)[https://x.com/nutanc/status/1838813258972549549]

We will release the algorithm. I just put this out on Hackernews thinking this will also not be noticed as my all other posts :)

But looks like some interest is there. Will post a follow up update soon. Sorry for the lack of details on the original post.

Hey thanks. Yeah, even we felt it was very obvious and why no one has done this before :)

Will share more details soon.

Dammit, now bureaucrats in other countries will jump on this as they have something easy to copy and get ahead in their profession.

Actually what is also needed is a notLLMs.txt.

robots.txt exists, but is mainly for crawling and also not sure anyone follows it or even if they don't follow what's the punishment.

Had tweeted about this sometime back. Found a component which was open source earlier and then removed and only minfied JS was provided. Give the JS to Claude and get the original component back. It even gave good class names to the component and function names.

Actually this opens up a bigger question. What if I like an open source project but don't like its license. I can just prompt AI by giving it the open source code and ask it to rewrite it or write in some other language. Have to look up the rules if this is allowed or will be considered copying and how will a judge prove?

Talking in LLM parlance, you are put in a different context in the embedding space.

This is awesome. Does Milvus lite also support binary embeddings?

What about the privacy concerns? So basically I can just look at a couple of people talking and eavesdrop?

It's useful because you get to increase your startup valuation if you use "RAG".

AI in education will be the FSD in driving. Everyone thinks we are very close and the tech is just about there. 10 years later we will have made some progress but nowhere near the utopia promised.

Education has a lot more edge cases than driving.

> We offer Customers a choice around these practices. If you want to exclude your Customer Data from helping train Slack global models, you can opt out. If you opt out, Customer Data on your workspace will only be used to improve the experience on your own workspace and you will still enjoy all of the benefits of our globally trained AI/ML models without contributing to the underlying models.

Sick and tired of these default opt in explicit opt out legalese.

The default should be opt out.

Just stop using my data.

Yes. The component has this rule in place as thats how the educator decided to add the rule. But we are updating the component to include other options also.

Hey. Thanks for trying it out and for taking the pains to share your feedback. Have seen some places where the content mismatch is happening and the continuation is missing. What we have realized is that getting the content creators(math educators) and the site creators(developers) to synch up is the biggest problem in this :)

We have done one round of content cleaning, hope to have most of the content with no issues by this month end.

This looks really cool. Possible to connect with you to see how we can also use this for https://books.innings2.com/demo.

This is a free website for math content from grade 6-10th. This fits with our design philosophy. Want to see if we can make something interactive with vector-graph

Cool to see stuff like this which makes math fun for everyone. I love stuff like this because it brings together two things I love, math and programming.

I mean textbooks are cool. But with the tools available to us we should be able to make almost any textbook interactive. It will need effort, pedagogy, programming skills and design. But it's certainly worth it.

Making an effort with this. Started with 6th to 10th grade math. Let's see how this goes.

It does not have to do with the book or any other medium. It comes from the learner. You are right about the problems mentioned.

1. Stuff does not have longevity. This is because most of the new tools are outside curriculum. This does not encourage students to learn something extra.

2. Proprietary. Books providing add ons have a different goal. They want to sell their books. So the addons or just that. Add ons. They are not complete.

These were some of the thoughts which encouraged us to build [1]. Keep it curriculum focused so kids don't learn anything extra. No books to sell :)

[1] https://books.innings2.com/

This was the core premise on which Innings2 books was made. In this day and age textbooks can be made interactive. So, the CBSE math syllabus (main syllabus in India) was made interactive for 6th to 10 grades.

https://books.innings2.com/demo

The attempt is not to replace a particular neural network which has already been trained by using Sigmoid or Rel functions. If one does this then one would necessarily have to use non-linear maps. The whole point is that such a non-linear technique is not necessary for classifications. It is not necessary to confine clusters by hyperplanes for solving a classification problem. Our focus is on individual points.

We believe the brain does not do nonlinear maps!

Unfortunately the source code is currently not open sourced. Some more details at (https://www.researchgate.net/publication/370980395_A_NEURAL_...), the source code is built on top of this.

The approach is used to solve other problems and papers have been published under https://www.researchgate.net/profile/K-Eswaran

We are currently trying a build a full fledged LLM using just this approach(no LLM training etc) and also an ASR. We should have something to share in a couple of months.

Will add a lot more details next week. Have been postponing it for a long time.

Yeah, sorry, needed a much bigger canvas than a comment to explain. Let me try again. The example I took was to show mapping from one space to another space and it may have just come across as not learning anything. Yes. You are right it was someone else's pretrained LLM. But this new space learnt the latent representations of the original embedding space. Now, instead of the original embedding space it could also have been some image representation or some audio representation. Even neural networks take input in X space and learn a representation in Y space. The paper shows that any layer of a neural network can in fact be replaced with a set of planes and we can represent a space using those planes and that those planes can be created in a non iterative way. Not sure if I am being clear, but have written a small blog post to show for MNIST how an NN creates the planes(https://gpt3experiments.substack.com/p/understanding-neural-...). Will write more on how once these planes are written, how we can use a bit representation instead of floating point values to get similar accuracy in prediction and next how we can draw those planes without the iterative training process.

We have been experimenting with the paper(https://www.researchgate.net/publication/372834606_ON_NON-IT...).

There is a mathematical proof that binary representation is enough to capture the latent space. And in fact we don't even need to do "training" to get that representation.

The practical application we tried out for this algorithm was to create an alternate space for mpnet embeddings of Wikipedia paragraphs. Using Bit embedding we are able to represent 36 million passages of Wikipedia in 2GB.(https://gpt3experiments.substack.com/p/building-a-vector-dat...)

He quit to build a blogging platform.

https://x.com/karpathy/status/1751350002281300461?s=20

Do you think this worked so cleanly because there is a tutorial similar to this and its in the dataset?

https://github.com/liabru/matter-js/wiki/Tutorials

Wikipedia neural search running on a laptop(2GB small model and 10GB large model).

https://speech-kws.ozonetel.com/wiki

Thanks to @CohereAI for releasing the Wikipedia embedding dataset. I saw that for 36 million passages the embedding size would be around 120 GB. So if I had to host the embeddings and enable neural search on this dataset I will have to do LLM ops and runs vectorDB clusters. This was a good dataset to test out the efficacy of the Alpes KE Sieve algorithm. We built an embedding space based on @huggingface all-mpnet-base-v2 model. We created two models, one with 540 dimension bit embedding(small) and another with 2200 dimension bit embedding(large). We were able to embed all the 36 million passages in 2GB and 10GB respectively.So you can now have local wikipedia vector search locally. Since the size is so small, we just used np.array and no vector databases. You can test it out in the link above and share your feedback.

Have written a little about it here, https://gpt3experiments.substack.com/p/building-a-new-embedd...

Will do a bigger blog post soon on the approach. Can mail you some details if you are interested.

What if we have an alternative to deep learning that does not need those thousands of GPUs?

One thing I always hated about chatbot sites when they were the craze and the the AI help bot sites now is the fact that these sites do no provide a chatbot for their own site. I mean, why isn't there a CommandBar for commandbar.com?

I actually see that commandbar.com has an intercom chat widget.

The funny thing is that Google Bard already knows about this story and can perform generative AI tasks on this like summarizing etc[1]

[1] https://twitter.com/nutanc/status/1656533992785723392?s=20