191 karma · joined October 8, 2019
And the point is, you could easy do `const db = new Database("./database.sqlite")` instead.
The wrapper makes it so manipulating your database is just like manipulating a plain Javascript object.
I'm guessing some important bottleneck was being hit, but I have no idea what it was (maybe 1GB RAM was causing spillover to disk in a query or something? like that?).
It seems important to understand what bottleneck I hit -- but frankly, I have no idea.
One thing to clarify: It's a 300K x 300K similarity matrix, which means I have 300K embeddings. Each embedding itself only has dimension 512.
In other words, the similarity matrix is the similarity between each embedding & every other embedding in the 300K set of embeddings.
Regardless, I think Dask will be useful here.
I'm not aware of speed test sites that allow you to select the location (i.e doing a speed test from cloud provider A to Hetzner).
I'll look into what's going on by running some CLI speed tests on hetzner.
The thesis was to make annotated textbooks, so if 100 people read a book then the 101th person could benefit from the knowledge of the previous 100 people.
I hope you guys consider the multiplayer aspect of note-taking!
https://github.com/kriszyp/lmdb-js/discussions/170
I think there's a reason Tail Scale used a literal JSON file for up till 150 MB. of data: https://tailscale.com/blog/an-unlikely-database-migration/
So, I think the unconventional wisdom is correct.
I recently used lmdb for webhighlighter.com (specifically the wrapper: https://www.npmjs.com/package/node-lmdb), and it was a fantastic decision.
A lot of people here say "use SQLite for small projects". But even using SQLite can be significant over-engineering. Running migrations? Writing SQL? That's too much effort for me.
For example: In my application, people can leave comments on a (what is effectively) a post. A SQL-native solution might have a table for comments, with foreign keys to post IDs. That's 10x more engineering then I want to do for an MVP. I just store all the comments as an array on a post. This means reads read all the comments and writes require reading all the comments, appending, and then re-writing. That's totally fine, and will probably scale me to 100x my current traffic.
LMDB-JS is great. It allows you to serialize arbitrary JS objects to LMDB using the message pack encoding system. This makes for some super concise code.
Here's my entire data layer: - Interface: https://github.com/vedantroy/grape-juice/blob/main/site/app/... - Implementation: https://github.com/vedantroy/grape-juice/blob/main/site/app/...
TL;DR -- I won't use SQLite, for, I don't know, my first 10K users?
See: https://gist.github.com/vedantroy/80fb2feb21b07b6f53032b8725...
I wrote that. It allows me to do stuff like:
const Button = tw.div(`
bg-blue-500
rounded
`)
Any complaint about how tailwind forces overly long class names should probably look at this comment first.Highlight text on the page and then a button will pop-up, allowing you to create a comment.
Web browsing is currently a solitary activity. You may discover a cool blog, but, unlike a Google Doc, you can't leave a comment on it for everyone to see.
Web Highlighter fixes this. Go to webhighlighter.com/upload and and paste in (almost) any URL to get a real time, collaborative version of the web-page. Share it with people and let them comment away.
Features:
- View other people's comments & reply to them
- Write your own comments by highlighting text on the page and hitting the comment button
- See other people's mouse cursors and what text they've currently selected on the web-page
- Chat with people by pressing the "/" key
- Everything is real time. Comments, mouse cursors, replies all update instantly.
Besides that, there's also a ton of publicly available textbooks online. Example: http://neuralnetworksanddeeplearning.com/
``` <Question> {...} <Footer> < button> { ... } </button> <QuestionBadge> </QuestionBadge> </Footer> </Question> ```
That's easier than
``` <div className = " ... "> { ...} <div> <div> </div> ```
Has anyone used something like Tailwind-Styled-Component [1] and found it worthwhile?