1,278 karma · joined May 1, 2014
mag@entropy.be
I think we have to wait for the explanation is chat-bison PaLM 1 or 2
Yup, that's exactly what I'm saying there is something wrong with data model. I also noticed it's only using a single thread.
Out of curiosity what dataset size you guys are testing on? Indexing 2M documents on a server in 2023 should be instant, or at least <60 seconds.
Other learning is, not every software written in Rust is fast :-)
For any collection above 1.5M documents, insert times are crazy, like 60 seconds to index additional 6000 documents. There is something very wrong with the core data model and single writer LMDB bottlenecks IMO.
OK I get the occasional spam through spamassasin, but I can live with archiving 5 spam emails/day.
The problem was with token embeddings, not with math.
562 Billion parameters!
Even with int8 it's 0.5TB of memory to run this
Google is testing their system internally with XX thousand users, OpenAI with XXX million users ...
I've seen cases where you want to provision the new rack but unfortunately DC is full and you have to end up running infrastructure spanning across different buildings or even different DCs.
With Zen4 I would go with single CPU option, it's hard to saturate 256 threads.
Main cost in the cloud is always the database server, because it needs to be on 100% of the time, cannot be on spot instance, and needs lots of memory and CPU just in case of a spike of traffic.
http://fileformats.archiveteam.org/wiki/Zstandard_dictionary
Ability to run the whole product no your Laptop/in docker and have INSTANT RELOAD, you change something and test it in <5 seconds.
Tests/CI etc can be run when feature is complete and you are making a PR.
Maybe you can spin up some part of the infrastructure that are not latency sensitive in the nearby region?
Stay tuned for launch next week! :)
It's an amazing piece of hardware, you can mount it on a car and drive around looking for interesting places with RF emissions
I go to the data centre from time to time, mostly taking junior engineers for a sightseeing trip to show them how "the cloud" actually looks :-)
If you need colocation in London I can put you in touch.