Alexanfa

Noticed an error;

    return run(w, wS, r+1,m)

Should be changed to:

    return run(w, wS, m, r+1)

Ah, I'm using a set instead of list so I just always add and then toss remove.

Quanta:

    Round 1. Keep going through Hamlet, adding new words as you go. If you come to a word that’s already on your list, flip a coin again. If it’s tails, delete the word; heads, and the word stays on the list.

To:

    Round 1. Keep going through Hamlet, but now flipping a coin for each word. If it’s tails, delete the word if it exists; heads, and add the word  if it's not already on the list.

Old edit:

    Round 1. Keep going through Hamlet, adding words but now flipping a coin immediately after adding it. If it’s tails, delete the word; heads, and the word stays on the list.

Was just now solving it and came to see if others had the same issue. Yep, you are right.

    function generateRandomNumbers(c, n) {
      let randomNumbers = new Array(c);
      for (let i = 0; i < randomNumbers.length; i++) {
        let randomNumber = Math.floor(Math.random() * (n + 1));
        randomNumbers[i] = randomNumber;
      }
      return randomNumbers;
    }
    function run(w, wS, m, r) {
        function round(r) {
            while(wS.size < m) {
                const next = w.next()
                if (next.done) return true;
                wS.add(next.value)
                prune(next.value, r)
            }
            return false
        }
        function prune(v,r) {
            for (let i = 0; i < r; i++) {
                const flip = new Boolean(Math.round(Math.random()))
                if (flip == false) {
                    wS.delete(v)
                }
            }
        }
        function purge(wS) {
            const copy = new Set(wS)
            copy.forEach(ith=>{
                const flip = new Boolean(Math.round(Math.random()))
                if (flip == false) {
                    wS.delete(ith)
                }
            })
        }
        const done = round(r);
        if (!done) {
            purge(wS)
            return run(w, wS, r+1,m)
        }
        console.log(`Round ${r} done. ${wS.size} Estimate: ${wS.size / (1/Math.pow(2,r))}`)
    }
    const memory = 1000
    const words = generateRandomNumbers(3000000,15000)
    const w = words[Symbol.iterator]() // create an iterator
    const wS = new Set();
    run(w,wS, memory,0);

I think you could apply to product manager roles within your speciality and do very well… for instance at a sales saas tool

Hey! We did some texting on https://github.com/twitter/the-algorithm without issues(*), that was around 150K lines of code.

And it works great, I wish I knew more domain details about twitter to be able to really leverage it, but it quickly finds parts of your system and how to build features like "removing echo chambers from the-algorithm by recommending posts to a broader audience that is likely to have opposing views and interact with others."

I think one of our users tried indexing their internal code on about 200k LOC.

The outputs are however limited to 4k or 8k depending on what model you use.

*(initial indexing might be slow and re-indexing might need to be disabled)

Hey,

What you can do is to use caching and embeddings. Embeddings are like X,Y coordinates (except with OpenAI it's 1500 dimensions not just two). You use embeddings on any type of text. Here you would embed user prompts and store answers. And what this means is, you would be able to get X,Y for "best restaurants close to the museum", "best places to eat" etc.. If a new user comes along and mentions food and museum, it's likely to rank highest for "best restaurants close to the museum". I haven't done this and idk if possible, but you could try to then check if this is "close enough". If it is, then you can probably return the same generated answer for this user. You can also embed answers and see if that has better result.

Process: 1. When user asks a question, you embed it and search with it. 2. If similar prompts exist; return the answer from the most "relevant" prompt. 3. If no similar prompt exists; generate a new answer, and store it to be associated with the prompt.

This might decrease quality, but maybe in "enough" cases, it does just fine.

I'm developing tzap.io that uses some of these techniques, join our discord if you want to get in touch: https://discord.gg/88xDVYbPVB