cooljoseph

Last year I also had a rather similar idea, but dropped it before I went very far in working on it. I wonder if you and I had similar ideas?

1. Start with an LLM, so that your model understands natural language.

2. Replace RoPE with a tree embedding scheme, and causal attention with a sparse attention on the graph structure. (You could use full attention... but it's cheaper to use graph attention.)

3. Chop off the final unembedding layer, replacing it with a projection down to two scalars, one for logits and one for confidence.

4. Each option of a choice is represented by a number of tokens in leaf position; average these tokens' logit outputs to get the option's logit. Average all of the confidences from all of the options to get the choice's confidence.

5. Train the logits by KL divergence from a true distribution (or NLL on samples from a true distribution).

6. Train the confidences on a subset of the data in which you know the entire true distribution.

The hardest part is getting real world data for workflows, but I wildly speculate that you can get by with only ~50,000 documents if you first adapt domains using synthetic data.

The inputs are natural language, they're just also structured into a tree. The first example on that very page shows natural language instructions:

    questions = {
        "refund_requested": Noul(
            instructions="Does the customer request a refund?",
        ),
    }

Thanks for the early access! I was testing the Lisp idea out in the playground, but I don't think the model is smart enough right now to generate actual code. I tried having Jev finish generating the code for a Fibonacci number function, but it kept wanting to create a literal number instead of refer to a variable which is a number. This happened both when I gave Jev the current program as a string and when I gave Jev the program as structured data.

Maybe I'm just not doing a very good job at prompting Jev, but I think right now it's not quite capable enough to generate Lisp code.

Link: https://console.typesafe.ai/playground?share=shr_148e1248984...

A few questions:

1. Do you provide any kind of largest common subtree caching for cheaper input?

2. Have you tried auto-generating Lisp programs structurally?

3. Have you tried augmenting a Lisp language with a `choice` function that makes choices given a prompt, the environment, and the continuation stack?

I've done something similar for a couple tools.

I tend to make them as Python servers which serve plain html/js/css with web components. I know this is a bit more complicated than just having a single html file with inline js and css, but the tools I made were a bit too complicated for the LLMs to get just right, and separating out the logic into separate js files as web components made it easy for me to fix the logic myself. I also deliberately prompted the LLMs to avoid React because adding I didn't want to need a build step.

The only one I actually still use is the TODO app I made: https://github.com/cooljoseph1/todo-app It stores everything in a JSON file, and you can have multiple TODO lists at once by specifying that JSON file when you launch it.

This sounds somewhat like a normalizing flow from a discrete space to a continuous space. I think there's a way you can rewrite your DDN layer as a normalizing flow which avoids the whole split and prune method.

1. Replace the DDN layer with a flow between images and a latent variable. During training, compute in the direction image -> latent. During inference, compute in the direction latent -> image. 2. For your discrete options 1, ..., k, have trainable latent variables z_1, ..., z_k. This is a "code book".

Training looks like the following: Start with an image and run a flow from the image to the latent space (with conditioning, etc.). Find the closest option z_i, and compute the L2 loss between z_i and your flowed latent variable. Additionally, add a loss corresponding to the log determinant of the Jacobian of the flow. This second loss is the way a normalizing flow avoids mode collapse. Finally, I think you should divide the resulting gradient by the softmax of the negative L2 losses for all the latent variables. This gradient division is done for the same reason as dividing the gradient when training a mixture-of-experts model.

During inference, choose any latent variable z_i and flow from that to a generated image.

> The system is full up at the moment and you can't really add anything without removing some things.

Middle school seems rather un-full to me. Right now students start learning about fractions in 4th grade. They don't move on to algebra until 9th grade. What is there in the middle? Not much, in my experience.

Maybe instead of having a giant no-math gap during middle school, they could move everything down and free up some space later on.

No, surveys like where researchers show up to hospitals and look at the police reports for the injured cyclists.

Sorry if I wasn't clear in my wording. By "survey" I was trying to point to the specific kind of research methodology where you survey people about what has happened in the past instead of trying to control variables like in a typical experiment.

I wasn't talking about random internet polls or self-reported blame analyses, but actual research papers.

> I'd bet accidents are more often than not mostly their fault.

That's actually not true. Most surveys I've seen show that drivers are at fault ~80% of the time.

It is possible to construct φ exactly with a straight-edge and compass. Would the approximation of 5π/6 - 1 be used because it's easier to calculate quickly?

Are you sure about that? According to Wikipedia both gypsum and calcite are used. Apparently, gypsum is used for colored chalk, and calcite is used for white chalk:

> Chalk sticks are produced in white and in various colours, especially for use with blackboards. White chalk sticks are made mainly from calcium carbonate derived from mineral chalk or limestone, while coloured chalk sticks are made from calcium sulphate in its dihydrate form, CaSO4·2H2O, derived from gypsum.[6][7] Chalk sticks containing calcium carbonate typically contain 40–60% of CaCO3 (calcite).

https://en.wikipedia.org/wiki/Blackboard#Chalk_sticks

Wikipedia cites the following articles:

[6] "How chalk is made – material, making, used, processing, procedure, product, industry". madehow.com. Retrieved February 17, 2021.

[7] Corazza, M.; Zauli, S.; Pagnoni, A.; Virgili, A. (2012). "Allergic contact dermatitis caused by metals in blackboard chalk: a case report". Acta Dermato-Venereologica. 92 (4): 436–437. doi:10.2340/00015555-1296. PMID 22367154.

The first of these seems more relevant... I'm not quite sure what the second citation adds.

The camera is positioned so that the side window is visible through the front windshield. I think the "black coating" you are seeing is just the interior of the van, and that the entire front windshield is usable.

They seem a lot easier than USAMO problems, or even Putnam problems. I suspect that top students nowadays could easily solve them all in a day.

I was having some difficulty figuring out how Hy actually is translated to Python (and wasn't even sure if it was compiled or interpreted). Eventually I found on Wikipedia the following: > Hy is a dialect of the Lisp programming language designed to interact with Python by translating s-expressions into Python's abstract syntax tree (AST).

Also, looking at the code on Github suggests this compiler is written in Python (see https://github.com/hylang/hy/blob/master/hy/compiler.py).

I kind of wish this was made more clear on the main website. Perhaps, instead of introducing Hy as "a Lisp dialect that's embedded in Python", introduce it as "a Lisp dialect that compiles to Python's AST". The words "embedded in Python" don't make it very clear just how it's embedded into Python. The various ways you can embed a Lisp look very different and have very different tradeoffs.

For example, off the top of my head, I could "embed" a Lisp by writing an interpreter (in C if I care about performance) and letting it be called from Python, perhaps passing in a Python list instead of a string to make it more "native". Or I could "embed" a Lisp by compiling to Python bytecode. Or I could "embed" a Lisp by translating it directly to Python source code. Etc.

Regardless, interesting project!

Also, a curve which gets arbitrarily close to every point in the unit square actually touches every point in the unit square. This is because (by definition) a curve is a continuous map from a compact space (the unit interval) to a Hausdorff space (R^2), and so its image is compact, and thus closed. A closed set contains every point that it is arbitrarily close to.

The Hilbert curve does contain every point in the unit square. It is a limit of curves, and so can contain points even not in the intermediate constructions. This is similar to how the limit of 1/x as x -> infinity can be 0, even though 1/x never equals 0.

It looks simple and elegant to me, and I've only used Clojure a little before. Perhaps you're just not very experienced as a programmer in anything except mainstream languages.

I don't understand why this was downvoted. In case it's not clear: (S)he's saying to split the key into multiple shares that can be used to reconstruct the key if you have a large enough quorum. Then store each share in a different place. As long as you don't lose too many of the shares, you'll be fine. And one baddie is NOT enough to get the key.

Do you know why introductory textbooks don't define the determinant in terms of the exterior product? This is how some "real" mathematicians I've talked to define it. It also is more intuitive (in my opinion) to define determinants as "signed volumes" than some sum of products multiplied by signs of cycles.

The product of eigenvalues definition is also somewhat intuitive to me ("How much does the matrix scale vectors in each direction? Now multiply those numbers together."), but it's harder to motivate the fact that adding rows together doesn't change the determinant, which is kind of important to actually computing the determinant.

If you live in the US, you can go to https://reportfraud.ftc.gov/.

How do you know they didn't have other books about civil rights? Maybe they had an entire library filled with such books, but that wasn't brought up in the article.

Those blog posts are pretty bad. Just read the original paper, https://arxiv.org/pdf/2402.09353. The key section is 4.1.

How fast is Goblins? I've been considering using it for a project, but I'm worried that it would be too slow. Would it, e.g., be fast enough to serve as a database?

Is this not literally a federal crime? It sounds like HP installed a backdoor on its printers to download malware that disables the printer.

If it's not a crime, I think it should become one, and I don't think a terms & conditions document should be able to protect against it (in the same way you can't, e.g., enforce a contract with illegal clauses).

> And in 2024 we still have no competing NLP architecture.

No, we do. State space models are both faster and scale just as well. E.g., RWKV and Mamba.

> Transformers aren't merely some lucky passengers riding the coattails of compute scale.

Err... they are, though. They were just the type of model the right researchers were already using at the time, probably for translating between natural languages.

Yes.

You can also have another model "mentor" a new model you are teaching to speed up training. You don't have to start from scratch with zero knowledge. This is done a lot in what are called distillations.

Is this the same "naked festival" that finally allowed women to participate (instead of being male-only)?

Excludable = you can exclude people from benefiting from it. (E.g., Fixing global warming is not excludable while temperature control in your own house is.)

Rivalrous = more people getting it leads to less for others. (E.g., Software is not rivalrous, but hardware is.)

Instead of saying "wild speculation" and using a lot of charged words, it would be more helpful if you explained why you think it is wrong. This just comes across as you trying to denounce the other side as low status and looks bad--though you're probably right.