samsartor

No, the probability distribution is the same. Watermarking changes the rng sequence used to pick from that distribution.

I want to agree with this. My fear is that governments and corporations love using technology to launder evil shit. Once security is automated, entire groups of people can be quickly and effortlessly disenfranchised by flipping a bool on a server somewhere. No appeal, no recourse, no human faces. You get to find out from some LLM that you aren't allowed in the country anymore because the "algorithm" identitifed you as a risk. uwu so sawd

It only works if the "intellence-driven security" is being developed in good faith.

The premise of this post is that tech founders used to be admirable nerds, but have since changed. I wonder if it isn't the other way around. We're the nerds. Us. Here. We used to admire tech founders because sometimes they were nerds too, but then we changed. We grew up. We got wise to it.

The author wants founders to stop projecting “an obsession with wealth and power” and instead “focus carefully on projecting an obsession with core nerd values”. And maybe it doesn't occur to them (as a fellow nerd) that _wealth and power were the whole point_. The author enjoyed being blind to the greed of it all, and now being unable to unsee they are begging the founders “please please just pretend a bit better”.

Elementwise equality! Given two dataframe columns or ndarrays, users often expect `==` to give out a column or ndarrays of bools (like `+`, ``, `*, `&`, and just about every other binary operator).

Personally I think it's fine to use "diffusion" to refer to the whole family of models

Yah I think E2E-TTT is a lot more like what people in this comments section are picturing. I can't tell that this method updates model weights at all during the "sleep" period, only the usual SSM state updated by any Mamba model after each token. They just optimized the model to use that SSM state _more_ when an eviction is about to happen.

The abstract and method sections only mention updating the SSM state during "sleep" (ie the same vectors that change after each token in stock Mamba) not any of the actual weight matrices. AFAICT this is just another attention compaction paper, with misleading tile? It is not very clearly written

Under the hood, the StartWRT UI is just another OpenWRT package, and it plays nicely with luci.

I helped a bit to develop this UI myself. Support for vlans was baked into it from day 1. The idea being good admin/guest/iot/hosted/etc separation without extra access points.

My understanding is that humans have very limited genetic diversity compared to most other animals, because of the population bottlenecks we've been through. And further, that diversity is mostly between individuals, not between groups. The distinction is easy to see in cats vs dogs: they both have similar overall genetic diversity but two Chihuahuas have virtually all the same genes (the small angry ones) while two tabby cats are more distinct. The two cats have different combinations of big/small nice/mean smart/dumb, but the genes average out to the same "typical" kind of cat in both cases.

Because humans get around so much, and because we think interesting-looking people are hot, the diversity is spread pretty broadly across the whole population. The average european person and the average east asian person are a little bit different genetically, but way less different than any two real europeans or two east-asians are to one another.

In short, the distributions of individuals overlap so much that the trendlines are pretty close to useless. And historically speaking, the people who tried to make a hard distinction out of those trendlines had awful motives.

I think that this is something of a misunderstanding. There isn't a litteral RISC processor inside the x86 processor with a tiny little compiler sitting in the middle. Its more that the out-of-order execution model breaks up instructions into μops so that the μops can separately queue at the core's dozens of ALUs, multiple load/store units, virtual->physical address translation units, etc. The units all work together in parallel to chug through the incoming instructions. High-performance RISC-V processors do exactly the same thing, despite already being "RISC".

I've been working on a reactivity system for rust over the past couple of years, which uses a lot of these ideas! It also tries to make random concurrent modification less of a pain, with transactional memory and CRDT stuff. And gives you free undo/redo.

Still kind of WIP, but it isn't secret. People are welcome to check it out at https://gitlab.com/samsartor/hornpipe

https://ruffle.rs/ is pretty solid

I feel like my papers are better for having gone through peer review, and I'm a better researcher for having had a few rejections. Of course the reviewers can't hover around in your lab watching everything you do. But even if reviewers can't check the validity of the evidence in your paper, they do a pretty good job ensuring that the claims you make are supported by the evidence you present. That's a valuable if imperfect guardrail! What would be the alternative?

Several people in my family have a MTHFR gene mutation that screws stuff up, including causing problems with anxiety+depression. But a simple B12 shot every couple of weeks does wonders.

Yes. Pretraining and fine-tuning use standard Adam optimizers (usually with weight-decay). Reinforcement learning has been the odd-man out historically, but these days almost all RL algorithms also use backprop and gradient descent.

I'm skeptical that we'll see a big breakthrough in the architecture itself. As sick as we all are of transformers, they are really good universal approximators. You can get some marginal gains, but how more _universal_ are you realistically going to get? I could be wrong, and I'm glad there are researchers out there looking at alternatives like graphical models, but for my money we need to look further afeild. Reconsider the auto-regressive task, cross entropy loss, even gradient descent optimization itself.

In a library of all possible strings, this is just text compression (as the other comment observes). But in a finite library it gets even simpler, in a cool way! We can treat each text as a unique symbol and use an entropy encoding (eg Huffman) to assign length-optimized key to each based on likelihood (eg from an LLM). Building the library is something like O(n log n), which isn't terrible. But adding new texts would change the IDs for existing texts (which is annoying). There might be a good way to reserve space for future entries probabilistically? Out of my depth at this point!

My old team shipped a web port of our 3D modeling software back in 2017. The entire engine is the same as the desktop app, written in C++, and compiled to wasm.

Wasm is not now and will never be a magic "press here to replace JS with a new language" button. But it works really well for bringing systems software into a web environment.

The simulatable stuff is almost perfect. It's the stuff that can't be simulated that fails.

Take the last flight as an example. The booster experienced what was (probably) a structural failure in the propellant fuel lines. Simulating stress in the structure under static conditions is quite straightforward. Simulating the stress as the rocket ascends vertically and the tanks empty is hard, but doable.

Simulating the dynamic loading as the rocket flips? The fuel sloshes around, the sloshing fuel changes the kenimatics of the rocket, the kenimatics of the rocket change how the fuel sloshes, the engines try to correct adding a new force, the thrust from the engines creates increased force on the fuel increasing the pressure to the pumps, the performance of the engines changes because of the new fuel flow, that alters the acceleration further causing fuel to slosh, gass bubbles are intrained in the fuel from all the sloshing thus altering its flow/sloshing behavior, valves open and close creating pressure waves in the fuel that travel up and down the fuel lines (the water-hammer effect alone being enough to burst the pipes if valve closing is not well-timed), and the rocket itself flexes as all this happens, testing every exact detail of the manufacturing which you have to go out to the factory and physically measure. No simulation software ever imagined can handle all that coupling of systems.

The usual solution is to make some conservative estimates (the center-of-mass of the fuel will move by at most some amount, bubbles will last at most some time, the engines will have so much control authority, etc). But that requires experience. And this is aerospace, so safety margins are tiny.

And remember, optimization problems can be _incredibly_ high-dimensional. A 7B parameter LLM is a 7-billion-dimensional optimization landscape. A grid-search with a resolution of 10 (ie 10 samples for each dimension) would requre evaluating the loss function 10^(7*10^9) times. That is, the number of evaluations is a number with 7B digits.

This also tracks with my experience. Of course, technical progress never looks smooth through the steep part of the s-curve, more a sequence of jagged stair-steps (each their own little s-curve in miniature). We might only be at the top of a stair. But my feeling is that we're exhausting the form-factor of LLMs. If something new and impressive comes along it'll be shaped different and fill a different niche.

Except for some niche Janko-layout keyboards like the WholeTone Revolution and Lumatone, isomorphic pianos haven't every really caught on. However, isomorphic layouts are very common on accordions! I have a chromatic button accordion at home and credit it with making music theory finally "click" for me. On my fiddle I can play in keys of G,D,A (and the relative minors) quite easily but struggle on weirder keys and can't handle chords at all. On the accordion it couldn't be easier!

I did read it, all the way through! It's really good. The part you are quoting is setting up the ELS, which does not memorize entire images due to the inductive biases of a CNN (translation symmetry, limited receptive field). But the equivalence to a patch moseic is still due to the assumption that the loss is perfectly minimized under those restrictions.

And I was impressed by the close fit to real CNNs/ResNets and even to UNets. But what that shows is that the real models are heavily overfit. The datasets they are using for evaluation here are _tiny_.

Edit: oh the talk is here btw, if anyone is curious https://youtu.be/c-eIa8QuB24

I have mixed feelings about this interpretation: that diffusion models approximately produce moseics from patches of training data. It does a good job helping people understand why diffusion models are able to work. I used it myself in talk almost 3 years ago! And it isn't a lie exactly, the linked paper is totally sound. It's just that it only works if you assume your model is an absolute optimal minimization of the loss (under some inductive biases). It isn't. No machine learning more complicated than OLS holds up to that standard.

_And that's the actual reason they work._ Undefit models don't just approximate, they interpolate, extrapolate, generalize a bit, and ideally smooth out the occasional total garbage mixed in with your data. In fact, diffusion models work so well because they can correct their own garbage! If extra fingers start to show up in step 5, then steps 6 and 7 still have a chance to reinterpret that as noise and correct back into distribution.

And then there's all the stuff you can do with diffusion models. In my research I hack into the model and use it to decompose images into the surface material properties and lighting! That doesn't make much sense as averaging of memorized patches.

Given all that, it is a very useful interpretation. But I wouldn't take it too literally.

Deepness in the Sky is one of my favorite books of all time! Fire Upon the Deep is a serious let-down by comparison, but the wolves are a cool concept.

As I've gotten older I've realized that I have very little in common with Vinge philosophically. But he was a person who thought very deeply, and it shows.

I have a crate with a "perfect" derive macro that generates where clauses from the fields instead of putting them on the generic parameters. It is nice when it works, but yah cyclical trait matching is still a real problem. I wound up needing an attribute to manually override the bounds whenever they blow up: https://docs.rs/inpt/latest/inpt/#bounds

I'm in ML-land. I thought we were all hopelessly tied to conda. But I moved all my own projects to uv effortlesly and have never looked back. Now first thing I do when pulling down another reseacher's code is add a pyproject toml (if they don't have one), `uv add -r`, and `uv run` off into the sunset. I especially like how good uv is with non-pypi-published dependencies: GitHub, dumb folders, internal repos, etc.

I always assumed "gaussian splatting" was a reference to old school texture splatting, where textures are alpha-blended together. AFAIK the graphcis terminology of splats as objects (in addition to splatting as a operation) is new.

I'm doing my PhD in ML shit. Before that I was a systems programming guy, lots of C++, bit of CUDA, big fan of Rust. On the side I'm obsessed with RISC-V. Own a couple of boards. I made a stupid little cuda-like-compiler on top of the RISC-V vector extensions, just for fun.

What I'm saying is, tensorrent couldn't find a more excitable third-party developer if they grew one in a lab. And you know what? I can't make heads or tails out of all their various abstractions. I've tried! I've read the docs, I've read the examples, I've gone to meetups. I think OP is right that "one more abstraction bro" probably doesn't solve the problem.

At a guess, the problem isn't a technical one, it is an organizational one. They don't have anybody to stand in for me, or devs like me (eg dumb people). There is no product leadership on the API design. Just a lot of really brilliant engineers obsessively tuning for their own usecases, unwilling to ever trade-off a hit in performance or expressivity for readability or writeability.