samsartor

The absolutely dumbest way is to compare the number of parameters with your bytes of RAM. If you have 2 or more bytes of RAM for every parameter you can generally run the model easily (eg 3B model with 8GB of RAM). 1 byte per parameter and it is still possible, but starts to get tricky.

Of course, there are lots of factors that can change the RAM usage: quantization, context size, KV cache. And this says nothing about whether the model will respond quickly enough to be pleasant to use.

Worth clarifying that you are talking about information content, not entropy. A single text file or png has information, the distribution of all possible text files or all possible pngs has entropy.

I'm not an expert, but let me brainstorm a bit here. Something closer to the specific correlation might be what you want? In vague terms it would measure how much the various bytes of a file are related to each other, by considering how much more likely they are to be given values taken together vs considering each byte individually.

But once again, having extremely high specific correlation might indicate a trival low-complexity example? I'd have to play around with it some more to get a good intuition for how it behaves.

Edit: It seems like this might also be very sensitive to parametrization. The specific correlation in terms of byte values would not be much more useful than the information content, because the marginal distributions aren't very interesting (eg over all files, how likely is byte #57 to be 0xf3 or whatever). It would be a better measure with more meaningful variables, or even something like a markov chain where you consider many short substrings.

Anyway, specific correlation (like information) measures a specific file. The expected specific correlation over all possible files gives the total correlation, which (like entropy) measures the whole distribution. Total correlation is also the KL divergence between the joint distribution and the product of the marginal distributions! Total correlation is also the same thing as mutual information, just generalized to more than two variables. And specific correlation is the generalization of pointwise mutual information.

None of the images load for me. The site is https but the srcset attributes all use http. Ironically the src attributes are correct.

Removing impl Drop is like removing a function from your C API (or removing any other trait impl): something library authors have to worry about to maintain backwards compatibility with older versions of their library. A surprising amount of Rust's complexity exists specifically because the language developers take this concern seriously, and try to make things easier for library devs. For example, people complain a lot about the orphan rules but they ensure adding a trait impl is never a breaking change.

The video gets into that. The main purpose is to verify that the binary you're running came from the actual source code.

Dependent typing is mainly just that feature of Typescript that `x: number | null` becomes just `x: number` inside an `if (x != null) { ... }` statement.

If you dig deeper into the type theory, it gets really interesting and complicated because types themselves can start to include any arbitrary code. And that's where Idris comes in! But after doing a whole project on Idris in college, I agree it is way outside the overturn window. Python and typescript rightfully keep it simple.

I think it's also generally understood that Nvidia owns the training space, but not the inference space. They have a lot more competition there and the margins are smaller. More people running AI models is still good for them, but a drop in the bucket compared to the money they were making from training clusters being built out. And I think everyone just realized they can probably make do with their existing clusters.

Good point. I agree actually, many people do put the work in to understand the whole stack. But one person could not have built the whole thing themselves obviously. All I was trying to say is we already live with superhuman intelligences every day, they are called "teams".

Nope, not my point. My point was that even if we get superhuman AGI, the effect of self-improvement may not be that large.

I think we are already way past single-human intellence. No one person understands (or could possibly understand) the whole system from the silicon up. Even if you had one AI "person" a 100x smarter than their coworkers, who can solve hard problems at many levels of the stack, what could they come up with that generations of tens of thousands of humans working together haven't? Something surely, but it could wind up being marginal. Exponentially smarter AI meets exponentially more difficult wins.

All past data shows is exponential growth in the cost of AI systems, not an exponential growth in capability. Capabilities have certainly expanded, but that is hard to measure. The growth curve is just as likely to be sigmoid-shaped. Just a phase transition from "computers process information strictly procedurally" to "computers use fuzzy logic sometimes too". And if we've exhausted all the easy wins, that explains the increased interest in alternative scaling paths.

Obviously predicting the future is hard, and we won't know where this stops till we get there. But I think a degree of skepticism is warranted.

That could be the VAE? The "latent" part of latent diffusion models is surprisingly lossy. And however much the image is getting inpainted, the entire thing gets encoded and decoded.

Edit: I'll note some new models (SD3 and Flux) have a wider latent dim and seem to suffer from this problem less.

AI generated images are also biased strongly towards medium lightness. The photographer's adjusting of the tone curve may simply give it that "look".

A lot of current inpainting models have quite a lot of "signal leak". They're more for covering stuff vs removing it entirely.

Ironically, some older SD1/2-era models work a lot better for complete removal.

I tried Axum recently and had a fairly bad experience. Incomprehensible trait errors, generics all over the place, and mind-numbing async glue code. Salvo wound up being way more straightforward for my (admittedly simple) use case.

I haven't written a Rust web server since before async/await, back in the Actix days. Axum is definitely an improvement on that. But I think the async ecosystem still has a long way to go.

GNNs have been a bit of a disappointment to me. I've tried to apply them a couple times to my research but it has never worked out.

For a long time GNNs were pitched as a generalization of CNNs. But CNNs are more powerful because the "adjacency weights" (so to speak) are more meaningful: they learn relative positional relationships. GNNs usually resort to pooling, like described here. And you can output an image with a CNN. Good luck getting a GNN to output a graph. Topology still has to be decided up front, sometimes even during training. And the nail in the coffin is performance. It is incredible how slow GNNs are compared to CNNs.

These days I feel like attention has kinda eclipsed GNNs for a lot of those reasons. You can make GNNs that use attention instead of pooling, but there isn't much point. The graph is usually only traversed in order to create the mask matrix (ie attend between nth neighbors) and otherwise you are using a regular old transformer. Often you don't even need the graph adjacencies because some kind of distance metric is already available.

I'm sure GNNs are extremely useful to someone somewhere but my experience has been a hammer looking for a nail.

Reminder that Rust does not automatically schedule anything. Unless you _explicitly_ call `tokio::spawn` or `async_std::spawn` you are still living entirely in state-machine land.

Rust's `join_all` uses `FuturesUnordered` behind the scenes, which is pretty intelligent in terms of keeping track of which tasks are ready to make progress, but it does not use tokio/async_std for scheduling. AFAICT the only thing being measured about tokio/async_std is the heap size of their `sleep` implementations.

I'd be very interesting in seeing how Tokio's actual scheduler performs. The two ways to do that are:

- using https://docs.rs/tokio/latest/tokio/task/join_set/struct.Join... to spawn all the futures and then await them

- spawn each future in the global scheduler, and then await the JoinHandles using the for loop from the appendix

As other commenters have noted, calling `sleep` only constructs a state machine. So the Appendix isn't actually concurrent. Again, you need to either put those state machines into the tokio/async_std schedulers with `spawn`, or combine the state machines with `FuturesUnordered`.

Anyone who works closely with diffusion models knows the "world model" thing was hype. Diffusion models are really good at faking, starting with nonsensical large-scale structure and forcing it towards plausibility with lots of nice details.

Not for all intents and purposes.

If we are asking whether MOND is useful, then the answer is probably yes. You might use it for simulations of galaxy formation where Newtonian gravity is considered a reasonable approximation today. But MOND is not a correct model of the universe. There is no place in the universe that Newtonian gravity applies, only places where the error is an acceptable trade-off for simpler calculation.

https://en.wikipedia.org/wiki/Tests_of_general_relativity#Pe... is the example I learned in school. You don't need to be around a black hole for GR to suddenly switch on.

Newtonian gravity is an approximation. A perfectly acceptable one in many contexts, but still measurably incorrect.

TeVeS is definitely interesting, but it still has problems like you said. AFAICT gravitational wave observations are particularly bad for TeVeS theories. TeVeS isn't dead, but if dark matter theories are criticized for being patched up post-hoc, that standard should also apply to modified gravity.

My hangup with MOND is still general relativity. We know for a fact that gravity is _not_ Newtonian, that the inverse square law does not hold. Any model of gravity based on an inverse law is simply wrong.

Another comment linked to https://tritonstation.com/new-blog-page/, which is an excellent read. It makes the case that GR has never been tested at low accelerations, that is might be wrong. But we know for a fact MOND is wrong at high accelerations. Unless your theory can cover both, I don't see how it can be pitched as an improvement to GR.

Edit: this sounds a bit hostile. to be clear, I think modified gravity is absolutely worth researching. but it isn't a silver bullet

Instead of psql, I really like https://github.com/dbcli/pgcli

I think I'd be willing to accept software patents if the patent was required to include complete source code. Basically, if it acted as delayed open source.

Ostensibly the patent system exists to incentivize open publishing of new technologies, but it isn't worth squat if full of vague language written by lawyers.

For problem #3, I think you can also prove it with perspective. The 3 circles are identical unit spheres at different distances from the viewer. The spheres are connected by 3 infinite cylinders. The cylinders form a triangle, and so lie on the same plane. Under perspective, the plane has a vanishing line on which each cylinder's vanishing point must lie.

Imperative rust is really good for parsing, but you can also get a long way with regexes. Especially if you are just prototyping or doing Advent of Code.

I do still like declarative parsing over imperative, so I wrote https://docs.rs/inpt on top of the regex crate. But Andrew Gallant gets all the credit, the regex crate is overpowered.

My advisor also worked on this ML project for estimating electron density and temperature within tokomaks: https://www.cs.wm.edu/~ppeers/showPublication.php?id=Ozturk:...

Technically that counts as "AI in nuclear fusion", but it isn't any sort of breakthrough. In almost every case the effects of AI are marginal. Not zero exactly, but nowhere near the breathless hype.

Unfortunately first-past-the-post was on the ballot in a lot of states this year, and absolutely crushed ranked-choice: https://ballotpedia.org/Ranked-choice_voting_(RCV)#Ballot_me...

I live in Colorado and couldn't be more pissed off. We had a shot at viable 3rd parties and blew it.

Right now the only arguments to Future::poll are self and the task scheduler context. Any borrows of mutable state have to be captured by self, which means only one future with access to the state can be inflight at a time. If `&mut State` was an argument on resume, then the actor could schedule many futures simultaneously, which each getting temporary access to the state whenever it makes progress.

Actix does exactly that, but it needs a bespoke ActorFuture trait with extra poll arguments. But until https://lang-team.rust-lang.org/design_notes/general_corouti... gets worked through, there is no way for actix to support async/await.

Actors in Rust still kind of suck because there are no first-class resume arguments. In order to provide a `&mut State` you have to either:

- handle messages sequentially per-actor (like Ractor, and like Bastion in practice)

- create your own future trait without async/await (like Actix)

I keep hoping the language team will get coroutines off the ground, but it hasn't happened yet.

Sorta, in classical single-bounce ray tracing you still need to cast explicit shadow rays from diffuse surfaces. Path tracing gives shadows for free, because it is simulating global illumination rit large: ambient occlusion, shadows, bounce light are all special cases.