JHonaker

Did you post this on the wrong article?

The fact that I can go months without updating a rolling distro and things very rarely break is incredible. Tumbleweed is rock solid. (Personally, I've never experienced a break, but I know one just happened for a lot of people. Granted, I believe it was Nvidia or Mesa drivers at fault.)

Yea, the more I learn about things, the more I realize that everything is defined relative to something else. Even measurements are defined relative to some standard (that beautifully manicured kilogram ball, the speed of light in a vacuum, your ruler's hand width or foot length, etc.) That's both very satisfying and extremely frustrating.

Programming languages can't escape it either; see the tautologies at the top of your favorite programming language (metaclasses, metaobject protocols, etc).

That was a great article.

If I'd read this when I was first learning measure theory, I would have had a much easier time. In fact, it took me an embarrassingly long time to realize that sigma algebras were just the "nice sets and subsets" of things that we can extend measures from finite additivity to countable additivity.

I used to think mathematical objects were somehow "inherent". I was always amazed that people had discovered and proved so many interesting things about them. Once I realized they were often just defined to be the thing that has the property we want to prove something about, it got a lot less mysterious.

Note, I'm not saying we just stop there, or that this is somehow bad. The next obvious step taken by mathematicians is to start removing bits of the objects they study, and try to figure out what's still provable until we get to categories, logic, and start arguing about things like the axiom of choice.

Markov Chain Monte Carlo for program synthesis isn't exactly novel. The most immediate reference I thought of is Josh Tenenbaum's [1].

There's also a lot of demos in WebPPL (web probabilistic programming language)[2] like [3] for the synthesis of 3D space-ships. I highly recommend their associated books on The Design and Implementation of Probabilistic Programming Languages [4] and Probabilistic Models of Cognition [5].

I also highly recommend taking a look at the publications of the MIT Probabilistic Computing Project [6].

[1] Human-level concept learning through probabilistic program induction. https://www.cs.cmu.edu/~rsalakhu/papers/LakeEtAl2015Science....

[2] http://webppl.org/

[3] https://dritchie.github.io/web-procmod/

[4] https://dippl.org/

[5] http://probmods.org/

[6] http://probcomp.csail.mit.edu/

> Currently criticising LLMs on hallucinations by asking factual questions is akin to saying I tried to divide by zero on my calculator and it doesn't work. LLMs were not designed for providing factual information without context, they are thinking machines excelling at higher level intellectual work.

I would agree with you, but they're currently billed as information retrieval machines. I think it's perfectly valid to object to their accuracy at a task they're bad at, but being sold as a replacement for.

The Turing test has been a poor goalpost almost since its inception. It turns out, it doesn't require much to fool people. People thought ELIZA was going to eliminate the need for therapists and psychiatrists.

Hell, ELIZA still beat ChatGPT 3.5 in terms of % of people that believed they were interacting with a human in a recent study! [1] Notice that we only properly identified humans about 60% of the time too.

[1] https://arstechnica.com/information-technology/2023/12/real-...

Unless you live in California, it's not likely even an option to opt out.

I know. I was being a bit mean, but they seem to be implying that more parameters even with worse performance is preferable. That seems backward to me. There is no reason for parameter size to be intrinsically valuable, and I think people are a bit infatuated with it.

> That extra model size will vastly overshadow any worse performance of the models.

...What?

Yes, but a "range of prior distributions" doesn't mean every possible prior distribution. Sometimes, the information in the prior distribution is required to get you to a place where your computational system can efficiently explore a meaningful subspace instead of providing nonsense.

If meaningfully different priors lead to meaningfully different posteriors, you're probably missing something that would either eliminate one of those priors from contention or marry the differing behavior in some unifying explanation/model. Either way is a win in my book; both provide a new direction for research!

From the description of the link in the post:

> The historical claim is that in the last decades, we have seen an overreach of the logic of markets, with markets being defended precisely for their capacity to deal with decentralized knowledge.

Phew, I never really drew the parallel between the current bane of my existence (the trust in the almighty algorithm and the perceived objectiveness of anything with numbers in it regardless of their ability to actually describe the supposed phenomena; okay parenthetical rant stopping early) and the “trust in the market.” But I’ll be damned if I’m ever going to not make it now. I really want to talk at length about how we (machine learning practitioners) have made our own beds with people jumping to conclusions about higher order reasoning and the ability to recall anything used in a training set by being militant about that separation as individuals but failing miserably as communities; and I want to talk about how that has created this perception of exactly what she’s referring to: over-leveraged trust in diffuse, “decentralized” knowledge. But… I won’t. At least not yet.

I’m going to check this book out. Thanks for the recommendation!

> In this respect I think philosophy of mind, epistemology, language, semiotics, and to a lesser extent ethics have crucial bearing on the things we see happening nowadays

Whenever I hit that brushes up against these, I always find myself wishing that introductions to these topics were more easily found. I’m not saying they don’t exist, just that there’s no clear entry point to start learning about the philosophy of mind (or others). It’s like in Math, where you have books like Lang’s “Algebra” which are titled in the most literal way possible. It is simultaneously exactly what’s in the book, but woefully inappropriate for people who don’t already know what it is.

Do you have any recommendations?

Yea, throw the link to it out here. I was about to write something similar, so you’d save me the trouble! ;)

Squeak is certainly slow, but Cuis is moving at a good clip.

J is really great. I spent some time last year playing around with it, Dyalog APL, and some other new array languages like BQN.

I was extremely impressed by the breadth of integrations into different ecosystems that the J community had created (like R and the web tech).

Using the language reminds me of using Common Lisp. There are a lot of things that seem odd now, like how you define new words (i.e. functions), how namespaces work, or how the FFI/system calls work (i.e. !: ) [1]. Kind of like how in CL things are named "mapc", "mapcar", "mapcan", etc. Both kinds of quirks come from the fact that these people were really innovating in new frontiers, and Ken Iverson and Roger Hui just kept on developing their ideas.

[1]: https://code.jsoftware.com/wiki/Vocabulary/bangco for how it works and https://code.jsoftware.com/wiki/Vocabulary/Foreigns for what you do with it.

That's what I was trying to say. Anecdotes are the seeds of hypotheses, and enough anecdotes with well-understood conditions make a study population.

> Science is expensive and moves slow.

I don't know if I agree that science is slow. Certainly scientific consensus is slow though. The churn of ideas at the forefronts of fields is rapid. In my field (machine learning/statistics) I'd say too rapid/short term incentive focused.

I really take umbrage at the idea that science is some purely objective, ideal process. It's messy, and scientists are opinionated and stubborn. Some of the most obstinate people I've met are tenured professors... They kind of have to be. It takes time for good ideas to weather the initial criticisms, persist through replication and testing, and to take hold.

It’s getting better though. The addition of shortdoc, the `marginalia` package (which makes some already present doc text available during minibuffer usage), and the `helpful` package (which again just places already available information in the help buffer) make things so much better. I’ve been using VS Code lately, because the tooling for something at work is so much better, but the lack of introspection and hackability is really frustrating.

> Anecdotes are not science. But if enough people share an experience, sometimes there is more to the story.

Solely relying on anecdotes as evidence is not science, but they’re absolutely a critical part of it!

I'm just getting into woodworking. This has been a consistent source of confusion. It's no simpler with rough lumber. The usable yield of 4/4 (i.e. four quarter, or a one inch thick rough piece) is at least 13/16 of an inch after surfacing two sides.

We actually have an internship program with English graduate students to be tech writers for our data science groups. They're wonderful, and I will sing their praises anywhere I can. Having people whose primary skill is knowing how to write is extremely undervalued in Corporate America. If you're only using technical writers to help with documentation, you're doing it wrong.

I knew there'd be at least one pedant! And I almost altered what I wrote to pre-satisfy you!

Begone pedant, and feast thine eyes on the surreal numbers [0]!

> In mathematics, the surreal number system is a totally ordered proper class containing not only the real numbers but also infinite and infinitesimal numbers, respectively larger or smaller in absolute value than any positive real number.

(Also, go look at the `fil` unit in TeX for a good practical application of this ;).)

[0] https://en.wikipedia.org/wiki/Surreal_number

Based on the number of posts I’ve seen about arena allocators on here the last month or so, I’d say the audience is much higher than nonzero.

There's nothing wrong with concurrently developing something similar. I can almost guarantee there will be something different enough about what you've developed to be considered novel (if you care about publication). If you don't care about publication, then definitely keep going! Hell people are still argue about which implementation of complete specifications are the best (for good reason).

> Say I have a simple table of outdoor temperatures and ice cream sales.

You have more than that! You have knowledge about the world!

> What can the machinery of causal inference do for me in this situation?

Well, (I’m being purposefully pedantic here) you haven’t really asked a question yet. The first thing it can do is help you while you’re formulating one. It can answer questions like, “how can I anticipate how things I have and havent measured will the estimates I’m interested in/making?”

> If it doesn’t apply here, what do I need to add to my dataset to make it appropriate for causal inference? More columns of data? Explicit assumptions?

The first thing you need to do is articulate what you’re actually interested in. Then you need to be explicit about the causal relationships between things relevant to those questions. The big thing (to me) is that particular causal structures have testable conditional independence structures and by assessing these, you can build evidence for or against particular diagrams of the context.

It's only appears tenuous because the subjective choices you have to make when using frequentist methods are made for you by the developer of the method.

It's less comfortable to use Bayesian methods because you have to be explicit about your assumptions as the user, which opens your assumptions up for easier inspection. There's also way less specific information implied by priors than most people think. Informative priors should try to make distinctions between something that's reasonable-ish and something that's essentially infinity (take pharmacokinetics for example, the diffusion velocity of a molecule in your blood stream shouldn't have a velocity near the speed of light in a vacuum should it?). They should not be forcing your model to achieve a particular result. Luckily, because of the need to explicitly state them in a Bayesian analysis, it's much easier to determine if they were properly set.

Prior specification is essentially problem domain-informed regularization where you can actually hope to understand if the hyperparameter is going to work or not.

I never really realized how much syntax matters until I spent a lot time writing Racket.

I’ve learned Haskell, JavaScript (actually learned rather than the passing familiarity I had before), Smalltalk, and APL since then. The hardest part of all of them has been keeping syntax straight.

Things with very regular syntax like Lisps and Smalltalk were a breeze to get comfortable, but a significant portion of my time learning Haskell and APL were spent learning how to parse expressions and precedences in my mind’s eye. There’s definitely a bit of a “reading the Matrix” moment for both of them that you tend to forget after you get it though.

I’m not saying we should stop trying to develop novel syntax either though. Anyone that has ever tried to use the object system in Racket knows just how painful it can be to not have syntactic sugar for something important.

`(send object a-message anyone?)`

It really depends on what your idea of applied is, but Conal Elliot's work is really interesting.

His paper on Compiling to Categories [1] pitches programming language semantics as a cartesian closed category [2] allows for really cool stuff by mapping typical evaluation semantics to alternatives like building a computational graph visualization, a pretty printer, and imbuing the original program with automatic differentiation capabilities purely through the categorical interpretation of the same program. Essentially, if you can formulate a desired output of the program as a cartesian closed category, you can do it quite easily.

[1]: http://conal.net/papers/compiling-to-categories/

[2]: A cartesian category with is a category with an operation that lets you pair things together into a new object and get out the original parts with eliminators. Think construct a tuple and have the ability to get the first and second elements out of the tuple. A cartesian closed category adds in the ability to model (partial) function application to its arguments via "apply, curry, and uncurry." See https://en.wikipedia.org/wiki/Cartesian_closed_category

I believe you, and I'm coming to this conclusion myself, after being a keyboard warrior for years.

Do you have any sources I can read though? I always find it fascinating when our perceptions are so off like this.

I would hate that. An interface is useful when you can anticipate and remember what things are available and where they are. It doesn't matter if it's a visual UI or keyboard shortcuts. If you start moving stuff around on us, we revolt. (See every UI update on an MS Office product ever.)