Wonderfall

I do actually read a lot about LLM interpretability, and this is my own conclusion (should've phrased it like "they don't seem to have"). I do actually consider this an open question, so I'm a bit confused as to why you think this way - perhaps due to my phrasing (I just had a very long flight), but know that is not the case and I always doubt things. In fact I said after the text you quoted that the question is actually quite open (mentioning reasoning models, but to be honest, it's not exclusive to them, just more apparent in some ways).

I might also clarify (here and probably in my article when I have the time to do so). LLMs "do" build internal models in the sense that, at the same time:

- They organize knowledge by domain in a unified network

- They're capable of generalization (already mentioned and acknowledged at the very beginning of the article)

However these models, while they share parallels with human cognition, lack substance and can't replicate (yet) the deep integrated cognitive model of humans. That is where current interpretability research is at, and probably SOTA LLMs too. My own opinion and speculation is that autoregressive models will never get to a satisfying approximation level of the human-level cognition since humans' thinking process seems to be more than autoregressive components, aligning with current psychology. But that doesn't mean architectures won't evolve.

Do not misunderstand that because I said they're pattern matching machines, that they will be unable to properly "think". In fact, the line between pattern matching and thinking is actually quite blurry.

> From your article and comments, it sounds like the take is something like "humans don't actually reason autoregressively" which could be true, I don't know enough to know, but sort of like saying physics models aren't really how nature works: ultimately LLMs are executable models of the world, it's even in the name.

The conclusion states "Language and thought are not purely autoregressive in humans".

Which doesn't mean humans don't have autoregressive components in their thinking. At least that's my opinion. I don't make this bold statement and I don't know enough to know, too.

> Like a lot of my coworkers analyzing a production bug? I would agree if the statement were that LLMs were underpowered compared to a human brain today

Clearly not in the same way and that was what I was trying to explain with regards to the hallucination issue too. Humans are also learning from proofs, can apply frameworks, etc. there's no denying that. But the internal process of an LLM remains pattern matching and sequential prediction whereas there's more to the human's thinking process.

LLMs are underpowered in some aspects that can't be replicated with autoregressive modeling, but are already stronger in other aspects. That is what I think.

> but I'm not seeing evidence that humans do reasoning in a way that can't be correctly modeled.

Me neither, this is not what my stance is, and I'm actually optimistic about it. I just don't think we should be satisfied with only autoregressive modeling if the ambition is to reach or comprehend human-level intelligence.

I admit my phrasing was poor there, and I got too excited. I will clarify since I don't really disagree with you or what others said (claiming it's the best way is an overstatement).

Well, one could say that neural networks pioneers modeled their ideas on simplified brain structures representations. Modern neural networks have little in common with an actual biological brain, however, the inspiration remains there (even for modern NNs like CNNs). I recall the intent was there too, originally: providing a framework to study biological cognition in the 50s. Then it evolved to become a new paradigm in computer science so that we have programs able to learn and adapt for problems that are formally too complicated for deterministic solutions.

This is mainly a misunderstanding due to the way I phrased it. This is what I think. I know for a fact that is the case for other AI researchers having watched many conferences - "all of them" is not what I meant (I wrote "many other") and we certainly need people to approach problems from different perspectives and backgrounds, since they will benefit from each other in the end. Not going to lie I'm a bit disappointed to see these kind of comments.

I come from a medical science background, where I studied the brain from a "traditional" neuroscience perspective (biology, pathology, anatomy, psychology and whatnot). That the best way is actually to try to recreate it is honestly how I feel whenever I read about AI advancements where the clear goal is to achieve/surpass human intelligence, something we don't fully understand yet.

“What I cannot create, I do not understand.” someone clever once said.

I completely agree. I never said that "intelligence cannot be auto-regressive", I just questioned whether this can be achieved or not this way. And I don't actually have answers, I just wrote down some thoughts so it would sparkle some interesting discussions about that, and I'm glad it did work (a little) in the end.

I also mentioned that I'm supportive of architectures that will integrate autoregressive components. Totally agree with that.

LeCun is for sure a source of inspiration, and I think he has a fair critique that still holds true despite what people think when they see reasoning models in action. But I don't think like him that autoregressive models are a doomed path or whatever. I just like to question things (and don't have absolute answers).

I-JEPA and V-JEPA have recently shown promising results as well.

Yeah, this is mentioned in the article. (The LLaDa paper is even what triggered its writing!)

I guess semantics matter. Language is primarily hierarchical, but its presentation is what's linear. And LLMs mainly learn and work from this presentation; the question is, and one of the main points, whether emerging patterns is enough evidence to show that there's hierarchical thinking.

> There's not much signal here, just basic facts about LLMs and then leaps to very bold statements.

The article wasn't supposed to be informative for people who already know how LLMs work. Like the title said, just wanted to write down some thoughts.

> This is just silly. Humans forget things all the time! If I want to remember something I write it down.

The opposite was never stated. Human memory is of course selective.

> Here is an interesting experiment I use to help people understand next token prediction. Think of a simple math problem in your head, maybe 3 digit by 2 digit multiplication. Then speak out every single thought you have while solving it.

Now a point I'm happy to discuss! The process of solving it is actually quite autoregressive-like, but this is also an example of a common pitfall with LLMs: they purely rely on pattern matching because they don't have the internal representation of what they really deal with (algebra). But we all know that.

The main question is whether LLMs taught to reason actually show that they have this kind of representation. They still work very differently I'd say; even for tasks that seem trivial to humans, reasoning LLMs will make a lot of mistakes before arriving at a plausible-sounding result. Because it was trained to reason, there's a higher chance now that the plausible-sounding result is actually correct. But this property is actually quite interesting once applied to complex tasks that would take too much time and overwhelming for humans, and that's where they shine as powerful tools.

Author here and I welcome the feedback, but I don't really understand your point. My post is clearly not dismissive of efforts to make LLMs reason using CoT prompting techniques and post-training, and I think such efforts are even mentioned. The model remains autoregressive either way, and this reasoning is not some kind of magic that makes them behave differently - these improvements only make them perform (much) better on given tasks.

Additionally, I'm not dismissive of the non-linear nature of transformers which I'm familiar with. Attention mechanism is a lot more complex than a linear relationship between the prediction and the past inputs, yes. But the end result remains sequential prediction. Ironically, diffusion models are kind of the opposite: sequential internally, parallel prediction at each step.

(Note: added note on terminology since the confusion arised by my use of "linearity", which was not referring to the attention mechanism itself. I've read so many papers that are perfectly fine with the use of "autoregressive" for this paradigm that I forgot some people coming from traditional statistics may be confused. Also "based on the last word" was wrong and meant "last words" or "previous words", obviously.)

All that being said, I don't think it's fair to say one doesn't understand how transformers work solely because of semantic interpretation. I appreciate the feedback though!

Unsure why you think the storage access framework doesn't let you grant access to a directory. You can literally try it on a relatively modern app such as Lemuroid: the system file picker lets you grant access to your ROMs folder.

Whole access to the shared storage is deprecated by SAF and scoped storage. That doesn't mean there is no way to achieve the same productivity tasks you could achieve before: it's just that now you're granting explicit fine-grained access to the files and folders your app needs.

MANAGE_EXTERNAL_STORAGE still exists and is now reserved for apps that can justify their file management purpose. Since this is a highly privacy-invasive permission, Play Store requires a review for these kinds of apps.

Stock OS ships security updates on the latest major version. It means that you can only get a given patch level on the same version for a given device. CalyxOS wasn't rebased on Android 12 until fairly recently. As of January 2022 (prior to the Android 12 release), their vendor patch level was 2021-10-01 which means that at the time the OS was roughly behind 3 months in updates.

They were also shipping an outdated version of Chromium (v94) during the same period (this is important since Chromium distributions for both CalyxOS/GrapheneOS are updated through OS updates - and Chromium is whitelisted by the OS to provide the WebView, even if you happen to use another browser). Considering their userbase is privacy/security-conscious, I think they should've been aware they were more vulnerable than stock OS for a while.

Looking at their source code it's also evident CalyxOS is increasingly relying on the LineageOS codebase. Not that it's a bad thing (LineageOS has its own goals but they're not necessarily aligned with security-focused projects), but it's worth noting.

Once again, these management features don't expect an app to bypass the trust boundaries by allowing the user to arbitrarily add repositories. There is a to-be-exposed toggle that allows user profiles to install apps from trusted sources. It wouldn't work with the way F-Droid works. I can't explain more than that.

You have been provided with evidences, but you've been arguing in bad faith since your first comment by misinterpreting the paper. The security model also expects you to download apps from trusted sources because the signature verification is only enforced for app updates, that's a trust-on-first-use model. That notion exists within Android, and you have been given examples.

> In order to ensure that it is the app developer and not another party that is consenting, applications are signed by the developer.

Back to the original question; it is clearly explained why.

Sounds like a you problem: https://developer.android.com/reference/com/google/android/t... https://developer.android.com/reference/android/os/UserManag...

You're welcome to search for mentions of sources and app stores, and to browse the references.

Again, the paper is not in contradiction, you misread it. Not to mention, as explained in the original comment you were responding to, F-Droid weakens the security model for various other reasons.

Any regular app can be considered an installer. Such APIs like the one controlled by requireUserAction which allows seamless app updates since Android 12 are declared in the app in question, and can even allow apps to update seamlessly.

The management features, again, expect the app to represent a single source. F-Droid deliberately chooses to manage multiple sources that can also be added by the user within the same app, thus bypassing these features. That's the way they work and again, this paper is not exhaustive and is not in contradiction to anything that has been said (quite the opposite).

This paper is not exhaustive and there is further documentation on the APIs in question on the Android official website. You can easily guess the problem involved with the security model when the OS expects an app repository to represent a source of trust, but the app in question decides otherwise.

Chromium is a particular case, but is still equally considered an untrusted source unless explicitly allowed. Of course, the security model takes into account that apps can be installed from anywhere. That's why they're signed and they're running in their own restricted sandbox.

You misunderstood what they said. Indeed, Android can have multiple app repositories and this is an integral part of its security model design.

However, for the security model to be respected, each app repository should represent a single source. The device and user management APIs expect that in Android. F-Droid fundamentally bypasses the trust boundaries in that regard by allowing multiple repositories to coexist within a single client.

Not to mention it also results in a terrible UX given that the application IDs are often reused but signed by another party.

Well, surely there are biases in some ways, including some I may not even be conscious of. That being said, I try to follow a fact-based approach: yes, the conclusion is biased in the sense that it was written in a personal context (again, this article wasn't meant to be shared on several platforms like HN), but I can assure you the rest of the article follows this fact-based approach as much as possible. I even mention build reproducibility and Play App Signing: both mentions could be seen in favor of F-Droid (yet the reality is more nuanced than that).

I'm perfectly aware of the FOSS culture and why some want to use F-Droid to promote this. This is just not what I had in mind when writing the article. I simply intended to address some flaws or deliberate choices from F-Droid that I think are in opposition to the practical approach to modern privacy/security. It happens that I think GrapheneOS and modern Android in general follow that approach. Last month, a reddit troll tried to portrait me as a GrapheneOS developer (when I'm an occasional contributor) and used the article to explain how GrapheneOS would be "anti-FOSS". That is so wrong. GrapheneOS is FOSS at its core, and I personally value FOSS solutions. Not sure how someone would extrapolate the opposite from my article.

Knowing that, I then stumbled upon this thread and noticed people digging up random contributions in my GitHub account instead of reacting to the content. Contributing to an alternative project doesn't mean bias or endorsement. Contributing doesn't necessarily mean code, GitHub issues, or money. You could even see this article as a contribution to F-Droid in some ways (but I'm sure they're aware of the majority of the underlined issues, and I don't have the pretention to do better than them). Then again, I deeply think GrapheneOS and Accrescent are paving the way for modern FOSS solutions.

> False dichotomy. I still have the sandbox with F-Droid. This is chaff.

I didn't imply that you wouldn't benefit from the app sandbox by using F-Droid. I meant that the practical approach to privacy should come from relying on the permission model instead of trusting third parties.

In that sense, F-Droid adds very little to the fact that you still have to trust the upstream code with the permissions you're willing to grant.

If you choose to trust F-Droid, that's perfectly fine and I'm not trying to convince anyone to stop doing that. I also won't comment on your statements that Play Store is spyware because I have very little interest in that topic. You're free to believe that, and I respectfully disagree given the great service (whilst not perfect by any means) offered by Play Store.

Thanks for the heads-up! It's also a new account since I've never posted on HN before, so maybe that's why.

(Also I would like to correct myself on my previous comment: I meant "GPG" as the reference implementation of the OpenPGP standard, not "OpenGPG". I was very tired.)

> I understand there's much more to it than just that, it just read funnily to me. (Not meant as laughing 'at' you! Hope it didn't come across like that.)

No worries, irony doesn't hurt.

> Yes, but that's TOFU. Secure in most cases, but if it's used all the time, an attacker will catch today's lucky ten thousand (xkcd.com/1053) who first download a certain app or who just (re)installed their phone.

By authenticity I meant that you can already use apksigner to verify the fingerprint of the signature. For instance, Signal publishes the fingerprint on their website: https://signal.org/android/apk/

Since the APK published on the website is the same as the one published on Play Store, I think this can be a nice way to ensure the package hasn't been tampered with. A properly configured HTTPS server should be the baseline, with CAA and CT to ensure it wouldn't be easy for an attacker to issue rogue certificates for the website. Of course, this is still involving a TOFU model like you said with any CA system in the end.

Therefore, having certificate pinning by default for the app repository should be a nice progress to deter several types of MITM attacks (rather than placing too much trust in the distribution infrastructure). This comes nicely along the app signature system which inherently follows the TOFU model on Android.

Alternatively, GrapheneOS (as a hardened OS) has the idea to ship a database of known-good signature fingerprints for top used apps such as Signal or Element. This would be hard to do for apps that also have a third-party F-Droid build due to them reusing the package IDs in most cases (the OS can also whitelist the signatures for those, but this isn't ideal).

Sorry, of course meant GPG, OpenPGP being the standard on which GPG is based on.

Funny you should mention NewPipe since they discourage users from relying on the official F-Droid repo for the very reason of slow updates, which you don't want for an app relying on parsing web pages that are likely to change. The app itself (at least the regular build provided by the developers) sends you a notification once a new update is available. They also maintain a third-party repository, which again is fine at mitigating some issues of the official repository (while breaking the security model because the OS inherently wants one source per app repository, but at least it's more usable than waiting for the F-Droid build to be updated).

F-Droid can't do anything against "secret code" since they don't/can't analyze the whole codebase. Like said multiple times, they only run a few scripts on the available source code to remove known trackers, which is known to be a poor approach to privacy.

> because there are no restrictions on things that just work with root, donation links for open source open geo data contribution platforms[1], roll your own payment scheme without giving anyone a cut, put ads in it if you like...

Respectfully, that was not my point. The point was that having access to the source code fundamentally doesn't mean much. You can read more about why there since I don't want to open this debate again: https://seirdy.one/2022/02/02/floss-security.html

> As does every web browser, but somehow banking on websites seems to very rarely be intercepted?

Banking apps exist, and are required as a modern 2FA. Since 2021, strong 2FA is a requirement in the EU for banking operations. Mail clients also do this. DANE would be the ideal approach on web browsers. This might be up to a more general debate that doesn't belong here though.

> Wouldn't it be nice if you could actually see what this app sends to Google about you?

It's perfectly nice, and mitming is a great tool to ensure Google (and others) doesn't lie about what data they send.

> Previously you'd add a cert to your OS and you'd be good to go.

Now, this argument is rather moot though, since it's still doable. Not sure that the higher technical barrier would matter much, and most users will benefit from having certificate pinning anyway. Google is not doing this to prevent researchers to do their work. Nothing is permanently a blackbox anyhow.

> Hah, the author just spent ~2800 words criticizing the liberal inclusion policy, missing api target enforcement, outdated (now slightly misleading) permission listings, lagging signature scheme update, and then concludes with "just download the apk from github because it has an Atom feed"! If only f-droid knew that this was the requirement for an endorsement by OP :D (jk)

This is a bit exaggerated. I'm merely stating this is an alternative for "tech nerds", since Android enforces signature verification for app updates, so you once you ensured the authenticity, the source doesn't matter as much. This should be done with apksigner by the way, not tools like OpenGPG.

I didn't feel the need to expand on this.

> It depends.... how much does security matter to you? Is this to the exclusion of all other values?

I rephrased this conclusion, since it was indeed a bit binary for my taste.

This article wasn't meant to be openly shared on public platforms. This was bound to happen, but obviously I was not trying to make an article that should reach everyone. This is fine though, thanks for your comment.

I fail to see how it's fallacious since you're confirming my point.

> I'm not starting with an app, adding my trust of the developer, and adding my trust of F-Droid

You are. You seem to believe they actually read the whole source code when it's not the case: all they do is running their own scripts to scrap known trackers and the like (and again, I must say badness enumeration is a flawed approach), this is far from a stringent process and this would be a very weak approach in any threat model to rely on that.

You still have to trust the app developer, and you'd be much better off trusting the strong guarantees provided by the Android app sandbox anyway. Seems like there's a major disagreement here when it comes to our approach to privacy.

> Thirdly, Android's default app store, Google Play, also trusts a third party by default - Google, who insist on running a tracking rootkit on your computer, which is so much more egregiously invasive than trusting any one app that it renders any comparison with F-Droid moot.

Play App Signing is mentioned in the article.

There is literally a link in the article that takes you to the Debian website: https://wiki.debian.org/Teams/Apt/Spec/AptSign

I might rephrase that since they didn't technically moved away from OpenGPG (yet). It was a proposal at the time.

That's not a sustainable approach as these old APIs will be obsoleted by a future version of Android. The correct approach is to follow modern practices that will guarantee security & privacy. In the case of Termux, they're well aware of the issue: https://github.com/termux/termux-app/issues/2155

This is a similar situation to iOS which has a saner app ecosystem for this reason (in my opinion). That doesn't prevent apps with the same purpose to exist. iSH and a-Shell are examples of that on iOS. UserLAnd on Android takes the proot approach: https://userland.tech/

Author here. I'm neither a GrapheneOS project member or an Accrescent developer. I'm trying to push these solutions because I think they actually do things right. It's really frustrating that the first reaction for some would be to think that my motivation is to push alternative solutions for my own benefit. Let's try to move past this and other irrelevant personal judgments.

Author here. You're wrong, as I spent several paragraphs explaining how Play App Signing works and how it's being enforced for new app bundles. The Huawei app repository has a similar approach to Play Store, and to my knowledge Amazon app repository has always been doing this (and even more nasty stuff: https://developer.amazon.com/docs/app-submission/understandi...).