New F-Droid repository format for faster and smaller updates
f-droid.org
f-droid.org
I almost exclusively use FDroid, it's an amazing project. But I've never seen an app with so many issues
Let's see: search for ssh (with the intent of finding an ssh client, that's not too far fetched?), these are the top matches: HACS Hackspace Access Control System, KOReader Ebook reader, AVNC VNC client, FTPClient an (s)ftp client, Linux command library, MultiVNC, SshDaemon.
So that last one is the only one which effectively has SSH in the name, and then I guess that FTPClient would be something I might want to see as well, and then the VNC clients perhaps.
So maybe 'ssh client' then? Yeah no, KOReader still first, then AVNC, FTPClient, SSHDaemon, ....
Ok, I get it, search is hard, and I assume they intentionally didnt want to sort by number of downloads (?), but what it does now is close to useless.
Top results for 'ssh' in Foxy Droid: ConnectBot (SSH client), SshDaemon (ssh server), RSSAid (RSSHub... nothing to do with SSH), Easy SSHFS (Sshfs with ssh client), Trigger (open doors via SSH), FiSSH (SSH authentication via fingerprint), TeamBot (SSH client), SimpleSSHD (SSH server), etc.
Being buggy is only one aspect, the other was the annoying change of UI that presents apps in rows of large, gawky icons instead of the previous efficient rows of title text each headed by a small icon (as say in Windows 'List' mode).
Unfortunately, this leaves me stuck on ancient version F-Droid 0.102.3.
When looking for an app this new large-icons mode is much less efficient, especially so if one's in a hurry and doesn't know what the icon represents (as one would expect with a new program).
I'd have thought the large-icon disease would have remained quarantined within the Microsoft and Apple worlds but unfortunately the contagion's now spread to Linux and even infected the F-Droid app on Android. It ought to be resisted on the grounds that there are so many new icons about that their once great benefit has been deprecated (text again conveying a deeper understanding).
I've no objection to those who find large icons an acceptable work framework but developers shouldn't change UIs just on a whim. If they so desire a change let them provide it as an option—i.e. a UI revamp should allow users to select the old one as an option (we've all seen this UI nonsense before with MS Windows, the F-Droid app is about the last place I'd have expected to get infected).
This story has subsequently appeared on HN, it seems absolutely apt to include it here as it's directly relevant to the F-Droid UI matter: https://letterstoanewdeveloper.com/2023/03/06/understanding-...
When so many other clients have sprung up in response to F-Droid's painfully awkward UI one would reckon they'd rethink the change. This is yet another another instance of where developers think that they know best or where they impose their own personal preferences on users but get it completely wrong.
I note that it's not just the number of newly-available clients that have solidified my view, I was very surprised by the number of upvotes I received to my post. Seems many of us think alike.
F-Droid people please take note.
[1] https://www.f-droid.org/en/packages/com.machiav3lli.fdroid/
This is the reason I use NeoStore to install apps like NewPipe on my family members' phones. Before, when Newpipe broke they wouldn't know how to update it, now it just stays updated all the time.
0. https://www.xda-developers.com/android-14-new-apis-app-store...
I also don't think an alternative-to-the-alternative is the right solution. I didn't even know NeoStore or Aurora existed before now, and frankly I don't want to know the difference. F-Droid is already extremely niche, compared to the Play Store. It needs to play every strong card it has available at hand, rather than proliferate the fracturing.
I haven't rooted an Android device since 2014.
I just did some lazy research for the version of LineageOS I'm using. Am I missing something, or is it normal to find the procedure confusing and somewhat scary? There is github.com/LineageOS/Superuser and github.com/topjohnwu/Magisk, it seems I need to build Superuser from source (yikes); Magisk from F-Droid wants to install a newer version from Github (but that doesn't work); the version that works wants to patch a ROM file, which I don't know where to get from... The app lists changelogs rather than instructions.
I understand it's partly the fault of Android's security architecture, but I wish it was actually manageable for normal people.
The remaining solution was Magisk. Magisk was equally easy to install, but recently deprecated that easy install method so that only patching the ROM file remains (you get that file from the ROM you run, unpack the archive, if you are unlucky you need to unpack a second payload.bin file, https://wiki.lineageos.org/extracting_blobs_from_zips#extrac...). But not only is that more complicated, it also does not persist after a Lineage upgrade, which happens every week, see https://github.com/topjohnwu/Magisk/issues/3820#issuecomment.... Totally crazy for a project not in early alpha.
I want root to properly manage my battery via https://github.com/VR-25/acc ( with ideally https://github.com/MatteCarra/AccA as Magisk module). How ridiculously complicated that is now and that the install procedure is broken without them documenting it soured me quite a bit on Magisk, but I also start to develop some bad feelings about Lineage - because it is them that would be in the best position to provide a simple root solution. But they just refuse it, and even forbid discussing such topics on their subreddit - that alone shows me that the project is very user hostile currently. I doubt with that attitude they can make a turnaround, but I'd be happy to be wrong...
Sadly the other bigger ROMs are to my knowledge not better in this category, neither /e/ with their service suite, nor DivestOS with the passionate developer nor CalyxOS or GrapheneOS with their "security" focus are interested in giving the user full access to their device. Android seems to be a dead end here, real Linux is needed.
Somehow not surprising to me. Many large open source projects seem to be more about pushing a political agenda of some sort, rather than serving the interests of their users. Especially distributions, as a class of projects, are uniquely positioned to do that, as the role of a systems integrator doubles as the de-facto gatekeeper.
> Sadly the other bigger ROMs are to my knowledge not better in this category, neither /e/ with their service suite, nor DivestOS with the passionate developer nor CalyxOS or GrapheneOS with their "security" focus are interested in giving the user full access to their device.
I have no opinion on any of these except for GrapheneOS, but I totally understand why they wouldn't want the user to root their device - having root is indeed a security nightmare. I wrote about this more extensively in the context of traditional UNIX-like operating systems. I think it's a very good trade-off IF you provide the user with viable alternatives (let's remember, the original goal was to have unattended upgrades).
https://news.ycombinator.com/item?id=32971677
> Android seems to be a dead end here, real Linux is needed.
That was my reflection in 2016, when I changed to SailfishOS full time, and remained the case in 2019, as I switched again, to iOS. I keep an Android device or two around just to keep testing the waters, but for my personal goal of having a usable phone without Google, indeed looks like a dead end.
(I vouched for your comment, it was hidden, that's normal for new accounts.)
I've used Fdroid for years, downloaded and updated apps hundreds of times across multiple devices and never encountered a single thing you described.
Death, taxes, and people confidently generalizing from one-off cases of highly unusual experiences.
>I didn't realize the app was still being worked on.
By my count there have been three new updates to F-Droid just this year that were not alpha updates, and if you use F-droid these updates show up. If you use F-Droid, I don't know how you could possibly look at your app updates and not realize that is indeed being developed unless you're in the 99th percentile of highly unusual use cases. And if you're an unusual use case you shouldn't be generalizing from it.
I've used F-Droid for years and my experience matches the gp. I've read many reviews saying similar. Friends to whom I've recommended it have told me similar in person.
It seems yours is the highly unusual experience here.
But let's just observe the claim that F-droid supposedly isn't being updated. You can see that F-droid is being updated if you... look at which apps are being updated. Is looking at which apps are updated also a 'highly unusual experience' ?
I'm sure they work on it in the background and the upgrades do something, but parents experience is still normal and correct.
One of the apps that F-droid updates, is F-droid itself. F-droid has been updated three times this year. When it is updated, these updates are listed both in the notifications and the update section. Someone who claims that F-droid is not being updated is therefore an unreliable narrator.
This clearly shows, to anyone paying attention, that F-droid is indeed being updated, and I don't understand how your comment is responsive to any of this. It's only an internet comment sections that parent commenters experience "is normal and correct" without acknowledging or responding to any of this. This is important because it highlights the respects in with your comment in the parents comment lack of fundamental connection to objective reality and therefore are not reliable diagnoses of what is or is not happening with F Droid.
If looking at notifications in the app is something you're going to dismiss as an unusual experience, then you just don't know what you're talking about, and you shouldn't be positioning yourself as reporting on the normal case of what everyday users experience.
> Someone who claims that F-droid is not being updated is therefore an unreliable narrator.
It seems like you're trying to invent arguments to get mad at as no-one made this claim. The original commenter said two things, neither of which amounted to "F-droid is not being updated". They said:
1. That they "didn't realize the app was still being worked on" - this is just referring to their own perception of updates, not making any hard claim about actual work being done.
2. That it has "been years since the UI update" - this is referring to one specific large update a number of years ago, not to the small incremental updates since.
There's nothing else in their comment claiming lack of updates.
The main claim in their comment is that it's buggy, which can easily be true of an app with frequent updates.
I'm not sure if F-Droid is the buggiest app, but aside from the above 100% percent bug, had all the other issues the previous user mentioned.
If your connection has 0% packet drop then you may never experience an issue
(I write this as someone who has experienced nearly every bug mentioned in this thread.)
When they're attempting to present their experience as the typical or characteristic experience, it is indeed helpful to contrast it against other experiences.
1. Every so often I turn on my tablet or phone and see a notification that fdroid has crashed, asking me to submit the information (I used to, I don't anymore). I wasn't using fdroid recently, I don't leave it open in the background; the only background task it has is fetching update lists and downloading new versions to install later. If it crashes, it loses the downloaded updates.
2. Occasionally fdroid crashes on start, providing the same notification as above. Force-stopping the app and removing the activity from recents is the only way to get past this--if I force stop and restart from recents, it crashes again. If I remove from recents and don't force stop, it crashes again. I'm guessing whatever parameters it launched with are causing the crash to happen again.
3. The background service frequently stops while updating package lists (not while downloading packages). The notification remains. This is regardless of whether the update is manual or automatic; there is no foreground update. At 11:00pm I regularly have a non-dismissable sticky notification, last updated at 5:00am, that fdroid is updating repos. Tapping it does nothing, I have to open the fdroid app, which may or may not crash on opening. If it crashes, the only way to reset the notification is to force-stop it. This is also the only way to open fdroid after such a crash (see #2). While I have no confidence that the crashing issue is fixed, it sounds like the new index is intended to prevent repo updates from taking this long. Personally, I'd prefer to just download an SQLite file directly from a server and just dump that in the app to query.
4. After downloads are complete, the download list frequently doesn't update. I can initiate the install from either the page for that specific app or from the download notification, but the main downloads list just shows that it's still downloading.
5. Not really a bug so much as missing feature: there's no way to do partial update checks. No way to check for updates to a single app without downloading the entire repo. I know there's an update available for X Y Z apps and want to update them on the bus, but because #3, 30 minutes is likely not long enough to do so. There's also no way to check a single repository for updates, and trying to bypass this by toggling that repo off and back on causes fdroid to update all repos in the order they were added. If I want to install an app from a new repo, I have to wait for all my other repos to finish updating first.
6. I have ~10 repos in fdroid. I regularly get a notification that the format is incorrect or the server couldn't be contacted or some such. It doesn't tell me what repo that is. My only option is to go through and toggle all my repos off, enable them one at a time, and
I've mostly given up. I can't get the latest update for apps from the website because fdroid doesn't make that available, I can't install updates automatically because I'm not rooted (and thus can't just trust that it'll eventually succeed in installing). I used to run my own fdroid repo for these apps; I don't anymore, because it's not worth the effort to try and get fdroid to install them.
It's not because I'm using some obscure hardware or ROM; I've had these issues with: Nexus 7 lineage, Nvidia shield lineage, Moto G6, Moto G6 lineage, Samsung A32 stock, Samsung Galaxy Tab S6 stock. Fdroid is usually the first thing I install after getting a new device or after rooting a device; I've had fdroid crash on the third time starting it on a brand new device, before I've done more than enable adb to install fdroid.
*This is _not_ a one-off case, this is an ongoing issue that I've had for years at this point*
Instead, for the few apps I actually care about keeping up to date, I use Obtainium, which checks GitHub releases (among others) instead of fdroid (it can check fdroid's website, but that's not kept up to date). As a bonus, this also allows me to keep track of apps that aren't included in fdroid (mostly Tachiyomi forks). Obtainium can't check for updates for individual apps without fetching and parsing the entire catalog, because fdroid only provides a full catalog (see #5)
As for your numbered points, I would say that a occasionally experienced 1, and never experienced two, three, or four.
I've used F-Droid on a Moto G, a Moto G3, Moto Z 2 3 and 4, a Moto E, a Nexus 7, a Pixel C tablet, across numerous versions of Android and Lineage OS. I've installed and uninstalled F-Driod dozens of times and installed and uninstalled hundreds of apps over several years across somewhere between a half dozen and a dozen devices. I've probably used at most 3-4 repositories.
So, what does all of that mean? Well, nothing other than that I can't confidently generalize from a one-off anecdote. Even if I'm really really personally convinced that my anecdote is the most important anecdote to end all anecdotes.
The problem is that internet comment sections are driven by psychological forces that don't necessarily translate into data from which it's safe to draw broad generalizations.
I usually have to restart my phone, uninstall the app, and reinstall again from scratch.
(Edit to add:) maybe the problem is that you're not keeping it updated? As per many replies I don't recognise the buggy mess you describe.
App install + bg updates have been an non-issue to me[1].
This seems to have resolved the self-auto update issue I had as well. I can now see the F-droid client page from within the F-droid client itself.
It's great to see F-droid client updates. The app can be clunky at times, but they are doing great work with maintaining the repo. I feel like a better client experience would draw in and keep more users, so every step in that direction is appreciated.
Be sure to scroll down to click "Download APK" below the version that you want to download, and not the giant "DOWNLOAD F-DROID" button right above it.
This was a bit confusing on mobile before I came back and scrolled down the page some.
This is because the FDroid website does not expose the latest version of apps. The website itself is a static Jekyll website, and I've never found any information to indicate that the app listings aren't part of that, apart from them being missing from the source repos (which they would be if it's generated from the index at runtime).
F-Droid is one of those projects that adds so much peace of mind and usability to my computational life. It is a must-have.
How does that describe “this key nolonger exists”? Does it perceive ‘key: null’ as being the same as a key not existing?
Edit: oh, it’s in the RFC. I just missed it:
“ This design means that merge patch documents are suitable for describing modifications to JSON documents that primarily use objects for their structure and do not make use of explicit null values. The merge patch format is not appropriate for all JSON syntaxes.”
So it remains simple without any sort of DSL needed to describe changes like “remove this key” by somewhat hijacking ‘null’. Seems like a footgun if you happen to use ‘null’ as a possible value for your schema.
To be fair, using "null" as a value in your schema without handling the cases where a given key is missing (and therefore the value is null) is also a footgun.
I’m of two minds. I like how simple it is. But they’ve basically ignored a problem because the solution is complex. And so they’re hijacking a data type in a rather perilous way. Maybe they should have specified a magic string value that is treated as “DELETEME” ;)
That depends on the interpreter: A good implementation of json distinguishes between defined and undefined keys in the object. Think hasattr(obj, "key") in python, or undefined != obj["key"] in js.
Honestly, it seems to me that pre-strict-mode JavaScript actually made a fair attempt at making "undefined" a genuine missing value, except that then implied semantics people disliked, like an undefined variable reference having the value of "undefined" instead of being an error. Lua seems mostly OK having an actual nil, though. (There is admittedly some gnarliness, like a vararg function f being able to distinguish f(1,2,nil) from f(1,2).)
To see Torsten and others working on replacing my crufty-passed-their-use-by-date ContentProvider code with a modern Kotlin + Room implementation is heart warming. If any of you are reading this, please accept my deepest sympathies for having to pull that code out and rework it - that is not something I would have enjoyed doing! It is even better that it all lives in libraries that other clients can adopt if they choose so they don’t need to reinvent the wheel.
For those interested, yes, I am also responsible for writing the bulk of the code for the “new” (now several years old) UI in the offical client which often gets maligned on HN (and this thread is no exception). At the time I did my best to fight off edge cases and quirks of the Android system. F-Droid needs to ask Android which apps are installed and what their signatures are, then cross verify that with its own database to tell you whether updates are available. It also needs to know whether new apps have been installed outside of F-Droid since last time you opened it, etc. It also needs to pass downloaded .apk files to the system and request for them to be installed, then wait for confirmation from the system. The API’s Android provide for this kind of work, but they always seemed flakey, unwieldily, and slow to respond. The whole experience is full of race conditions, and each bistro seems to handle it ever so slightly differently.
I am still proud of how much we managed to achieve, and I’m also very pleased with the fact we were able to do so much great work around internationalised metadata, screenshots, encouraging donations to app developers, etc. But I’d be lying if I said I wasn’t still a little disappointed that I couldn’t iron out all of the kinks.
Life circumstances meant that I drifted away from the project and no longer contribute (other than via Liberapay donations to the wonderful fdroiddata team). However I would love to revisit the project in the future to see if I can address some of those edge cases to ease the user experience for all of us. You may have noticed that for such an amazing project, used by many people (not just the client, but the entire infrastructure), there has been comparatively few contributors. Despite that, I am still extremely fond of the value they bring to the Free and Open Source ecosystem.
You're of course welcome to contribute again! The hard part is that app stores are large and complicated apps, when done fully, so that makes it hard to contribute to. We do mark issues with "first-timer" https://gitlab.com/fdroid/fdroidclient/-/issues/?label_name%... and "help-wanted" https://gitlab.com/fdroid/fdroidclient/-/issues/?label_name%... if anyone is looking for a place to jump in. I think we can also see this in all the various other clients like G-Droid, M-Droid, Foxy, Droid-ify, NeoStore, etc. Many rapidly stop being maintained, and others leave out key functionality like localization and automatic mirror selection because it is a lot of work to implement. The new libraries should make it a lot easier for forks to implement these features.
Is it just that they're required for using CursorLoader, so that you can easily load things in the background without having to create a custom Loader implementation? And they're basically just a fairly inconvenient database abstraction?
But yes, you are correct that they are indeed an inconvenient database abstraction.
The only thing I will say is that despite the crummy abstraction, we did put the effort in to have good test coverage of each ContentProvider. It took a while to get the infrastructure up and running so that we even could test them, but once the plumbing was added, it became simple to add new tests to ensure they worked as expected.
This is important when you need to take into account things like "Get me all the apps, but filter on category, and then also limit those Apps to ones for which they have at least one Apk which is installable on my hardware, meets my AntiFeatures requirements, and then also pull back data about whether there is a version that can be upgraded to or not based on currently installed apps. I fell in love with the SQLite explain output. I found it really good at explaining what was going on with these mildly complex joins - much easier than the MySQL explain output I was familiar with.
[0] - https://gitlab.com/fdroid/fdroidclient/-/blob/b3773a156121cf...
Neither SHA-1 nor SHA-256 are signing constructs, they are hash functions. They provide integrity, not authenticity.
For this kind of thing (a central authority signing and public clients verifying), you'd need a public key signature scheme, like ECDSA or EdDSA.
For encryption you usually pick a large random key for symmetric encryption, encrypt the body with a strong symmetric cypher and that key, and encrypt the symmetric key with your asymmetric method and the intended recipient's public key.
For signing, you use a suitable secure hash function to produce a digest of the content, when sign the digest using your private key. Choice of digest function is as important as your choice in encryption method as an exploitable issue that allows you to generate the same digest for different content means that the signature, while secure in itself, is worthless because what has been signed is no longer effectively deterministic.
tl;dr: So when they say “digest algorithm for repository signing” they don't mean that they are using the SHA256 result as a signature, but that they use SHA256 to generate the digest that is then signed.
And if they are computing a hash for that anyway, the computation saving from signing the hash not the whole content might be worthwhile?
Granted, verifying only that hashes match for integrity would be cheaper than checking the signature. But since you need to to that anyway for authenticating the payload, why the intermediate step?
If it has no other purpose, then yes but only a small amount of overhead if the content is large because the internal hash is over a much smaller content. You have:
hash(<thousands/millions/more>bits) then hash(<hundreds>bits) & sign(<hundreds>bits)*
instead of
hash(<thousands/millions/more>bits) & hash(<hundreds>bits)
The bit you save, the hash(<hundreds>bits), is only significant if the input content is of similar size and you are performing the operation many times in quick succession.
> since you need to to that anyway for authenticating the payload
That is true from the client PoV. Server-side you might want to integrity check your data as part of regular proactive maintenance.
Of course I'm assuming here that the digest is stored so it can be used in this way – if not then I agree it is just overhead.
Well that's why they said "stronger digest algorithm". As far as I understand, the repository index is hashed (now using SHA-256 instead of SHA-1) and this hash is then signed as usual, using whatever public key signature scheme they are using.
If the hash is weak, an attacker may be able to construct compromised data that hashes to the same hash, and the whole signature becomes worthless.
I was able to get the new version by searching for F-Droid and going to the "versions" section and click update on the latest version
Then again, to download anything (including screenshots and other app info, let alone the app itself) you need a server connection so it's only mildly useful to not have to ask a server. And the search algorithm is... nonexistent I think is the most correct word to use. You get what you ask for. The top three hits for "camera" are Telegram and two identical looking SIP clients, of course. (There's not being influenced by algorithms and there's this.) I wonder if te project would accept a better algo if I'd contribute one; I've made one before at work for a not dissimilar dataset so I have some notion of what works, though it's not rocket science anyway.
That said, apk diffs seem important as well, some updates are significantly large (more than 100MB) and see frequent updates. Although in that case it might need to cache apks which requires space as well (I would opt in, though!).
This isn't something like a news reader.
I think homebrew moved away from repository approach too.
Your assumption on how it works is flawed to begin with. Check the comment from one of the original authors https://news.ycombinator.com/item?id=35003844
I think most F-Droid contributors are using Google-free devices these days, so that's where it works best, especially when built into the ROM like CalyxOS, Lineage-for-microG, etc. Unfortunately, that means we pay less attention to how things run on Google devices. So if you're running F-Droid on a Google device, please be sure to report issues so we are aware of them!
I like to make a point of keeping my software up-to-date, but waiting for F-Droid to download/uncompress/analyze the repository index has annoyed me so much I've stopped doing it whatsoever, unless I make a conscious decision to actually do it and invest time time.
Looks like they switched from downloading the entire JSON on updates to downloading a patch
I appreciate it, but it's a weird flow. I'd expect touch "update" tab would run the "check for updates" without needing the pull down (maybe set a cache-time in settings); "update all" would do all the updates without any other taps (or with at most one [per app]).
So... An rss feed?