If the domain knowledge is on the code, you can only refactor. If the domain knowledge is on the team, then rewrite tends to be faster.
Most rewrites fails because the team rewriting is not the team that did the initial development.
Looking at https://github.com/owncloud/core/graphs/contributors, most of the initial core contributors contribute now to Nextcloud instead.
I am interested in seeing how the rewrite affects end-users. Currently, it seems to be mainly focused on File Sync and I don't see things such as calendar or contacts management. (they totally could appear in the future though)
I'd assume that limiting the scope of components, makes a rewrite also easier. Nextcloud for example has a ton of hooks that allow you to write apps to customize the behaviour (want users to sign the ToS before downloading a share? Should be doable etc.). When you leave these out, implementing new things just got a whole lot easier :) (so I guess I am envious on the fact that they got rid of a ton of backwards compatible APIs to maintain :-) )
Disclaimer: Contributed to ownCloud for a few years, then to Nextcloud.
Personally, I am looking into kopano or etesync to replace calendar and contacts. We already embed kopano konnectd as the OpenID provider so, there are synergies.
We do maintain backwards compatibility for the WebDAV, OCS and OCM apis and the clients are all working. The inner hook system of oc 10 obviously is gone, just as well as the internal PHP APIs. Instead we now rely on the CS3 APIs that describe a set of services in a microservice architecture to extend or customize functionality. Something that has always been fragile, because a bug in an oc/nc app could affect all other services.
Furthermore, we did not have to start from scratch but contributed to reva, which has been powering CernBox for a few years. It is the reference implementation for the CS3 api.
So, we are very much Protocol driven, which makes changing the implementation language possible if needed. And everyone is free to implement his own service replacement if he needs to. The extensive acceptance testsuite will tell you what is not working compared to oc 10 or ocis. For now, we are pretty happy with go and the initial benchmarks.
One more thing regarding core developers having moved from oc to nc. The server is just one piece of the puzzle. None, literally, none of the desktop, iOS or Android developers paid by ownCloud moved to the fork. It feels as if they were not even considered important. The sync protocol and the end user experience on his daily driver really is what matters. I am very happy that we as ownCloud finally grabbed the chance to really tackle the file sync and share part with the right tools for the job. Without being pulled in a gazillion of directions. I cannot express my gratitude for the rest of the oc sales and support team that keeps oc 10 well running and bought us the time to actually make this step. Truly exciting times
Keep in mind this is still tech preview. There is still a lot of work to do. Helping hands welcome. And yes we need to work on our communication skills...
Full disclosure: oc employee no 7 and ocis tech lead here. Originally hired to work on full text search in ownCloud 10. Good old times. Cheers to all the oc and nc devs: it should be possible to implement the internal PHP API of oc/nc using roadrunner. That might allow existing PHP apps to be wrapped in a dedicated microservice. But that is not something we are investing time in. Feel free to ping me on https://talk.owncloud.com
Currently, all our clients sync using webdav. Changing the protocol would require rewriting that part of every client.
Another reason is that ownCloud as well as ocis are used to access files that resides in other existing storages, eg. S3 or CEPH. Translating those protocols to a file based sync is a lot easier than adapting syncthing (just my gut feeling).
Another aspect is that the file based sync is also state based. We can use the etag to detect changes and immediately start syncing in a breadth first approach. We are not doing that, yet. But the windows cloud vfs we implemented for the desktop clients makes this one of the next steps.
The sync in ownCloud feels slow because the clients are polling. There is currently no persistent connection that the server could use to push changes. But to be honest we have been bitten by firewalls between server and client so often that I doubt we can do real push notifications. Which is why I personally am aiming for long polling. For mobile devices we have to rely on the existing notification infrastructure from Google and apple anyway. Anyway, there are ways to speed up sync. Which switching to go made a lot more implementable. But one step after another.
We already implemented TUS for uploading files in ocis and all clients. It is a well designed extendable upload protocol that covers a lot of corner cases we have experienced first hand with all our clients. We contributed our experiences and are planning a batch extension that we would use to group lots of small files in a single upload. Delta sync should be a TUS extension as well, IMO. But that is mostly in my head.
First, we want to get the basic file sync and share features fully implemented.
So ... Yeah ... Changing the language and the architecture kind of made a knot disappear and we are moving forward.
In that regard, I personally have an eye on the Microsoft graph api which we could use for file management. Input welcome.
Anecdotally I've been recoding from scratch one of my applications at work (converting from python to rust) and while it's a lot of work and it put new features on hold in the meantime I'm fairly optimistic that it's going to be a big win in the long term. Not only will the Rust code perform better and be more maintainable, the rewrite lets me address architectural flaws with the previous solution, some caused by lack of foresight and others by tacked-on features that weren't expected when the code was originally designed.
The longer I work in this industry, the more I’m convinced it is the dominant software development paradigm.
Search for “jwz cadt” to read more (can’t link it here).
Messaging between systems is hard. When things are async, it's harder. When there's async concurrency, it becomes spicy.
I'm managing a Nextcloud instance at the office. Even with this relatively mature state, it can bork itself pretty decently. Also, it has separation of responsibilities and a system admin is not an admin the traditional sense so, a lot of things are hidden between logs and other layers.
Complicating this further by adding microservices is a brave idea at least. We're going to see what's going to happen.
I hope they succeed but it's not a piece of cake.
Gecko was a technical success.
On the non-technical side, it isn't at all clear that Netscape-the-company came out worse than it would have done if it had tried to stick with the buggy Netscape 4 rendering engine.
https://ubuntuforums.org/showthread.php?t=1398220&page=11&p=...
Chrome was absolutely faster in web rendering than Firefox and more stable and the UX was cleaner, it made Firefox with its slow buggy performance and awful theme's look like myspace to Chromes facebook.
I also remember being very frustrated with Firefox on Ubuntu ~10.04 at the time and when Chrome came along it was exactly what I'd been waiting for.
Quantum/Servo/Rust/UI refreshes seems like this was Firefox catching up to Chrome v1 and honestly in 2020 it feels like Firefox has finally caught up significantly.
It's still not as stable as Chrome but it's getting there, webrender is a massive leap forward in performance:
https://testdrive-archive.azurewebsites.net/Performance/Chal...
That test is now showing at 6.65 seconds down from 45 seconds without it.
Regarding microservices I don't know any OS projects that have failed because of that (because there aren't many OS products that use a microservices architecture), but personally I witnessed many corporate software projects burn and fail when trying to rewrite an "ugly" but functional monolith as a collection of microservices.
The thing is that the energy which you invest into rewriting your entire software in a new stack could also be invested into improving your existing stack. LAMP is not dead and actually evolving quite well, so there are definitely ways to improve the stability and speed of a LAMP application without rewriting it in Golang (even Facebook still uses PHP, albeit with some modifications). Having written extensive applications in both scripting languages and Golang I can say that the latter has its advantages but is also slower to develop in and in many respects not as agile as a scripting language like PHP or Python.
Doing a rewrite of a popular project with thousands of deploys in the wild will also force you to split your attention between keeping the old system running and building the new one. Since most of your users will want to migrate from one system to the other without losing their existing data and configuration you'll have to ensure that there is a clear migration path that works, which can be quite hard. Also, all external systems that are interacting with existing installations (e.g. via APIs) will also need to be supported by the new system.
In the end it might be possible to make it, but in my humble experience I think a complete rewrite is almost never a good idea and the energy you put into it is often better invested in making gradual improvements to your existing codebase. But that's just my 2c of course.
There are legitimate reasons to rewrite a system in a new language, it's not a a decision that should be taken lightly though. From a business perspective it's also very expensive, as having a team of 5-10 developers rewrite a system that already exists and works can cost millions of dollars, so you should have a really good reason and clear ROI objective when deciding to do this.
Many years ago CERN chose ownCloud over other solutions because of the state based sync. They could use that to let researchers sync petabytes of data residin on EOS, their custom built storage solution. That ownCloud supports custom storage implementations made this a lot easier.
The did suffer some database bottlenecks and decided to extend EOS with features that the sync clients needs: tree modification time propagation (so that the etag of the root changes when anything in the tree changes) and size tree accounting (so you can see how many bytes are hidden in a folder including all children).
The basically maintained a fork that was half PHP, half c++. Together, we added APIs and interfaces to the codebase to make the file cache implementation exchangeable.
They went ahead and implemented a golang service that could serve the api requests, while the web ui was rendered in PHP. It should be possible to dig into the details by looking at the public CernBox repo. Code archeology ;)
Anyway, all that was before the nextloud fork.
To be honest, CERN has tried to convince ownCloud to switch to a different architecture for years. With some long held opinions leaving the company we were free to reassess our options.
ownCloud has long had problems with long running operations being killed by eg. php-fpm timeouts. So I was evaluating and comparing different PHP frameworks like reactphp, swoole and amphp. I wanted to be able to offload workflows that are triggered after an upload has finished to a proper background job. And I really wanted to stay in PHP land because of all the already written code and existing apps.
But I noticed that they all had one thing in common. The all reimplemented a redis and a mysql library which made me wonder why?The existing drivers would block network IO, killing every concurrency gains you could gain by using a reactive framework or the go like coroutines of swoole.
It finally dawned on me that PHP may not be the best language to implement a service that has to deal with file IO. A systems language is more fitting.
Go or rust? Two years ago that was way easier to answer. Furthermore, CERN had a working server side API of the ownCloud webdav and ocs endpoints written in golang.
We sat down and discussed how o file sync and share solution would have to look like at the protocol level. What services are necessary and how could we make the existing code more modular to support other storage backend a than EOS? How can we get rid of the centralised database?
The result were changes to the CS3 api, making reva a reference implementation and using ocis to tie it together with user management openid connect and thinking about migration strategies. There is still the possibility to wrap PHP in a sandbox like service using roadrunner.
I would not have dared to start from scratch. But with an existing codebase that was used in production the decision became easier. The story is not over, yet.
Yes, we are leaving things behind. But we can embrace new things as well. And I am happy to be able to work with an awesome team to see this through. Every helping hand is welcome.