CodeFlow: Improving the Code Review Process at Microsoft (2018)
queue.acm.org
queue.acm.org
CodeFlow's workflow was nice, but everything else about it was _awful_. Capturing iterations, comments, threads, etc was the good stuff. The UI - eye-bleeding and amateurish. Network operations - terrible.
Despite all that, the workflow was so good that I'd take it over github CRs in a heartbeat.
When i was there - Codeflow had the following feature
- Multiple reviewer states ... accepted/reviewing/awaiting changes/decline to review/reject change. - Multiple comment states ... pending, won't fix, fixed etc. - Iterations based code review flow - after you review - you come back easily view the next iteration (relative diff) - you can really compare back and forth and track comments through iterations - github tends to loose comments through commits - Support for bots to comment like stylecop etc
Codeflow workflow felt like an MS-Office application like Word or Outlook compared to Github - Heavy UI based workflow with no API/commandline counterpart sometimes - Desktop app - not website - Lots of knobs/states - Very easy to work back and forth in the UX
I agree with an earlier comment however - the UX was pretty ugly - the workflow account for lots of states Github doesn't but there is a nice linear flow to the progression of a github conversion that (at least at the time i was there) . IIRC it was a managed .NET - WPF application which are kindy clunky - not as nice as a website with clear links etc.
CodeFlow was also integrated into VSTS, whose review tooling is very much inspired by CodeFlow. Reviewers could use their web browser or the client app, and it all fed into the same system.
Most of the features that set it apart from Github initially have been copied by now - comment threads with resolutions, individual reviewer states, etc, have all been recently introduced; the gap between CF and GH has shrunk substantially.
That said, CF still does a better job of retaining comment threads and associating them with lines of code. Maybe it's just different UX decisions by the product teams, but it makes a difference when reviewing complicated patches.
Pre-Github, projects would need to fire-up their own Bugzilla or Fogbugz instances, set-up all the project tracking criteria (areas, iterations, severity, etc).
But unstructured label tags seem to work surprisingly well in practice: projects that would have otherwise demanded 20+ fields for work-item tracking now seem fine with issue labels.
So a ready-made setup at an existing service open to multiple projects would work. I guess that's what Sourceforge was.
Sourceforge could have been github, but when a company turns on its users while failing to deliver value it won’t last.
Because git is decentralised it means all the git-hosting services become commodity, and we’ve learned that having the best UX keeps people around - even if your service is commodity, so I expect this basis will ensure GitHub (now part of Microsoft) will stay in the community’s good-graces.
Bingo. It surprises a lot of people how little you need to be successful if everyone's on the same page. I've worked places that had thousands (yeah, thousands) of apps, dozens or hundreds of millions line of code, and had little to no issues with collaboration (oh, they thought they did, but compared to almost any other company, they really didn't), and process/tooling involved github issues, markdown files for docs and little else. Sometimes they'd toss in trello in the mix and that's about it.
You definitely can do better, but as a baseline, it's more than enough.
I have to agree, here, wholeheartedly. Put a bug in that would break the build? No worries, as soon as you kicked-off a CodeFlow, it would start a prod and smoke-test build for you. Everyone could also see those breaks and block the review from completing, until you fixed them in a new iteration.
You would think that, with everyone moving (or having moved) from Source Depot to VSO/Git, CodeFlow would've falled into the deprecated products pile but because it was such a worthwhile utility, teams for entire products carried it forward, rather than relying on GitHub's CRs[1].
[1] - It became more of an amalgamation between the two [Source Depot and Git-basd-VSO] but CodeFlow was your one-stop shop to view it all.
This sounds exactly like PRs with status checks for protected branches (usually master)? Edit: Link with UI screenshots: https://help.github.com/articles/about-required-status-check...
- Easy to see the changes an author makes over time to a review
- Integrates with the Package and Ownership system to automatically assign reviewers
- Easy work on changes that touch many files
- Inline lint and code coverage results
- Built in display of unit test and CI results and overall coverage %
- Draft mode to delay review until tests & lint are clean
- Comments move properly as you make changes
- Easy to mark comments as done and find outstanding ones
- Comment on multiple lines instead of a single one
Things that are missing:
- No ability to comment on single characters
- Cannot mark certain files as important
- No way to explicitly mark a change as "simple" (like a rename)
- No web UI to make quick fixes (compared to GitHub is what is most missed)
Theoretically if there were a standard for managing all that data within the git source graph, you could implement this solely as a VS Code plugin.
https://marketplace.visualstudio.com/items?itemName=GitHub.v...
The extension is open source and the extension points it uses are all relatively well documented. I'm almost surprised there aren't more extensions doing similar things. (I don't believe that even the Azure Repos extension currently supports an inline PR experience like this.)
My team is fairly happy with github enterprise and the PR workflow but some things about CodeFlow sound better: being able to comment on a single character in a line or share comments between disparate code; and, having a local (non-web) client that runs faster.
Here's an example of how things usually are at MS. I worked at MS awhile ago. At that time Git was just being developed, and MS used it's own VCS based on Perforce (which is pretty good, I should say for a centralized VCS). Code reviews were done over email. Kind of like Linux kernel development. If you wanted somebody to review your code, you used a tool to pack your changes into a *.cmd file which contained a diff itself and a Perl (!) script to unpack and view the diff. Then you sent this file to a reviewer over email, and he just ran it on his machine to see the changes you had done. This tool was simple, limited in features, and brilliant. It was hacked by somebody on the Windows NT team back in the 90s. Even if you couldn't unpack a diff into your repository checkout, you could still open the pack as a text file and review the changes, or manually apply them, since it was just a freaking text file.
Come the time MS decided to migrate everything to .NET. Some team at MS similar to TSE responsible for development tools decided that the old Perl tool was inadequate. You don't need no stinking Perl scripts! They developed a replacement in C#. Full of features you can't live without and maybe even an integration with the dev environment (I don't remember). The new tool was distributed in the binary form. The code changes for review were packed into a binary format with plenty of metadata in addition to the actual diff. The problem? Only this tool knew how to unpack this binary format. It worked most of the time and when it worked it was great. But when it didn't work, you were screwed. If it couldn't unpack your code change for any reason, it would throw some C# exception. You couldn't fix the exception and rebuild the tool (you didn't have access to the tool source code or knew how to build it). You couldn't extract the change yourself, since it was packed in a binary format only the tool could extract. You pretty much lost your change if you didn't store it elsewhere.
And this is generally how things were (and still are?) done at MS. They have teams of project managers analyzing and spec'ing out every bit of the development process. Then they create extensive tools to support it with elaborate UIs and automation of everything they can think of. But if you need to do something just a little bit differently, you're left out in the cold because those tools are too close and too brittle to adapt to your specific needs.
It sounds like the DevOps winds have shifted in the last few years at Microsoft. Even Windows (!) is now in git and properly dogfooding Azure DevOps (Boards, Repos, et al), according to blog posts and BUILD talks.
The toolkit is very small and lacks good definition of elements (both visually in that it lacks contrast, and practically in that I have no idea what is clickable or editable).
The page structure isn't that clear either, where are the column headings? And on my widescreen browser window the isn't a limit on horizontal stretch - I have to play with the browser to find a good width. Also, I can expand code, but how do I shrink/hide it?
And no, you can't shrink/hide code, other than by re-diffing. What can I say -- it's surprisingly hard to implement and didn't seem worth it.
GitHub: 31M users
BitBucket: 6M users, 1M orgs
GitLab: 100K orgs
The GitLab number tallies with Emily's answer, which gives me some moderate confidence that these numbers are in the right neighborhood. I'm also going to assume that the users-vs-teams numbers are all in roughly the same proportion.
Based one these numbers it doesn't make much sense to target BitBucket unless either 1) they're growing much faster than GitHub (unlikely) or 2) their customers are a much better fit for Reviewable (possible, but it would have to be a truly significant difference in userbase composition). Targeting GitLab doesn't appear to make sense at all (as a business), though perhaps they're growing fast.
https://marketplace.atlassian.com/addons/app/bitbucket/top-s...
Is what makes me uncomfortable about modern webdev. Valuable product work constantly deprioritized by migrations to new frameworks. Magpieing has always been a risk in hobby projects, but in the past there was a smaller torrent of nee things to always be migrating to.
Lots of people have worked for both Google and Microsoft. The article said CodeFlow was good. You said no, Critique is better! But I guess you don’t actually know one way or the other.
Which is too bad. Because it’d be great to hear thoughts from people who’ve tried both.
Nope, I did not. I just said that people really want Critique, which they do. Even CodeFlow itself is evidence of that.
The way we do things is the right way because that's what we do. Any disagreement is ignorance.
but it gets worse. the UI can't or won't do trivial rebases. so either you upload a new, rebased revision, and get the reviewers to re-review (because stale reviews get wiped), or just rebase, push, and mark as merged. i get you are supposed to review rebases, but this can grind a team to a halt. you're waiting for approvals, but by the time you get them someone else has pushed a new change... a colossal waste of time (edit: especially if you require 2 approvals, which i do think is good practice, and some AWS teams do it).
this also means people are trained to ignore the merge button or expect it to be greyed out. but it might be greyed out for other reason though. in GitHub, you have big green check-marks or red icons to indicate which conditions have and haven't been met (reviewers, CI, merge conflicts). in CRUX, hovering over the greyed out merge button gets you one cause at best (there is a page/tab that shows more detail, but almost nobody looks at it).
obviously it depends on your team, but it requires near constant vigilance to maintain high quality with CRUX, especially with a big team or less experienced devs contributing. process is supposed to make quality easier, not harder!
there are many other smaller issues, but that's the big one IMO. as for ReviewBoard, personally i prefer Gerrit, which itself doesn't have a great UI, but makes up for it in the workflow.
I personally love the Crux tool and some of its add ons.
As detailed, that only prevents merging via the UX, not pushes to the main branch. Maybe the team does suck, but I still consider it a shortcoming of CRUX et al.
Disclaimer: I work for Alphabet.
In CodeFlow any comment is a floating object with a line connecting it to whatever selection it was created for and you can move it around as you like. You also can easily filter out visible comments by state, author, participation in them.. Features around comparing iterations, marking already reviewed files and similar are the same as far as I remember.
Another reason for my opinion might also be that CodeFlow is probably better optimized for large screens, but this might be caused by the weird Google standard of 80 column limit for source code which I will just never understand in the 21st century where everyone has 24" and up full HD to 4k screens.
I am comparing state of both from 2015 where I worked with both in the same year.
even if you have a big screen, it's still useful to be able to fit two or three files side by side on a single screen. 80 chars might not make sense as a hard requirement for, eg, a c++ codebase, but it's still a worthy ideal to shoot for.
Amazing tool. Great integration with version control, code search, continuous integration.