Microsoft’s Azure DevOps: An Unsatisfying Adventure | Hacker News Reader

Microsoft’s Azure DevOps: An Unsatisfying Adventure

toxicbakery.github.io

260 pointsjedikv191 comments

PM for Azure DevOps here. We've been investing heavily in our user experience and our CI/CD experience, so I'm sad to see that we've disappointed here.

Some of these complaints I would agree with - in particularly, we're not (yet) caching build resources - though we're working on this now. But most of these complaints I was quite surprised to hear; not an experience I would want someone to have or what I see from the majority of our customers. So this is certainly a place where I'll follow up with the author to learn more.

Funny I got an email from an Azure PM after I stopped using the free tier asking if I ran into problems. I explained my issue and what I gave up on, and.... never heard a response. I thought that was kind of weird, why even ask?

They ask the question, but any response is prohibited unless they cycle it through a pr department. It's about public narrative, and how it changes sentiment. So unless that response pays them, it's better to avoid saying anything than saying the wrong thing and pissing someone off, especially the wrong influencer.

(Former Azure DevOps PM here) That's not quite true - did you hear that from someone? When I was there, I had pretty much full freedom to talk with customers, as long as they hadn't opted out of contact. Never had to talk to PR or have emails looked at.

Sorry that you didn't hear back though :(

What does "opted out of contact" mean? I wonder if I am missing some communications because I unchecked some "don't send me spam" checkbox now!

That usually means the "unsubscribe" link at the bottom of your email. Clicking that is usually quite permanent.

I think there are likely a few places - i.e. the unsubscribe link (@Larrik), a button in the sign-up flow, and probably also somewhere in settings.

You should reach out to some PM's on Twitter! I know many are fairly active, and I'm sure would love suggestions or feedback.

I dunno if you guys have gotten feedback on this, but the difference between build artifacts, artifact feeds, and the output of some pipelines' tasks in terms of artifacts is really confusing.

I know it's probably getting translated to yaml from the old way which was a bunch of forms with build options, but we had some trouble with this. At the time, yaml releases were also still in development/preview mode.

One thing I'm going to miss is being able to discover things through the build wizard. Now it's just a blank yaml file and I feel that hurts the initial impression. I'd really like an option to build yaml files using the old way, but still get the yaml output for version control, that way I have all of the options and their potential values immediately available and I don't have to memorize and type everything.

There's a bunch I like about Azure DevOps though. Hopefully I haven't missed something really obvious.

Azure Pipelines PM here. Definitely hear you on the current YAML experience. We're working on updates to the editor now to allow you to quickly browse and insert task snippets. It's going to be a big improvement.

The old UI was better, and forcing people to this weird pretty UI doesn't make sense.

Several products aimed at tech people seem to make this mistake. We are power users, we don't want stuff hidden away. Stop having UX that's designed as if we were trying to browse twitter.

Looking right at you, JIRA.

If you wouldn't hide certain things within JIRA you would go completely bonkers. JIRA allows you to fully customize the interface regardless.

This is spot on, I've been using tfso or devops, brand of the month. I hate the New UI, its really clear that the people are focus on making something that doesn't need to be pretty, pretty at the expense of functionality.

For example, when you go to do a release and you of course want to do release notes, previously you could copy and paste the related issues output into any document.

Now its been CSSified to be big, bubbly, and utterly useless to do anything but look at, and there is no print or export feature.

The new UI was useful when it was opt-in and we were migrating from on-prem TFS to what was then called VSTS. It made it easy to tell which system you were using.

I'm more used to the old UI but some features are not easily discoverable. I wrote some posts on how to build branches and pull requests as it's not obvious: https://unop.uk/build-and-release-all-pull-request-merge-res...

I also liked the old UI far better. I have trouble finding things and navigating to where I want to go in the new UI. In the old UI it was easier to get an overview and to see where I have to click.

Azure Devops is something I want to love.

The UI keeps changing, but doesn't fix underlying bugs that have been around for ages.

Multiple issues where the builds don't actually update in the browser when they go through the steps (f5 only fixes), the builds show out of order in the UI, and many many other wonky things.

Also releases aren't YAML yet. Builds are, and are a good feature.

Azure Devops is almost perfect, and with a bit of TLC would be excellent.

YAML build definitions are not supported for GitHub Enterprise, which is mystifying to me seeing as how they are supported for GitHub. This is a critical missing feature for us and will leave us having to manually create huge JSON blobs for submission to the API.

@nathanaldensr, I'm an engineer on Azure DevOps. We rolled out support for YAML build with GitHub Enterprise this month.

@thejosh I'm a PM on Azure Pipelines and I'm interested to hear the builds UI ordering and other wonkiness feedback if you want to share. You can email me at my firstname.lastname at microsoft.com and we can do whatever works from there.

I can echo what the author of the blog says, save for private build instances, we haven’t had issues with them.

The user experience leaves so much to be desired. The tools are not well integrated, the UI is really slow, there’s no dashboard view of active pull requests, builds, releases, etc for my favorite repos. Build/Deploy times are insanely slow for .NET Core micro service projects. We have a project with 4000 unit tests (not integration tests) and it takes over 7 minutes to do a full build + code coverage. The same operations locally take a fraction of the time.

> We've been investing heavily in our user experience and our CI/CD experience

I'm on the "new" UI and I'm not seeing it. You've just re-skinned the same buggy and broken cruft that was there before.

It reminds me of the Visual Studio 2019 blog posts about new icons, themes, and fonts as their big new features while ignoring the massive performance problems people have been asking about for years.

DevOps with or without the new branding/UI is the same poorly designed product that doesn't know what it wants to be and is super un-polished. It would be better if you made it really good for one specific user scenario rather than poor for dozens.

Honestly, while your criticisms are mostly fair, your approach is really unprofessional here. Yes, Microsoft is a massive company, but human beings are making this and I’ll guarantee you none of them are happy with where the product is. Please consider showing some compassion.

I’d also add that I use 2017 and 2019 daily, and build by build, I see massive improvements for performance.

I don't see anything at all unprofessional about that reply. It's critical, but it is not unprofessional imo.

Generally the whole area gets a continuous stream of small and big features. You can click through the release notes [1] and see the changes they've made. Over the past twelve or so months they've jettisoned XAML-based builds and added YAML builds.

As someone who's used it through VSO/VSTS/ADO days it's got foibles but yes, it's improving continuously.

[1] https://docs.microsoft.com/en-us/azure/devops/release-notes/

I don't think that's a lie. They have certainly invested heavily in changing the experience --- but unfortunately, and like many other products seem to be these days, just not in the way that you want.

(It's not just you. The "why did they spend all that effort to change things that didn't need to be changed" question has come up for me countless times upon seeing redesigns of other products.)

I kind of wish they'd promote whoever is the UX lead for MS teams to cover other products... it's by far my favorite windows app, second to VS Code.

I spend a few hours recently in Pipelines and the whole UI is confusing. One of the offender being the edit button being literally at the other side of the screen. Outlook.com UI was also recently destroyed so it is hard to trust Microsoft services, even if good, will stay usable in the future.

The whole pipeline debacle is confusing. I recently did a demo at a Microsoft DevDays showing Microsoft hosted Ubuntu 16.04 agents in my build and then a self-hosted vsts-agent container (provided by Microsoft which still has the old name, why?) and part of the point was to show how slow not having caching is, even for a simple Python build pipeline. On top of that Microsoft has some odd pricing in that they limit your self hosted pipeline concurrency to 1 unless you pay them. That seems like a bad decision because most people would probably go through the effort of spinning some VMs natively in Azure which would increase overall OpEx. But instead, like the OP of the article, people will self-host on prem or in another/cheaper cloud. Microsoft gives no incentives to use native build agents (they are unreliable and slow) and almost force users to go self hosted but also charge for concurrency.

I've also had very odd issues where agent pools will have an idle worker but the pipeline will sit saying there are no agents, waiting. Eventually it goes, but... It's just a bad impression when you're showing it and it's not consistent.

Also, the UI. Tis horrible. Take a nod from AWS guys and get rid of the widgety attempt at entertaining some exec who doesn't use Azure. AWS interfaces load consistently for the most part and do a much better job in being not annoying, comparatively.

Another bad user experience is if you write a pipeline, by hand, without the GUI and try to run your pipeline - well, good luck. Authorization for certain components of your pipeline will fail, even if they follow the pipeline documentation to the T. Why? Because the pipeline web GUI workflow does linking of authorization between components on first run and if you only kick your build off using, say a code check-in, the pipeline will just fail. And it will give you a non-descript authorization error until you pull all your hair out and randomly do a commit from the Repository GUI of which it will pop up a message asking you to authorize the thing that had been failing for the last hour. Either document this better or give users a descript error of the actual problem. This, again, is oft where I find Azure lacking.

I am from the engineering team in Azure Pipelines. We agree that the resource authorization experience needs improvement. We have been working on it. We would like to make sure that you can only include the reference to a resource (secure file, service connection, variable group, agent pool, etc) in your YAML file if you have permissions to access that resource. And, it becomes particularly hard to auto-validate this if you are pushing your YAML change directly to GitHub or another repository since we do not understand the identity of that user. If we are not sure, then the build fails with a resource auth error. You can do two things to rectify this - (a) with a recent change we rolled out, you can allow all pipelines to use a resource. This is a toggle that you will see on the resource (b) You can get someone with permissions to open the pipeline in the Pipelines hub of Azure Pipelines, and queue a build. You will get an option to correct the resource auth problem. We also made a recent change so that new resources that you create default to "usable in all pipelines". We are taking steps to improve this experience without compromising on the security of these resources for those that need that capability. More is planned in this area over the next couple of months. Thanks for understanding. I will also update the docs on this topic with all the updates that have been rolled out this week.

Thank you for the great explanation on changes you've made and will continue to make in this area! What is the best way to inform persons such as yourself about UX issues? Appreciate the thoughtful response.

I work on the UX side and you can email me at firstname.lastname @ microsoft.com (it's my username and in my profile).

We use Azure DevOps extensively at my work and, after having used GitHub, Gitlab, self hosted solutions, Jenkins, TeamCity... DevOps ranks dead last.

It's the entire experience. The UI is terribly clunky everywhere. The worst for me are pull requests. Incredibly tough to work with people on a pull request. I can't even point you to "a" particular problem - for us it's broken everywhere. From the weird file comparison with overlays to the missing support for (us) important file formats like ipython notebooks. The way notifications bubble up in a facebook like format on a pull request. The emails that have 20000 lines and never the exact information I need. The amount of clicks to go from comparing two files to open the current version. Other people mentioned the lacking caching support for builds, so I am not going to go into more detail here. Another missing piece for me: Where are my statistics. I'd love to see commit, PR, response, comment etc statistics per team member / teams / groups.

Using Azure DevOps board is a nightmare of its own...

btw: most of these issues are part of the whole "azure" experience. The Azure portal suffers from the exact same interface issues of horizontally stacking fields. One wrong click and your last edits have disappeared into the ether

This. Please please let there be a thorough accessibility makeover, and soon.

I have managed to forget the search terms I entered, but the first thing I have stumbled on, is this :

https://www.w3.org/TR/WCAG20-TECHS/SL15.html

Keyboard shortcuts across Silverlight applications.

I have no idea what became of Silverlight. But I liked it very much more when I understood Silverlight was the future.

(how anyone understood the tech road maps at the time Silverlight was introduced, I can't imagine)

I personally see the problem is that search engines are disinterested in the furtherance of any kind of consistency capable of reducing the utility of a layer of discovery that they can provide. If you explained to your senator, that whoever dominates the browser market, effectively controls how all the information contained in the www is displayed and arranged by publishers, and the standards bodies intent to introduce semantic markup and accessibility across the www potentially can eliminate the need for search engines for a variety of constituencies and professional workers, no less cutting spam most certainly in the process, maybe they'd see the inherent conflict to be as undesirable as I see it to be.

I keep thinking I need to at least spend a while considering how possibly a browser might technically introduce undo / redo for arbitrary websites. This is what I would be trying to accomplish, if I held the necessary influence with the Edge team. Even the ability to be informed whatever it is that I just did to the page, would be enormously valuable to me and everyone I know. I'm going to risk thinking that since Microsoft are contributing to the chrome codebase, and the default supposition is surely that Microsoft wouldn't want any advancement of any features that have desktop equivalence, maybe a diplomatic possibility exists for introducing the semantics and accessibility consistency we desperately need, by a little sacrifice. I would love to be able to hit a key combination inside a Azure page, that thanks to the aforementioned accessibility standards, had a (possibly expanded) command line consistent equivalent and, permitting a little browser specific markup, could, with a single further stroke, break the interaction out of the browser and into powershell. It seems to be the aim for Microsoft to be the universal developer interface for all. This is entirely consistent with the same idea. The number of developers is only likely to be increasingly close to the whole user base of PCs, as voice and other interfaces develop. Get VSC for VB6 on Android and the job's done, in one short generation ...

Same here. Azure has a killer feature in India that neither AWS or Google have - multiple data centres. Because of Indian data residency guidelines, that becomes critical for business continuity planning.

But the Azure UI is so bad that people are willing to forego stuff like that just to avoid being on Azure.

The Azure product teams have asked for feedback from startup multiple times with consistent answers on UI - I'm no longer willing to believe that this isn't the consequence of some global UI mandate to make things look "windows-y"

AWS has a full region based in Mumbai (ap-south-1) with multiple availability zones, each of which consists of multiple data centers.

umm.. yes we are all aware of this.

Its a single geographic region. Its not allowed according to financial or healthcare data regulations in India.

P.S. i personally am working with the "Big 4" consulting guys to get a white paper to work with this. But big missed opportunity for Azure. All because of bad UI.

I just manage cdn and domains with this interface, it is super annoying, slow and complicated, already. I don't want to know how it is with more complex stuff...

The API is probably your only chance..

One additional point... it would be GREAT if I could have a chat channel in MS Teams with just notifications of @user/group where I am a part of that user/group in VSTS. I get so many emails, I really can't/don't keep up with it. Generally, email for this type of thing sucks, and the email doesn't show the meat of what you need to see.

A dashboard interface for messages/notifications in VSTS would be helpful.

Aside, switching to a GraphQL type interface with something that updates your UI consistently would be nice. Some screens update fine, others just get lost/disconnected and you don't even know where you are at.

Overall, I haven't experienced the agent issues in TFA, but I've had plenty of other issues and continue to use the beast that I know. I do think the issue tracking is decent and the integration is as nice as other products I've tried imho. Sometimes simpler is easier though.

Windows Development MVP here.

I feel like I must shoulder some of the responsibility here for not being louder about these issues. But must say, I'm disappointed to hear you're "surprised" about the UX issues. I've been telling your folks the UX is dreadful (e.g. as far back as pre-launch) and kept hearing back "we know, we're fixing it". I'll start formalizing the feedback and push it through the pipes, stay tuned. I'm also local (Bellevue), would love to come in and try to pipeline our relatively simple oss .net/wpf/uwp app. I suspect it'll be an eye opener for the both of us.

Some examples:

* You can't build a pipeline with a git repo. that contains submodules

* Found it impossible to edit the PATH for some custom tooling

* The New Pipeline experience just doesn't make a lot of sense, new users clicking around will eventually end up at the wrong Docs.

Here's my experience with Microsoft.

I believe their feedback mechanisms are placebo, pacifiers for loud people. Microsoft pesters you to give feedback and works hard to ignore them. And then goes ahead with what they initially planned. At the end, they'll announce - "We had delivered what you wanted. Based on feedback..."

And then you start wondering if you're in the minority and other people are simply crazy until you see multiple complaints about the same things.

Their music app, Windows phone, Windows 10, Skype and Edge are where I experienced this. In all cases, user experience kept getting worse with each "IMPROVEMENT."

The only time Microsoft listens is when people ignore official feedback channels and there are massive outcries on many fronts. Even then, Microsoft often retreats only temporary.

I've never seen an answer on the weird social.microsoft site that had anything resembling a remotely usable technical response.

Also the responses you get on Azure support tickets is 10 worse than the ones from Amazon support.

Yes amazon support is pretty amazing. They’ve walked me through fixing a VPC I screwed up. No complaints here.

These large cut and paste answers with completely irrelevant instructions and a lot of begging for points. Totally useless and infuriating if you've spent a lot of time writing a good report.

Microsoft could further save money by building an answer bot.

It'll cost less, answer faster and wouldn't be any worse than the current outsourced indian community support staff.

Don't give them ideas...

I assumed that they already had, but that it wasn’t very good

Yes. Incidentally Connect used to be a placebo for us enterprise lot. This has been going on for a long time.

And I get told regularly by people who only use the Microsoft ecosystem that’s it is me who is the problem as well!? These people are just assuming the status quo is being punched in the dick all day.

And even worse when dealing with PlayReady. In the time i have been working with it, they have been switching between at least 4-5 different "Connect"-like portals for feedback and support. Each time dumping all the content from the previous that was no longer accessible.

I'm not in Redmond, but do send an email. I'm my HN name @microsoft.com. We should be able to sort out some of these issues for you. I'd like to hear more about the docs problem. But:

You should be able to recurse submodules in a pipeline. In the visual designer, select "Checkout Submodules" in the pipeline's get sources step. If you use YAML, set "submodules: true" or "submodules: recursive" in the checkout keyword.

You should also be able to specify environment variables (including PATH) with the env keyword. But do reach out and we can get to the bottom of this.

Hey, just one bit of feedback, I tried getting my github linked project to build late last year, but always failed to checkout sub modules.

The build log showed permission issues, and I did try workarounds suggested like changing the sub module URL and adding ssh keys as part of the checkout step, but in the end, my 8 hours allocated to “get vsts building” ended with this problem and I have not continued. The performance of builds was also poor.

It’s unfortunately burnt me enough that I’m just making an ec2 instance with teamcity.

It is likely that this is a variant of the issue with secure files. Secure files require a song and dance around changing the default branch or committing directly to a default branch to trigger a build after setup of the secure file. Coming from CI systems like Travis and Circle, this is an extremely confusing approach that I hope they consider changing.

Yes, we rolled out some improvements to how we authorize secure files (and other resources such as variable groups, agent pools, service connections) that you use in a YAML file. First, we introduced a toggle on each of these resources to mark them as "authorized in all pipelines". For new resources that you create, this will be on by default. Second, when you queue a build in the new YAML editor under Pipelines hub, then you get the option to correct the resource authorization issue. More work needs to be done here so that we can give you the right experience while maintaining the security of protected resources, and all of that is planned over the next month or two. Thanks.

That sounds like a great improvement to the process!

I'm sorry that the product wasn't easier to use, and that it wasted so much of your time. For anyone else running into this, we do have docs up about how to access authenticated submodules: https://docs.microsoft.com/en-us/azure/devops/pipelines/repo... (that shows how to do it against another Azure Repos instance, but a variation will work for any Git provider with a PAT-style bearer token)

To your path issue... most of my own pipelines do an `npm run STEP` and I use JS scripts mostly... this covers me in window, linux and macos for the most part with Node. ymmv here though. That's just my $.02 and it's been relatively easier doing it that way.

I kind of wish $bash (git for window's bash version) was available as a shell option as that might make things easier as well.

Another issue is searching is painful... "TFS" vs "VSTS" vs "Azure DevOps" and half the time, the results are dated.

>though we're working on this now

beetlejuice,beetlejuice,BEETLEJUICE!

Why can I only choose github as my repository when creating a new devops project? It used to be possible to use vsts (git)

Sorry, I don't understand what you're seeing here - you're creating a new project in Azure DevOps? When you do this, you get the choice between Git (hosted in Azure Repos) or centralized version control (TFVC hosted in Azure Repos). We don't have an option to do anything with GitHub here.

So I think that I'm not understanding what you're creating, or where. Can you drop me an email with a screenshot? It's my HN username @microsoft.com. Thanks!

I will clarify. if you go to portal.azure.com --> Create a resource --> DevOps project --> Bring your own code --> In code repository you can choose between "GitHub" or "External Git". What I expected was that I could point it at VisualStudio.com - unless vsts has become the external git? ps: happy azure devops / azure user here ;)

Aha, through the Azure Portal. I'm not sure what that works that way. I'll look into it, thanks for the explanation.

> We've been investing heavily in our user experience and our CI/CD experience, so I'm sad to see that we've disappointed here.

Maybe you should start investing smartly.

I was pretty excited to see steadily improving GitHub integration. Unlike the others here, DevOps has been my most positive experience compared to a few others like Travis and AppVeyor.

However, this confidence was seriously undermined when I noticed recently that my GitHub pull request validation build stopped working when I closed the DevOps tab.

Yes, seriously. I have to keep the DevOps tab open for my builds to actually trigger. My mind boggles at how such a design could even be possible.

I'm an engineer on Azure DevOps. GitHub PR build triggers are based on notifications our service gets from GitHub (purely service to service), and is unrelated to any state of the UI. Can you reach out to me on email so we can investigate? Use my HK name at microsoft.com.

There's a lot of comments about our UI and I thought I'd post a quick followup here instead of replying to each one individually. We are definitely reading these comments. UI itself is polarizing, and changing a UI moreso, but there's a lot of critical feedback here that we're going to investigate, try to understand and improve.

Really appreciate you engaging here - in addition I think there would be a lot of value in taking a closer look at the feedback that you're already getting on user voice, especially for issues that you think you've closed.

Great example here: https://visualstudio.uservoice.com/forums/330519-team-servic...

It looks like you did something that users didn't actually ask for in 2017, closed this, and then have been ignoring 1.5 years worth of comments being told what users actually want. From our end it's simple - the embedded burndown chart on the scrum board should be able to show story points, not just hours (like with any other scrum board tool). I shudder to think how many engineering hours went into solving the wrong thing here...

The old UX was not perfect, but I find that it worked better in so many ways. New UX does not seem finished, just pushed out in beta state. Like O365. Sad.

One thing I can certainly agree with is the "no horizontal scrolling" beef.

I have a 4K laptop, evan on high res, the lack of horizontal scrolling in Azure portal and other Azure tools is a dailly irritant.

Stop trying to badly emulate Android phone look & feel ... for desktop usage !!!

We are professionals, we can handle scrollbars without screwing up.

I've migrated my Boost builds from Travis and Appveyor to your CI offering recently. It wasn't super smooth, but I got it to work quite fast and I like it a lot.

So sure, I'm missing caching that I've had on Travis. But honestly, it builds reliably fast anyway. On Travis, I always needed to wait a LONG time to get macOS agents, on Appveyor, we were limited to 2 parallel builds. Here, everything starts all at once, completes under 10 minutes (there's quite a lot to build and test).

If I could, I would love to have an sccache ( https://github.com/mozilla/sccache ) compatible distributed free caching solution for my C++ projects. That'd be a killer feature!

You can add as many additional parallel builds as you want to AppVeyor for $50/m each - is Azure CI pricing comparable to AppVeyor with only two concurrent builds?

Last time I checked, the free offering was 2 CI builds in parallel on AppVeyor, which is just too low and slow for feedback on my project.

Also, considering I make exactly $0 per month from it, it's either be patient or use another service that had more capacity and faster builders.

I believe that Travis-CI had a higher limit for parallel jobs, but your builds would most often end up queued because of a low capacity for a certain type of agents. Also, something quite noticeable was their lower limits on build time, which is why for some builds a cache was necessary. It got a bit hairy when you needed a full successful build to properly populate the cache but they would never finish in time. And of course, the biggest drawback from Travis-CI back then was the lack of Windows support.

As for Azure CI, I believe there's something like a 10 parallel builds limit. And I've got exactly 10 configurations!

Often when I click Releases it is completly blank. All our releases are missing. I have to refresh 2-3 times before they show up.

The UI has so far changed five times in the past three months. It is insane. The current iteration is at least fine. Before it was a big blob of round circle that showed that the steps that succeeded succeeded. And then a big text saying it failed. It was worthless.

I’m considering migrating an appveyor setup to devops. The main driver is to get a pipeline setup with more parallelism and a more complex graph with resumable steps without fiddling with environment parameters, or really jumping around in a clunky ui maintaining a hundred “projects”

Haven’t looked to deeply what devops has to offer yet, but things I’d be looking for before migrating.

- I want to be able to develop, run and debug most, or all, of the steps in the graph on my development machine without roundtripping through long chains of triggers and pushing to code repos and pull requests

- I want to run as much as possible in parallel, preferably agnostic of wether it’s on the same machine or several

- I need to build and assemble many components from same repository (monorepo setup)

- only impacted components should build

- Sub-components may depend on different build tools (service fabric services with both webpack and msbuild based packages)

- Some steps may depend on configurations and secrets with a different lifecycle than the code branch being built

I imagine something clever could be built on docker to support both running things locally as well as customize build nodes.

I don’t care to much for scripting in yaml. In fact I spent the day converting some yaml (and cake) scripting to Invoke-Build. Invoke-Build is quite nice actually, the only thing I miss is the option to orchestrate tasks in parallel onto several build nodes. That plus something like ncrunch build/test engine for csharp projects

(really the entire vs/msbuild things needs to be replace with something truly distributed, incremental and repeatable)

I can recommend https://concourse-ci.org/ for pipelines that are difficult to reason about or orchestrate in, e.g. GitLab and other CI tools, due to external actors or lifecycles beyond a single codebase/repo.

Builds are 100% stateless within the CI cluster and rely on tracking/publishing just about anything you can imagine or plug as a resource; out-of-the-box inclusions being docker images (registry), S3 (e.g. minio, AWS) and git.

It's a bit strange to reason about in the beginning but I have yet to try anything that comes remotely close to Concourse when it comes to orchestrating across multiple sources of input/output and non-linear pipelines.

Awesome, looks almost exactly to what I imagined building.

Thanks!

I forgot to mention the semver resource which is a bit weird to use for a single repo pipeline but comes in handy for out-of-band jobs like running an external automation test suite asynchronously.

Makes me really appreciate my setup of:

- GitHub for source control

- AppVeyor for CI

- BlazeMeter for Load Testing

- Google Cloud for Cloud Hosting

- Slack for Team Chat

- MyGet for private NuGet feeds

- Trello for task management

- ELK cloud for log management

Yes it is many different services, but as a small team I like that we can pick the best tool for the job instead of trying to find a hammer for all nails which doesn't exist.

It also makes us less worried about changes, because we are not invested in any vendor so much that if they do something which makes our lives too difficult that we couldn't easily move on elsewhere.

Also it forces us to design our software in such a way that we don't lock ourselves into a specific vendor. For example we would have all our builds scripted instead of configured via some GUIs with the configuration stored on some vendor's cloud. The build script runs from anywhere. We can run our builds on Azure DevOps, TravisCI, CircleCi or anywhere. We just prefer to use AppVeyor because it's the smoothest CI server IMHO, but the transition elsewhere is just a matter of invoking a single build script.

I genuinely also prefer to work with a product which is doing one thing really good instead of having one product which does everything mediocre. From an operations POV it is just as simple as having everything in one place. We have one browser tab open for each service and it makes us actually a lot more productive. I am always only 1-2 clicks max away from what I need.

Teams/Companies who choose an all-in-one solution (e.g. everything in Azure + Azure DevOps) don't care about the productivity of their teams IMHO. They care more about the one person who has to do the accounting for their subscriptions or something like that :/

Sadly MyGet is no longer free :(.

I was just building a presentation on this topic and the power of an open approach.

While I certainly have had a few annoying issues myself, full credit to the team behind DevOps (nee VSTS) for doing an amazing job over the last >12 months to overhaul and improve the product. I have been impressed at the pace things are improving.

I have not had the same issues with build machines disappearing into the ether (we use Windows hosts however) but definitely agree that the (lack of) caching is maddening. Our apps are not huge but the NuGet & npm restore times now account for 25-30% of our build time.

I exchanged an email with the Product Manager of some Azure DevOps component that comprises the build pipelines. They’re well aware of the need for caching. Hopefully it comes soon!

Unfortunately, they've been aware of this problem for years - I'm an Azure DevOps fan, but I really just don't understand why they didn't fix this an age ago :/

Thanks for the kind words; I agree that caching is a problem. We're working on it now.

Any idea on when it will land?

I'm on a team at building a C++ component and just installing boost can take over 20 minutes for a build.

Yeah, this has been my experience.

I haven't had a build agent disappear, but lack of build caching is very frustrating. The hosted build agents are slow to begin with, let alone needing 5 minutes for every build to restore packages.

AFAIK there's no way to "only run changed tests" too.

EDIT: Forgot to add that, despite the above, I'm pretty happy with it. The whole VSTS experience (Wiki/code/PRs/etc) is pretty great, it's nice to have everything in a single place.

> AFAIK there's no way to "only run changed tests" too.

https://docs.microsoft.com/en-us/azure/devops/pipelines/test...

That looks great - must have overlooked that last time I checked.

We use paid Azure DevOps for Source Control (although we're watching GitHub closely).

We tried to also use Azure Boards (Work Items, Boards, Backlogs, etc). Ouch. It is a complete UI mess of disjointed ideas. Instead of implementing one thing well, they implemented two dozen things terribly.

We actually migrated from Azure Boards to Office 365's Planner (KanBan) just because the UI wasn't so horrible. It is simple and gets out of your way. In fact O365's Planner, Teams, and OneNote Online products might be superior to anything Azure DevOps has produced.

The only nice thing I have to say about Azure DevOps Boards is that it integrates into Visual Studio, but frankly even that is barely passable. I'm predicting they scrap DevOps completely and just use GitHub as a clean slate.

I don't even use VS if I can avoid it... web dev is so much smoother in VS Code, and the laginess of VS is unbearable to me. I do have a couple .Net Core projects I mostly use in VS Code and one MS SQL DB project that I have to use VS for.

May want to checkout GitLab as an option as well.

Interesting post. I feel for him using this tool. Had my own experience with team foundation server on premise. Although a different tool, many of the problems seem quite similar.

A little rant:

To the cloud infrastructure of MS: I don't know if anyone uses it for anything serious and doesn't complain about severe performance issues not only related to builds, but everything. From administrative sites to hosted environments. I read a lot of excuses mainly saying I should adjust my expectations regarding speed for cloud solutions... Problem is that there are other providers that don't have these issues.

Seriously, you need only to use office 365 if you want to get an impression of bad performance. Because everything is slow.

This is from a location in Europe, but I really doubt there are significantly more performant locations.

This is true of all of the services I've tried across Azure, AWS and Google cloud.

Microsoft are always last place in terms of performance and reliability.

For Kubernetes I actually keep track of exactly how bad they are with an automated testing tool.

Results of the last test are here: https://kubedex.com/is-azure-kubernetes-aks-any-less-terribl...

I've complained about UX before in the context of the Azure portal being, in my opinion, terrible. But that's subjective and I respect that others may like it, although not sure how they do.

This is very interesting data, thank you for that. Maybe MS has improved a bit although I haven't felt any improvements yet. But the changes might not be rolled out completely. Performance wise it can only get better in my opinion.

You should do an updated test comparing AWS as well. I feel like there should be more continuous benchmarking efforts of cloud vendors. This is the last one I remember in this genre (https://www.azurefromthetrenches.com/azure-functions-signifi...). I wonder if someone could make a business doing that.

Also, I've found in terms of UX GCP > Azure > AWS personally.

I felt the same way when I worked on a large Azure project years ago. It seemed that nobody uses it for anything serious, reliability was awful (their API would return 500s in the middle of the day, no notice of downtime, VMs would lose network connectivity, disks would take 30+ minutes to attach, and much more).

The client I was working with is very big and notable, so they had lots of help from Microsoft, and even so, there were so many problems with the service. I'd hate to see how smaller customers are treated.

I remember high CPU alarms waking me up, just to see that Azure OMS Monitoring agents were consuming %99 CPU on every node of our client's database cluster. I'm not sure why that happened, yet the immediate reaction was to uninstall it and never use it again.

OP's story about build agents turning unresponsive makes me think of this. MS is the best cloud vendor to deal as a consulting partner, yet the overall suffering caused by Azure almost makes me feel like it is not worth it.

Look, developer tooling is always going to be hugely dividing. Some will love it, some will hate it, some will hate it because it's not like the Thing They Loved At Their Previous Job.

Now, can we talk about the real WTF, which is the name. "Azure Devops", really? Two extremely potent keywords each on their own? You didn't like people being able to google answers to their questions about your product?

TFS, VSTS, "Azure DevOps" and often the results are outdated... "ADO" won't work either. They probably should have stuck with VSTS and just kept the abbreviation or made it "Azure VSTS" where vsts would at least continue to be searchable.

Another nicety would be to pay StackExchange to add an Azure DevOps subsite, assuming there isn't already one.

How do all of you people get on this broken stuff on Azure with GCP and AWS (I think) being so good? Is it like free credits?

I can only assume windows people are used to pain

pretty much it, also azure is slightly better at pricing in asia

This mimics the mac vs windows argument I had with my brother-in-law where it boiled down to the fact that you could run windows on something cheaper than a macbook. Even though in order to remap caps lock to something reasonable you have to edit the registry. In 2019. The best answer I think involves windows server 2003 resource kit and its UI, which is so old it could get a driver's license.

You save money but what kind of life is that?

It's not even about the price. I have no problem tweaking my Linux desktop to use an international Dvorak layout (useful for those European languages with ä's and é's) and switching the caps and ctrl keys. Say what you want about Gnome 3, but this works just fine.

On Windows.. I get a US only bare bone Dvorak layout and few ways to remap the caps/ctrl keys. Messing with the registry is at best annoying on my gaming (only) PC but impossible on a corporate locked down desktop. Then I'm lucky if the corporate package repo comes with AutoHotkey that can remap keys in a very hacky way.

A lot of Windows 98 still shines through in the basic usability suite even on the newest versions of Windows. Even the on-screen keyboard on a (Android) phone is more fun than the HID mess in Windows.

> useful for those European languages with ä's and é's

As I'm not typing them frequently I'm happy with a Compose key binded to Scroll Lock.

I would love to find an .xinitrc in some dark corner of windows 10 where there are a bunch of options commented out and you have to pick the right ones or else your monitor explodes.

This would be an upgrade as at least I would kinda trust that a file I found on my own system from the vendor can probably be trusted. I trust the options would be at least mostly self documenting (more self documenting than a random list of hex digits). And, if I'm going to use software written in the 90s, I'm going to go with good old vim.

I love vim, but that .xinitrc stuff is really out of date by now. Gnome 3 now uses Wayland fairly well and the setup is done with the GUI app Tweaks and the system settings.

https://www.autohotkey.com/

I'm only one person, and my team is only one team, but we have been completely satisfied with Azure and DevOps so far. But then again, we are a small team and tend towards many smaller projects rather than fewer large projects, so some of the issues mentioned here w/ caching and missing stats, for instance, may just not apply for us. We don't really need to see stats of how many pull requests each dev has made when there's only 4 of us doing backend work.

Additionally, we are a non-profit, and MS works with us in a moderately generous way with pricing, whereas AWS wouldn't.

I can only second that. Team of 10 with about 30 projects and Azure DevOps was the best that could happen to us. The only problem I saw in here is that the build ui does not automatically update sometimes. Another thing that annoyes me is that the tasks and stories do not update in realtime when you work on them from multiple machines.

All of them have problems. The other 2 have nothing close to a devops product suite like this at all.

I can’t even fathom it myself. I begrudgingly spent a month on azure as I worked for an MSFT partner. This was to establish if it was the way to go. It wasn’t. It was a friction filled mess.

AWS slowly crept in because there was no entry barrier, no magic Harry Potter spells to get anything working and it actually worked and stayed working.

Now I went back to azure to see what has changed Q3 last year and it was the same “eating sawdust” experience. Literally everything is clunky.

When it comes to these providers, time is money, possibly more so than worrying about the instances. Azure costs a lot more in human attention.

AWS is not perfect though. CloudFormation quite frankly sucks balls but there’s enough interest and so little friction that there are lots of third party things that work well with it.

Azure, not so much. It’s a leper colony.

1. I don't use Google products if there is any other choice.

2. I think Azure is better overall than AWS. It's way easier to grasp the different features, while I haven't used Azure Devops, I've had little issues with the rest.

What is your scale and how do you implement automation ?

I don’t see the relation between Azure DevOps (an issue management and build system competing with Jira, Gitlab and others) and GCP/AWS?

Even if I run on some other cloud than Azure, I still need the software. I could even run Azure DevOps on Amazon’s cloud I guess (If I choose the self hosted option) but the UX complaints would apply there as well...

My current job has various .NET/Windows-based software products that can run as PaaS projects in Azure - from what I've seen it wouldn't be practical to run them on AWS/GCP without managing VMs.

Tangential: Reading this _really_ makes me appreciate the extremely stable system my team has build internally with GitLab and TeamCity. Azure DevOps sounds extremely frustrating (besides the point that I cannot stand the name they gave this product, and GitLab made that same mistake with the Auto DevOps feature name).

Thanks for using GitLab!

We're open to a better name for Auto DevOps https://docs.gitlab.com/ee/topics/autodevops/ but we haven't been able to find one. Suggestions are very welcome.

BTW We're planning to add per per minute for the CI runners and add shared MacOS and Windows runners so people that don't want to manage their own no longer need to.

That's awesome. We have so much customization around TeamCity that moving away from it doesn't make sense, but we're starting to look at the CI runners for other things, along with the k8s integration, all great stuff in GitLab. Yeah, the DevOps name though, it's unfortunate. DevOps is a culture, not a thing, product, tool, or process. Seems more like calling it Auto CI/CD is more appropriate, maybe? Anyways, thanks for the response!

Thanks for the Auto CI/CD suggestion. I struggle with it since it doesn't cover things like license management and monitoring.

I agree DevOps is a culture foremost. But at the same time it is starting to turn into a product category (DevOps tools).

Hope you are able to adopt GitLab with ease.

I think title should be updated to "Microsoft's Azure DevOps CI: An Unsatisfying Adventure", as all the complaints here are about builds but this service supports lots of other features.

My experience with using DevOps nee VSTS was mostly satisfying. For various reasons unrelated to the post, my team ended up installing the build agent on our own bare metal hardware, and other than needing to restart the agent, I didn't have any mission critical issues. Totally agree with all the UI complaints.

I use Azure DevOps everyday and like it quite a bit. It'll keep improving but I've had no complaints. Microsoft is in a very enviable position, not just having one of the best IDEs but having control over potentially an entire CI/CD pipeline from Github to Azure, while having one of the major programming platforms in .Net to enable best-in-class integration. No one else can really say they have all of that, not Atlassian, Oracle or Amazon.

The CI component, Azure Pipelines, is the only hosted CI tool that supports all major platforms, at a minimum Windows/macOS/Linux.

The value of essentially fully automating out the role of DevOps engineers is a long time coming, and I used to do that sort of work but saw the writing on the wall with things like Azure DevOps. I think a lot of existing DevOps roles will transition over time to 'Cloud Engineer', specializing in setting up and maintaining hosted CI/CD platforms. There will also be a lot less of them as a result. It's becoming a commodity, MS is just one player there. I love what they're doing and eagerly embrace it, Microsoft has been knocking it out of the park. It works well for us.

We moved to Azure DevOps for Windows testing for multiple large projects (Next.js, Now Desktop, Hyper) and it's been an absolute pleasure. Much better than our previous Windows CI solutions.

We are using the on-premise solution with our own build agents to build iOS. Looking to move actually into Azure Devops to reduce the effort of maintenance on our own build agents. I've been using AppCenter for hobbyist projects and I feel its been a good replacement for BuddyBuild. Seems to be pretty reliable additionally we've thrown a lot of builds at Azure Devops to validate how well those hosted-agents can handle a large project which also worked out well. So all in all, we are excited about the hosted-Mac agents as well as the flexibility to BYO-agent (helps in the transition).

I have seen a lot of these same problems noted in this link as well, just didn't see to prevent from getting our work done as much. So YMMV depending on your product/devops use case.

Personally have been using their APIs with some custom CLI scripts to get around some of these issues.

There were some enormous UX issues with the old look & feel of TFS/VSTS as well. Navigation was nearly impossible to understand, for example.

While I appreciate the work they put into DevOps, it still feels like the same person that designed the Azure Portal UI has had a say in the UI redesign of TFS into DevOps and that must either mean that a) Microsoft never realized what a horrible UX the Azure portal is, or b) they do realize but they are too invested in this new "design language" to do anything, so instead DevOps adopted the same style as Azure.

I'm in the process of migrating TO DevOps (on prem) so obviously I have a keen interest in the direction its taking. Build pipelines being declarative, navigation being clean up etc. are fantastic improvements. But it still feels clunky to use, especially around the backlogs/boards.

VM Agent based builds are an obsolete model. A docker/container-based pipeline is the current state of the art and allows for very fast builds with caching built in and isolated stages with narrow responsibilities.

There is an official Linux image for a VSTS Build Agent container - but, frustratingly, not for a Windows one! People have been begging for such an image for a long time now, but Microsoft do nothing... I just don't understand why they treat Windows as a 2nd class citizen!

We offer container jobs [1] and recently added the ability to use service containers [2] as well. Disclosure: I'm a PM for Azure Pipelines.

[1] https://docs.microsoft.com/azure/devops/pipelines/process/co...

[2] https://docs.microsoft.com/azure/devops/pipelines/process/se...

Funny enough I had not had the time to write about this yet. This resolves one of the other issues on my list. Thank you.

> For a few hours, builds ran smoothly. As expected, eventually the agent died during a build. Builds sat queued, my frustration grew, and AWS notified my instance needed a doctor. After an hour I gave up hope for the unresponsive server and commanded it to reboot.

Sounds a lot like some process on the build server is leaking memory. We have a couple of older services that have this problem, and if we don't catch it early enough the server will become unresponsive and unable to even handle ssh connections.

As a maintainer on an opensource project thats recently switched out to azure pipelines, I'm actually overall happy with azure pipelines, its considerably faster (hosted agent) for us in matrix builds then travis both on latency to start a build and actual compute power, and cheaper then self hosted drone. Hosted builds have been reliable for us, we notice a hiccup less than once a month on a single build (avg 20+ builds a day), minus a service incident yesterday.

That said I agree with a lot of the criticisms noted here (no retrigger on web hooks to update status, complicated web ui, log ux, lack build cache). It is possible to bring your own cache implementation, I just stuffed/restored things into object storage, also filed an issue for build cache https://github.com/Microsoft/azure-pipelines-tasks/issues/91... noting several other closed issues for the same.

It does feel like the platform is moving forward. All that said there's a lot of complexity and feature set in azure DevOps and pipelines that I haven't explored and try to ignore in the ui. At the moment its the only cloud provider build solution that actually even attempts to work well for opensource projects afaics.

Thanks @kapilvt, I'm on the Azure DevOps engineering team. Are there specific improvements you'd like to see in the logs UX?

At my company, we are pretty happy with Boards, Repos and Pipelines.

One thing that needs to be improved so badly is their Test Plans section, not so fan for their way to setup agents for Load Testing.

I've been using this day in and day out for several years now on a ton of projects. We mostly use hosted git, and pipelines. The lack of NPM cache is a giant pain, and that's why we run our own build agents. Other than that, I can't say enough about it. Compared to some of the monsters I've seen built with Jenkins, I love the simplicity of the builds and releases we have created in Azure DevOps.

My mind was blown when we invested a week in setting up DevOps, then realised it has a critical flaw in that the automated build agent will just go into hibernation if there was no build activity for a few hours. So then you have to write a script that invokes a manual build via the API just to keep DevOps "alive". It's like they forgot to properly setup the IIS Worker Pool timeout values or something.

@nbevans could you provide a few more details here on the agent and where it is hosted? The agent itself does not rely on the IIS worker pool and we are not aware of any bugs in this area. If we have some we would love to get them addressed.

@nbevans I'm an engineer on Azure DevOps. This is the first we've heard of this issue, can you provide more details? What error do you see? By hibernation do you mean Windows hibernation? Feel free to reach out to me at my HN name at microsoft.com.

I don't know... having been on openshift, Jenkins and bitbucket for the last year I'd give a large sum of money to go back to azure devops.

Came here to say that I've been impressed with Azure DevOps so far. We don't have the same requirements as the Author. For example, we don't deploy artifacts to a dependency management system and we don't build on Linux.

I have small complaints here and there but I couldn't find another build service that has Windows and OS X agents for anywhere near what Microsoft charges.

We are actually pretty satisfied and everything's been running smoothly!

We migrated from TeamCity and OctopusDeploy a couple of months ago and we are already regretting it. The reliability of this system is just horrendous. The documentation is pretty much outdated, and quite a few plugins do not work. I have to write a bunch of powershell script to get it to work the way I want. And today, all hell broke loose.

Since this morning, we are stuck with an error message "TF400893: Unable to contact the server. This is most likely caused by a network error. Please check your connection and try again.". This has cause a couple of release pipelines to be get stuck, with cancel option not working either. The paid "parallel jobs" are being consume in this stuck pipeline and no other project from my organisation can be built or deployed because of it.

Pretty much regretting having spent so much time configuring the pipelines. We might switch it doesn't get resolved soon or if it happens even one more time.

We use Azure DevOps for a fairly large project and so far its working good (i.e., not great, but better than okay-ish). The point with the lost communication is something we also experienced more often than I would wish for, however, on a standard day I do not see the issue appearing at all.

Yes, the mobile page could be optimized (but this is not a deal breaker) and yes some pages do not subscribe for notifications, but usually all these things are not in the hot path.

TL;DR: Lots of improvements in the recent year and many things to come - it may not be perfect but it certainly is usable and provides value to our team.

The majority of the product indeed does work pretty well. We make heavy use of their Boards and Test Plan features and outside of specific but minor UI/UX complaints, it is a competent tool.

I'm genuinely curious: what exactly is so hard about installing and system administering one's own servers inhouse that so many people would put up with these complications?

For us its: availability outside of VPN and avoiding the run around of making a public site with security, integration with active directory, has repos + a wiki + boards all in one place. No configuring 3 different OSS products to integrate with AD.

We use the other stuff more than the CI, but will probably begin using it for some more smaller projects soon. It'll be in the same place which is great and no need to tinker with servers which is awesome.

If you have problem with configuring service users in AD, I guess you have problems with security or group policies on a regular basis.

Or AD isn't our responsibility and we can just use the tool (Azure devops) that is already done before we even launch it and provides all the features we need?

I guess you don't have more important things to be doing than screwing with ancillary tools.

There isn't any true difficulty; people are running multiple sites that could all fit into 1 dedicated 64gb RAM server from Hetzner on a different infrastructure that allows them to develop their resume...

Are you saying that it's simply a case of resume inflation at employer's expense?

Sometimes yes. It's not just DevOps but CVops :-)

If I catch anyone at my company doing CVops, there will be hell to pay. Not at the company's expense and certainly not on my watch!

I'm using Azure DevOps (Previously VSTS) more than two years, and meanwhile I create couple of Pipeline to perform CI/CD.

> .NET MVC Web Application Build and Deploy Using FTP > .NET MVC Web Application Deploy in Cloud (Both Azure and AWS) > .NET MVC Web Application Build on Linux Visual Studio Agent Create Docker Image and Published

I didn't face any errors from devops, my experience overall good with DevOps. but i agree that it's not user friendly in term of UI and other workflow.

What I never really understood is why does Microsoft Test Manager only come with Visual Studio Enterprise? Do only enterprises test their software?

> Do only enterprises test their software?

Microsoft also has operating expenses. They need to bring home some income to keep the lights on.

Isn't that what Azure is for? Use the software as a loss leader to get them using your services, which is where you make the real ongoing money.

It just strikes me as an odd thing to put on a luxury tier like that. Reiterates that quality and testing is often an afterthought rather than something everyone should include. If MS wanted the revenue they would put something the most senior higher paid developers wanted rather than the testing team.

You have plenty of alternatives...

True, but I thought the goal of Azure DevOps was to be a complete package solution. And so while all the pipelines and code and work items are stored there, it would be a pain to keep test cases in something else. And why no integration with Excel?

(Azure DevOps PM here) Test Manager comes with VS Enterprise, VS Test Professional, MSDN Subscriptions and is also available for direct purchase through https://marketplace.visualstudio.com/items?itemName=ms.vss-t.... You are welcome to take your pick from what suits your needs.

As a subscription, it's more per month than Visual Studio Professional. As part of VS Test Professional it's almost twice as much for the first year. And you can't really do any test plans without it so its a huge part of DevOps that is just unused for us, unfortunately.

I don't have much complains about DevOps UX, the only problems that sometimes impact me are the caching/random broken builds.

But Azure Portal UX is another story, the Blade UI is one of the worst things I've ever used, I hate scrolling horizontally, I can't open things in a new tab, it's really frustrating.

I've yet to look, let alone try, Azure DevOps. There would have to be some compelling reasons to choose that over TeamCity, which I have had nothing but great and positive experiences with.

Any big reasons DevOps is something to consider, or is it mostly ideal for new projects?

Their build pipeline has been down all morning as well, for both hosted and self-hosted. Just sits pending. No deploys this morning, woot!

wow that sounds pretty seriously broken. Has anybody else here actually used it & can comment?

I've been using Azure DevOps for around 2 years, and TFS for several. Most of what this article says, I simply don't see in (almost) every day usage.

The single issue I agree with is the lack of caching - installing NPM packages can take up to 15 minutes in the worst case. MS have known about this for years, so I honestly don't understand why they haven't done something about it.

I also think the hosted build machines are underpowered, despite their claims of Xeon CPUs - build on even an Azure B1ms VM (a low-powered, burstable SKU) are up to 3 times faster. In fact this is how we solve the NPM caching and performance issues - by running the VSTS Build Agent (it's OSS!) on relatively cheap Azure VMs.

Aside from these niggles though, I actually think the Azure DevOps product is fantastic.

I'm interested to hear you have not had the same CI issues I have. One theory I have with no substantiating evidence is that Linux agents are least commonly used. The Linux agents on hosted builds run in a hypervisor but I've seemingly had the same issues running self hosted agents (as I mentioned AWS).

I can't find an actual issue and again as I mentioned in the post, replacing my AWS instance with a new AWS instance also eventually hit the same issue. Maybe it's some perfect storm of problems but that is just more speculation.

Regarding caching, with NPM you actually have a couple good options (best together):

1.) Make sure you are using `npm ci` instead of `npm install`. The `ci` install command is often much faster and skips a lot of stuff that `install` does. It's also better at sticking purely to your package-lock.json versions. I suggested on UserVoice at one point that Azure Pipelines should switch to `npm ci` as the npm task default. `npm ci` is a relatively recent npm command and I don't everyone has caught on to it yet.

2.) You can use Azure Artifacts for NPM packages. It acts as an alternate NPM feed and caches packages to your Azure DevOps account. Artifacts only supports NPM and NuGet, so it isn't a general caching solution, but in this specific case it helps. I've noticed the builds I have using Artifacts are a bit faster than going straight to NPM. (I started using Artifacts for private NPM packages, so I'm not using it in every build yet.) The current biggest caveat to using Artifacts as your main NPM feed for a project is that it doesn't support NPM audits currently. (I also made sure to report that on UserVoice.)

Why on earth would you use a different deployment stack in CI than you would in production? This seems to miss (at least) half the point of integration testing.

Making this decision is bad enough, but making it because you've chosen a bad CI/CD stack is even worse. Use the tools that work, instead of breaking things to match.

I'm not sure to what you are referring.

Are you talking about Azure Artifacts? Like I said, I'm using Azure Artifacts as a Private NPM repository for proprietary packages. It also providing caching benefits was a pleasant surprise.

For my projects using Artifacts as their NPM repository, they use an .npmrc at the repository top level to only use that feed (and its pass through capability to the public feed), so for those projects it is the same stack from dev through to production. As I said, the one caveat is that if I need to `npm audit` (which is generally a good idea) I currently have to redirect to the main public NPM feed. Other than that it is a solid replacement NPM feed.

I find it weird that in order to fix a problem Microsoft has created (slow builds due to lack of caching) is to give Microsoft more money. Artifacts isn't free.

Sorry, it's easy to forget what the commodity prices of some of the Azure DevOps tools are given that various combinations of MSDN licenses and Office 365 plans cover them. It's great that you can pick and choose now, though.

Like I said, caching is only a useful side effect of Artifacts for npm and NuGet users (that have access to Artifacts). I would believe/hope that Microsoft may still have interest in adding proper caching to Azure Pipelines directly.

I was completely unaware of `npm ci`, so I'll definitely check that out. We'll stick with hosted agents that do caching though, as the hosted build agents are really slow.

I've helped a few clients migrate from whatever ad-hoc process they had to CI/CD with Azure Devops.

Apart from one minor issue with a bug in node.js timing when running the build server on burstable VMs (which is probably more to do with AWS setup than node), I've not had any issues that I can think of.

I don't have the problem with cache though, as I always run my own build servers/agents.

The UI has changed a few times over the past year, mostly around the rebrand from VSTS to Azure DevOps. But once learned, it's pretty straightforward and I'm sure I've barely touched the surface of its capabilities.

I, for one, am very impressed.

EDIT: oh, one improvement I can think of, would be to accept web-hooks rather than polling Bitbucket. But honestly, that's only a delay of a minute or two, so it's not a big deal. You could also easily enough create your own serverless webhook listener that triggers the build, so it's a bit of a non-issue, but I was surprised at the lack.

Thanks for the feedback @davewasthere. I'm an engineer on Azure DevOps. Our bitbucket support is built around web hooks, not polling. If you are seeing delays email me your azure devops org name and we can look into it. My email is my HN name at microsoft.com.

It seems no one wants to learn the lesson Facebook was forced to learn - No wholesale UI changes.

At my company we use a mixture of Azure DevOps, Jenkins, CircleCI, Github, Jira, Artifactory, and a host of AWS and Azure, and my view is that Azure DevOps (formerly VSTS, formerly VSO, formerly TFService, formerly TFS, formerly VSS---i kid i kid) is a semi-strong product that just can't get out of its own way.

The New UI is a clear step backward...and the Github integration leaves a lot to be desired. If you have Github Enterprise...why use Azure Repos...why are MSFT teams moving from Azure Repos to Github...and others moving from Github to Azure Repos. The SCM platform and strategy obviously hasn't settled yet...so Gitlab is just as viable an offering that's growing more compelling by the day.

The Azure DevOps hosted build system hasn't been viable in my opinion since it launched, so on-prem agents are a much better option. Take a look at CloudBees offerings, they set the poll position.

The key thing that continues to be a pain point is the reporting and project organization data structure limitations with Azure DevOps: is it 1 TPC per business unit, 1 TPC for the whole company, 1 team project for the whole company. What happens when you have 2...how do you consolidate...what's the path forward if you have to move things around. The answer for the last 8 years has been silence.

The request for a clean solution has been asked for since 2011. Yes 2011, look at the link below. 8 long years...and basically silence...and "it's a hard problem"...and multiple updates saying "We will provide an update once we start planning for the second half of 20XX."

https://visualstudio.uservoice.com/forums/330519-azure-devop...

Eventually, the promises just ring hollow, the proof is in shipped code, and in the meantime other platforms and competitors stacks keep building out better enterprise stories for migrating from Azure DevOps (here's to look at Github, Gitlab, Atlassian's platforms, Jenkins/CircleCI, TeamCity, Artifactory, Xebia Labs etc).

Honestly, the world is a MUCH more competitive place with lots of best in class point solutions that aren't that hard to stitch together and get better performance/features.

Hope springs eternal...that with the new vertical offerings the DevOps team can deliver rapid value...but 8 years is a long time to only hear crickets.

If you're starting out and you need a one-size fits all solution and you're a SMB company with two doze devs...go hog wild with Azure DevOps...that's not to say it doesn't work for large companies like Shell and Microsoft itself...but talk with teams about the limitations they run into and make your own decision on your DevOps transformation adventure...oh and never create more than 1 team project in Azure Devops...seriously...if you do, game over.

Sounds mostly like networking problems with the Azure Cloud.

I initially thought the same thing but I'm less convinced of that after trying an aws based build agent. I think the agents themselves may have a bug that combined with our pipeline needs, triggers some perfect storm issue. At the same time, we have seen builds fail to start at all and and sometimes fail to even checkout from GitHub.

The silver lining here is that Azure DevOps's code review tool is lightyears ahead of all other repos. I would like to see this support for github repos

Thats not 1080p

I'm not sure what you mean by this but a few other people have asked me about the described screen size problem so I decided to add some clarification.

https://toxicbakery.github.io/vsts-devops/dimensions/

I've had similar issues with Microsofts CI implementation. I'll describe some of the problems I've had here, note that the last time I've used it is about half a year ago, maybe things have changed though I doubt it. This was with VisualStudio TeamServices on an instance hosted by Microsoft itself. I deployed self-hosted build agents as they were orders of magnitude faster.

Probably the biggest problem I've had was that build agents are stateful. The agent checksout a repository when the build starts, and removes the directory when it's done. It doesn't however clean up other changes made on the system, such as cache directories for tooling and other data you might move out of your build directory. I believe that builds must be reproducible, and therefore must start with a clean and consistent environment. You don't want builds to fail down the line because a precious job broke the agents environment, or even worse, having jobs succeed only on the agent because some dependent files are left on the agent. Other CI solutions use Docker containers for this to build in, which is great, as they are immutable (or stateless, rather) by default. This is in my opinion the single worst 'feature', making these CI builds distrustful.

Secondly, automating these agents is hard. If you want to automatically scale these self-hosted build agents horizontally based on demand, good luck! When starting an agent for the first time, you've to go through some cumbersome configuration wizard to set it up. There is no simple static configuration file available to use instead, or at least, I wasn't able to find it at the time. To mitigate the previous problem with stateful agents I created a Docker container having an agent in it, which was in fact quite difficult to properly achieve. I automatically restarted these agent containers multiple times a day in order to clean-up any garbage that was unintentionally left by a job. This is a horrible hack to slightly improve their (in my opinion) broken system, but it had the bonus of making these agents more easily scalable.

Another thing is the build configuration. I prefer having a configuration in a simple YAML file in the same repository, to have it as _close_ to the code as possible. Although Microsoft does provide such functionality it doesn't seem to be common to do it this way, and if I remember correctly it's much more difficult to properly set-up and use than for example, a Travis CI configuration. They do provide a graphical interface to configure these build steps though, which is useful when getting started with the system so you can get a feel of what is possible. It provides some graphical blocks/steps to achieve various thighs, such as doing an FTP upload or running a CLI command, with all kinds of weird properties. Maybe that's because Microsofts things are generally less scriptable. But yet again, I prefer a simple script in an in-repository configuration file, because it feels somewhat constrained and limited.

Sadly, it doesn't stop here:

- sometimes jobs just didn't want to stop, and no, bashing the abort button a million times didn't fix things. In some situations it took an hour before jobs were properly killed.

- the interface (VisualStudo TeamServices) is horrible to navigate, have had to wild-click many times to find what I need. I'm sure this will improve once you use it for a longer period though.

- and there are many other inconsistencies and little problems, such as build logs loading half of the time.

Maybe I'm grumpy. Maybe I'm ranting about nothing. I don't know. Maybe things have changed, or will change, at least I hope it will. I didn't have a great time with their CI system, my experience was horrible as you can probably tell by now. It feels like an unfinished product, that was released to quickly just to satisfy customers, so they didn't have to switch to some other platform not owned by Microsoft for their CI needs. What amazes me is that there are quite a few free alternatives available which are much simpler and more elegant, such as GitLab, Travis and Circle CI. They are easier to navigate, are usually quite quick, don't have the issues I've (and the article have) described. And greatest of all, you can get it up and running in minutes without having to worry about it afterwards.

TL;DR: The author works for a Microsoft competitor and doesn't like MS products.

Strong words. Can you back that up?

I could find no evidence, looks like the guy is an android developer of some sort.

Same conclusion.

You are correct, I'm just an Android dev but the company I work for is a long time user of Microsoft products like Visual Studio and obviously Windows. Over time, if anything, we have become more invested in their products and I don't see that changing in the foreseeable future as they generally work well for our use cases.

You got me, I use Linux :)

This article is nonsense. Azure DevOps is a delight to work with. PR system is probably the best I've worked with so far. Build pipeline can improve but it's a far cry from a "horrifically broken".

IMHO, Microsoft has been getting a lot of things right lately. Keep up the good work.

Horrifically broken was an exaggeration, you are correct. Much of this was written after a stressful day compounded by dealing with our CI failing to do the thing it is supposed to do. I also agree with you that Microsoft has been making great strides in the direction of "getting a lot of things right lately" as you said.

We really hate to hear you are having such a bad experience. The engineering manager has reached out on Twitter and included me. I would love to see what I can do to help

I personally don't really understand this approach. In fact, this approach annoys me.

In this situation I feel like the product has already wasted my time and now people are contacting me to waste more of my time.

The blog is really detailed and includes screenshots. You could fix the obvious issues and then follow up with an update once it's released.

In the opening paragraphs I stated it was my hope that this gives me a gateway to communicate with someone that isn't support and isn't the void that is their robot response issue tracker.

That said, they generously reached out to me and appear to genuinely be concerned by my experience. I plan to keep posting as I run into issues and also as they correct my mistakes and also solve my problems.

Reply on news.ycombinator.com