Hacker News Reader

After using most CI/CD tools I can tell you that for your tech stack there is no reason to use Jenkins. It sucks, typical Java software that looks like shit, has a buggy interface but somehow runs well enough to be "Enterprise ready".

Go with Drone, you will thank me later.

I would implement the CI pipeline first, write tests afterwards and make sure to integrate with Mattermost/Slack, which will make you aware if something is broken.

I am currently on a Gogs, Drone, Mattermost stack and am loving it. Everything was super easy to setup and runs just perfectly fine for months now without me doing anything other than the initial setup. Jenkins was a pain in the ass and it was the best decision for me to get rid of it.

The beauty of Go is also that it isn't using any resources, this is why I run Gogs (runs perfectly fine with 512MB RAM) instead of GitLab (you need 8GB+ RAM).

They go hand in hand. If you write tests, but they're not run after each dev commits changes, you're not testing those changes for regressions. If you have CI without tests, what is the CI going to do? It could still be used for packaging executables/distributions, and possibly orchestrating deployments, but it sounds like that's not your goal.

Save yourself some grief and use a service like CircleCI, it provides isolated (Docker) environments that are spun up per test run. It supports many languages and background services (like databases).

As for writing tests, you would be best to start writing them at a "system boundary". This might take the form of sending HTTP requests to your endpoints and parsing the response.

I agree that they go hand in hand. In practical terms however it can take a long time to write comprehensive test cases for legacy code, even if only writing integration and system tests and ignoring unit tests.

A pragmatic approach would be to write a few tests quickly first. Then implement CI as quickly as possible. After the CI is working end-to-end running the few tests you quickly setup before then you can focus on expanding coverage of testing. Basically, I'd suggest writing only enough tests to test the CI first :-) Then once that's in place you can add tests.

One other thing that is important to get right is reporting (at the end of your CI flow, once each build and test run has completed). This will help you keep on top of regression but most notably demonstrate the real value of what's going on with all the work you're putting in now.

You don't have contact information in your profile but I wanted to add that I've worked in the role of a software engineer focused on automation (CI and testing) at large tech companies and I felt so strongly about how important the reporting aspect of this is that I started a web-based reporting service that is growing in usage. Check my profile for details, I'm the founder, contact me with any questions or feedback.

> before we refactor the code to support unit tests or start integrating with Jenkins (CI), we should start with at least adding some high level tests.

Yes, this is a good idea. before trying to make dramatic changes to internals (refactoring etc) it is a good idea to get some kind of automated tests around the system.

you'll get quite a lot of value from a small number of basic smoke tests / end-to-end integration tests.

> Giving contributors a way to run tests locally against their changes on their dev boxes, with a testing framework, would be almost as good as running the app in production

a nice way i've seen this done is to focus on automating a build script for launching the app and any dependencies it needs, running automated tests against it, then tearing everything down.

figure out how to automate into a script that developers can use to run automated tests

figure out how to automate the environmental set up (e.g. installing package dependencies)

once you know how to do those things reasonably reliably for dev machines, set up a jenkins CI build to do the same thing: check out the code, set up dependencies, then run the automated test script

Since you've got a web app, it may already be difficult to get automated reliable set up and tear down of dependencies under control. E.g. you use a DB - can you automate setting that up and populating it with test data before a test suite runs? Your app automates infrastructure tasks -- how can you test that without actually triggering the changes, etc?

> figure out how to automate into a script that developers can use to run automated tests

That was my thought exactly. A test suite that starts with a setup, runs a few e2e "smoke" tests to make sure basic functionality is there, then a teardown.

Since the real action is handled by Ansible, I think it should be relatively easy to set up a dummy repo that only simulates playbook runs without actually doing anything. That way you can hit "build" or "patch", get a valid response back, output even, but it's not actually going out and hitting anything.

My main thing is, now that we've nearly tripled the number of devs working on it, and it's now being used by another team (was 2 teams using it, now 3), we'd be irresponsible to add any new features or do further development without a way for contributors to, at the very least, type "python runtests.py" or just type "pytest", and see that the site still has basic functionality.

Here's an example: There's a system in there that allows admins to assign roles to users. Some JS was added for a totally unrelated feature, but it ended up intercepting form submits on that page. I didn't find out there was a problem until a couple days later.

Another time, I was giving a demo to 4 different managers, and I went into this one section to do something, got a 500 error. Turns out I had a couple lines at the end of a view function at the wrong indent level.

It's embarrassing. Before trying to set up Jenkins, Docker, whatever else, a few basic high level tests that cover 5-10 areas of the UI would surely keep us from embarrassing ourselves at the very least.

Thanks for your input.

Your situation sounds remarkably like the state of our (also Python) codebase 18 months ago.

Start with CI and a couple of trivial tests that don't require refactoring - a smoke test or an end to end test that verifies your home page returns a 200 status code. Expand from there.

I say this because tests are useless if they don't get run. When I finally got CI set up for our code base, my next three weeks were spent updating abandoned tests that had been written to verify a dependency upgrade and then hadn't been run in months.

The real win, though, was integrating CI with Slack. Before that nobody noticed if a build broke, and I would have to point it out to them. Displaying an alert publicly in the middle of our primary channel of communication made it something that couldn't be ignored.

Once you have CI running, developers will start to feel the pleasure of expanding test coverage and using tests instead of clicks to verify their work... and the pain of breaking the test suite.

Before we set up CI, every member of our dev team was somewhere between unenthusiastic and mildly resentful of requests for test coverage. That attitude has changed - most of our engineers now take pride in writing tests that pass, and one formerly resentful developer voluntarily chose to target 100% coverage (something we don't yet require) for the most recent feature he built.

Also, I want to echo another commmenter's suggestion about using a hosted CI service - assuming you're a small team. It'll get you up and running faster and require less maintenance overhead. Since we don't use Docker and have a fairly complex stack that won't run on typical services like CircleCI, we use a service that handles the queue, displays pass/fail, supports notifications, etc. but lets us run the test suite itself on our own boxes.

Ask HN: Legacy code – tests first or setup CI?