dosisod

Creator of Cicada here. Thank you for the feedback! I've mentioned this in a few threads already, but the reason for making a new DSL for writing the workflows is that YAML makes it hard/cumbersome to express more complex workflows. Using a programming language though gives you more control over how your workflows execute. While there are already plenty of tools that use existing programming languages (ie, Python/TypeScript) to configure workflows, having a custom DSL allows you to make some of the more abstract terms like caching, conditional execution, permissions, etc. more explicit.

To your last point, I have experience with using CD in production, but not to the scale where I have builds stepping over each other and causing issues. I agree that serial builds are important in this case, and is something that I will need to look into (conceptually it sounds pretty simple).

Creator of Cicada here. I created a new DSL because I wanted a more declarative, expressive, yet capable file format, something like Dockerfile but with programming capabilities. In terms of "just running commands", Python (or any other language) is just as good if not better.

Where Cicada shines though is that it makes higher level concepts like conditional execution and caching front end center, so you can describe more with less lines. The DSL itself isn't very mature at the moment, but I plan on adding stuff like allow/deny capabilities and more in the future.

Creator of Cicada here. I mentioned this in another thread, but I have an experimental "GitHub Actions to Cicada" converter tool I'm working on that makes it easier to import GHA workflows to Cicada. GitHub already has an importer tool to import other git providers to GHA, but of course they don't have any export features.

Like you said, there are lots of intricate details you need to get right, and each CI/CD provider has a different ethos about how CI should be done. What I'm trying to do with Cicada is create one workflow format that gives you the power of GitHub Actions with their numerous event triggers, but make it work for other providers like Gitlab. Having one format that works with many providers is better than converting multiple formats back and forth, IMO.

Nice product! I haven't heard of Garden until now.

Adding a CLI for Cicada has been on my todo list for a while, and is long overdue! This should be easy enough to do.

Creator of Cicada here. Currently I am using GitHub SSO to reduce spam for the logins. When installing Cicada locally though you need to create an admin account, though there is no way (in the UI) to create new local users. Adding more sign-in methods (ie Gitlab, Bitbucket) is definitely on my radar.

Creator of Cicada here. Thank you for these questions! I'll try and answer them all:

0: Secrets are stored using Vault. Read this commit message [1] for a full breakdown.

1: Currently you cannot include other .ci files. I have been in feature creep mode for months now, and I've been forcing myself to stop adding features and start talking to users. The goal is to make Cicada more or less a general purpose programming language, but the first step is making it work well for defining CI/CD workflows.

2: If it is necessary, I could back-port the self-hosted runners to Python 3.10/3.9 or earlier. And, since the runner interface just uses websockets, I (or someone else) could make a runner in a different language, ie Rust or Go.

3: There is a "event" global variable that includes info like "event.branch", which is the branch that was pushed, but it does not include stuff like the default branch. Currently you could do `on git.push where event.branch is "main"`, but something like "event.branch is event.default_branch" would be even better. I'll work on adding that.

The value add currently is that Cicada is FOSS, platform agnostic (works with GitHub/Gitlab), and uses a language that consolidates the workflows and scripts into one manageable file format. Existing CI systems are already packed with features that people expect, so the current struggle is catching up to this and then adding more on top of that. I'm trying to focus on what sets Cicada apart: That it gives you more control over your workflows, while being expressive and easier to manage then YAML and shell scripts.

[1]: https://github.com/Cicada-Software/cicada/commit/2659f79b500...

Thank you for the feedback. What in particular is making it hard to start trying it out? I'm curious because lowering the barrier for entry is important for getting people to start using this.

I cannot seem to find any references to "goto refs" in GitHub Actions, where is this documented? I've never seen this before so I am curious

I havn't heard of any products or services that test the actual CI pipelines themselves, which seems like an interesting target market. I can't say how many times I have wanted to test out some CI workflow on some throw-away clone repo without mucking up the live repo.

It's good to know that you can use includes with Gitlab, And, from my quick glance at that second link, the setup process seems a bit involved. I don't like having to set up new user accounts just so that I can run a CI workflow locally.

I think these are all great points! I have came across Concourse CI, but have not looked too deeply into it yet.

I hadn't thought to use TOML for configuring CI pipelines, I'm curious what that might look like. TOML is indeed very flat, so it would be interesting to see what the equivalent TOML pipeline looks like compared to a YAML one.

In addition to some of the other answers about the match statement, mCoding has this excellent video[0] explaining the match statement, specifically using it to match complex AST trees, like Refurb does. Note, not all of these checks use the match statement[1], sometimes it is more verbose to use a match statement vs a normal if/else statement.

[0]: https://youtu.be/ASRqxDGutpA?t=470 [1]: https://github.com/dosisod/refurb/blob/master/refurb/checks/...

I think this would be a good change. I agree that the imported names should be spelled out: You shouldn't have to go hunting down the docs before you go and change anything. Again, this would increase horizontal space, but I think it is for the better (the errors are already pretty long anyways).

I think that using Refurb on new projects would be a great use case, but even for existing/old/legacy projects, Refurb can still be a good option, it really just depends on what is already there.

I agree that going and updating your company's codebase would probably be a bad idea, considering that there are some kinks here and there. Refurb tries its best, but it is very early on in its development, so errors that are emitted should be taken as suggestions, not gospel.

I have never heard of Sourcery, but I will take a look at it. From what I can tell, Refurb is a CLI only tool (at least, for now), whereas Sourcery is more aimed at IDEs. Refurb should play along nicely with most other linters, though you may need to configure things first.

Refurb can take any number of files or folders, for example:

refurb file.py folder/ another_folder/

Hope that helps!

The reason I didn't just create a pylint/flake8/mypy plugin is because they all use a very specific methodology, whereas I wanted the freedom to architect it the way I wanted. Also, I really needed good type information, which is why I used mypy[0] as the AST parser/type checker.

[0]: https://github.com/python/mypy

Also, I don't know if this was part of your critique, but are you also mentioning that the "a" variable is not in the suggestion? The reason behind that is long names, such as "super_special_file" would greatly increase the length of the error messages, making it hard to see the "meat" of what is being changed. Hence the x and y naming.

Perhaps I should add a flag to allow verbatim (or close to it) errors. Recreating the source lines exactly will be pretty hard, but I think we can do better here.

Yes, Refurb error messages are not gospel: Make sure the changes make sense! Thank you for bring this bug up, I will make sure to address it tomorrow.

Interesting, I have never heard of this project before! From looking at the README, Refurb is more focused on simplifying the codebases, making them more expressive, whereas pyupgrade seems to have the goal of upgrading you from version A to version B (specifically Python 2 to 3). Near the bottom there are some newer Python version (3.6+) related upgrades, which I was considering adding to Refurb. I expect there to be some overlap between my project and some of the existing projects out there, so I will have to weight the costs of adding these in vs leaving them out.

Thanks for mentioning that project!

You're right! I guess you can't rely on sets being iterated over in order (not that you ever should rely on that). Moreover I was trying to show that you can use lists, tuples, and sets as temporary iterable objects, though but no-one (that I have seen) uses sets in that way.

Something that I forgot to mention in the other answers similar to this is that the Path().read_text() check only applies if the open() block is a single line, and that line is only used to read the contents of the file. If you are doing other things in that block, no error will be emitted. I agree that open() is pretty ubiquitous, but in certain cases (ie, if I am already using a Path object), using read_text() is a nice one-liner.

Yes, there is a --explain flag for explaining the checks in more detail. I thought about adding a "use --explain ERR for more info" at the bottom if there is 1 or more errors, probably would be best to add this in.

Thank you! I have found that pattern matching has made this project a whole lot easier.

One of the other cool features about this project is how the check() functions are loaded based off the type of the node argument:

https://github.com/dosisod/refurb/blob/master/refurb/loader....

This also supports type unions, so you can register your check for binary and unary expressions by typing node as `BinaryExpr | UnaryExpr`.

Because tuples cannot change over time, they are (slightly) faster to create compared to lists. Like you mentioned, they are being created just to be iterated over. It is a small enough of a performance improvement that it more of a style choice then anything.

Bonus fact: You can use set iteration in for loops as well! This has the added benefit of sorting the values as well:

>>> for x in {2, 1, 3}: ... print(x) 1 2 3

You didn't ask for that, but I felt like sharing it, to there you go

I agree, better clarification/classification of errors would be nice, as well as the different tradeoffs you should expect from using a particular check. In particular with the Path() related checks, they can seem contrived when used in isolation (open() is a pretty common idiom, like you said), but they become really powerful when you chain them together:

(Path("some/deep/folder").parent / "file.txt").read_text()

Similar to black, Refurb is opinionated. There are probably going to be checks which you will ignore. Somewhat of a non-goal with Refurb is bringing to light some of the different ways of writing Python code, and the pathlib module is, IMHO, a very underutilized part of the stdlib.

Yes, I would say that is correct! Like I mention at the very bottom of the README, Refurb is not meant to be an all-in-one linter or formatter: Tools like black, flake8, mypy, etc. already exist, so trying to recreate what they do would be a lot of work. Refurb is more of an addition on top of the tools I previously mentioned, and less of an altogether replacement.