ArturT

I've created a knapsack ruby gem for CI parallelisation that has over 122 million downloads. Primarily due to the fact, Gitlab is using it.

I spin off https://knapsackpro.com from the knapsack gem and we are helping our customers run fast CI builds.

I posted my add-on in Heroku Marketplace over a year ago https://elements.heroku.com/addons/knapsack-pro and did not see much organic leads from the marketplace itself.

Anyone using Heroku Marketplace as well? Any ideas how did you approach promoting your add-on? Thanks.

Sidekiq Pro version was inspiration for me to create a pro version for my Ruby gem called knapsack that automatically splits tests in parallel on CI nodes/jobs.

With the knapsack_pro version I focused on common needs of users and there was simple validation method to verify if particular feature was needed. People just pay if it brings value.

In case of free open source tool, people tend to want something but you don't know if they actually will you use it and sometimes people demand a thing while you have to put the hard work.

When you do paid version of your tool then somehow people are polite. :)

Nowadays I have also support for running parallel tests in JS.

More at https://knapsackpro.com and here is detailed technical explanation how it works https://docs.knapsackpro.com/2020/how-to-speed-up-ruby-and-j...

Maybe because of running Rails on Heroku is simpler than with Docker.

If you have no tests at all I would start with simple E2E tests to ensure your critical parts of app are working like sign up page and billing. You can add later other important sections of your app to ensure happy paths for your features are working fine.

When I started my own product for Ruby & JS developers https://knapsackpro.com I was doing mostly unit testing and later relayed on E2E tests for testing user dashboard to ensure happy paths are covered.

It's very easy to introduce weird bugs and it's much faster to run CI build to test your app than manually verify if it works correctly in your browser when you make some changes.

knapsack_pro gem uses Knapsack Pro API which is paid. See more at https://knapsackpro.com

It works similarly like tools for exceptions tracking, i.e https://www.honeybadger.io is paid but has MIT client https://github.com/honeybadger-io/honeybadger-ruby

Other than that I do support paying customers.

I started with writing open source Ruby gem https://github.com/ArturT/knapsack for running tests in parallel on CI and later I gave a talk on local meetup event for Ruby developers.

With time I wrote more blog posts https://docs.knapsackpro.com and shared on Reddit, Twitter, LinkedIn profile or Ruby groups. From time to time posts landed in RubyWeekly newsletter.

Also asking users for recommendation might help.

I connected with each person on LinkedIn I talked to or who signed up and I followed them on Twitter. People who like your product can easily see you publish new content and can like or share it.

End-to-end tests tend to take a lot of time when project grows. Some tricks like recording API calls with Puffing Billy can help save some time but still majority time is spend in browser.

This problem triggered me to work on knapsack ruby gem to solve problem how to split tests across parallel CI servers and run my whole test suite faster. E2E tests can vary in time execution so I decided to split them in a dynamic way between CI nodes to auto balance tests speed between nodes.

Recently someone let me know about fix for puffing-billy gem to work with my knapsack_pro Queue Mode. https://github.com/KnapsackPro/knapsack_pro-ruby#how-to-conf...

For annoying flaky features tests, I use rspec-retry gem to repeat the test a few times before marking it as failed. It helped for integration tests with external sandbox API.

I noticed discourse had a lot of flaky tests while using their repo to test my knapsack_pro ruby gem to run test suite with CI parallelisation. A few articles with CI examples of parallelisation can be found here https://docs.knapsackpro.com

I need to try the latest version of discourse code, maybe now it will be more stable to run tests in parallel.

Back in 2014 I work on a project where we had ~15min test suite running on CI and this was painful so I started working on open source solution knapsack for CI parallelisation for Ruby tests.

Later on developed more advanced way with dynamic tests allocation across parallel CI nodes not only for Ruby to get CI build as fast as possible. To give you some idea how it works check https://docs.knapsackpro.com/2017/auto-balancing-7-hours-tes...

or watch video https://www.youtube.com/watch?v=hUEB1XDKEFY

I work on knapsack gem to split tests across parallel CI nodes. To do optimal balancing I developed a dynamic test suite split with Queue approach.

Here is example for Travis CI https://docs.knapsackpro.com/2018/how-to-run-travis-ci-paral...

And this is the graph showing the difference if you compare knapsack_pro dynamic tests split with what CircleCI does which is basically only time-based split. https://docs.knapsackpro.com/2018/improve-circleci-paralleli...

Setup deployment takes some time so I prepared blog post and an example Phoenix app how to do it with edeliver, distillery and Ansible playbooks http://blog.lunarlogic.io/phoenix-app-deployment-with-ansibl...

We spent some time with friend doing research and trying to connect all pieces to make it work.

It's easy to lose yourself as a developer in problem-solving mode instead of looking at the big picture of a project. It's great to have people around you that reminds you the code is just a part of the more complex story - how to help your client solve a business problem.

Hi HN,

I turned my open source ruby gem knapsack into indie SaaS product. I was inspired by sidekiq gem. Here is the short story:

In 2014 I had a problem with large test suite in my customer Ruby on Rails project. I developed the open source gem knapsack to split tests among CI nodes in parallel.

In 2015 I had an idea to turn it into small SaaS product and by the end of the year, I had a first early adopter Buildkite team.

This year I released a new solution to dynamically split RSpec tests among CI nodes and I tested it with a few companies like Pivotal, VTS, Namely.

On one hand would be great to live in a world with small and fast test suites but on the other hand, we know how it is in the reality. There are large projects with dozen minutes or several hours test suite time execution.

I also wrote the article inspired by work we've done with one of my customers: http://docs.knapsackpro.com/2017/auto-balancing-7-hours-test...

I'm curious if there are similar problems with getting optimal time execution for large test suite across multiple CI nodes in projects done in other languages than Ruby. Have you experienced that problem? I'm thinking about building my tool in a different language and would be nice to test solution on the real project.

I'd love to hear your story about working with big test suites and what languages & tools you use. :)

I created ruby gem knapsack to speed up testing and I built a Pro version which is SaaS https://knapsackpro.com

I definitely learned a lot during last year while emailing with people interested in the product. Thanks to that I improved my tool iteratively while the early adopters discovered new areas or edge cases about testing I didn't even think about.

This week I published success story about running tests across 100 parallel CI nodes with my tool: http://docs.knapsackpro.com/2017/auto-balancing-7-hours-test...

It's pretty popular problem. I recently got this book "The Positioning Manual for Technical Firm" by Philip Morgan from my friend and it's basically about how to move from generalist to in-demand specialist. The book has exercises how to determine for who and what service you could provide to position yourself as specialist. Maybe you will find this useful.

Do you have large test suites in your company projects? How big are they? How do you deal with long running tests?

Probably keeping a few small projects with fast test suites instead of monolith would be a nice way but we all know it's not always that easy.

I'd like to build a client library for auto-balancing tests across CI nodes in other languages than Ruby and I'd love to learn what test runners tools you use in your favorite programming languages.

I recommend nice book "Don't just roll the dice" by Neil Davidson.

http://download.red-gate.com/ebooks/DJRTD_eBook.pdf

When I released my first ruby gem for rspec test suite parallelisation I decided to write a short blog post about it. If you don't have your own blog you can write the post on your company blog. Thanks to that you will get some promotion because your company or colleagues can tweet or share it on Facebook.

Another thing I did was to post my article on Reddit in the ruby group. This seems to be a validated way of getting some traction. Make sure your friend read the blog post before publishing so you can iterate after feedback to make it interesting for people who will first time hear about your tool. :)

Oh, and here is my blog post: http://blog.lunarlogic.io/2014/parallel-your-specs-and-dont-...

What do you think about http://sidekiq.org gem approach where you have free open source tool and paid pro version. I did similar thing with my open source gem knapsack for test suite parallelisation in ruby. I run pro version at https://knapsackpro.com

I created pro version of my ruby gem for test suite parallelisation https://knapsackpro.com I released it last year but started validating it by charging users since July this year. $1K/month is my goal by the end of this year, so far I'm halfway there.

This demo page looks awesome https://lunarlogic.github.io/auroral/

"How does one pick the test cases for something like the GildedRose? Emily Bache, who provided the test cases for this essay and who ran the Python user group kata that I participated in, developed the test cases manually." I created my tests based on empty example from Emily https://github.com/emilybache/GildedRose-Refactoring-Kata/bl... and right now I assume I have my own tests covering all edge cases because I discovered edge cases with golden master technique and I added missing tests to gilded_rose_spec.rb file.

"The solution shown here "recorded bunch of input examples and output results from the program we wanted to refactor." However, are those seeded versions themselves sufficient?" Golded Master tests are sufficient because I recorded all possible cases of inputs/outputs for reasonable amount of sell in days. See https://github.com/ArturT/GildedRose-Refactoring-Kata/blob/s... https://github.com/ArturT/GildedRose-Refactoring-Kata/blob/s...

"Even coverage analysis may not be sufficient." Agree.

"Coverage assisted test development is, in my opinion, an underused technique." I guess you're right. I wouldn't relay only on test coverage.

In my repository https://github.com/ArturT/GildedRose-Refactoring-Kata/tree/s... you will find completed test suite also with golded master tests. There are tests for additional item "Conjured Mana Cake" as well.

Is anyone use Vim without touch typing? I learned touch typing with typing.com course and the main reason was that I wanted to use Vim more efficient.

I was curious how local ruby meetup event affected my gem daily downloads. Does blog post lead to higher downloads? How long tail will look after all?

I jumped to RubyGems API but there is no way to get such data so...

Recently just for fun and as a way to learn of MEAN stack I created RubyGems Tracker to collect daily downloads and other stats. If you have your own gem feel free to add. Feedback & pull request much appreciated.

How you track your gems?

We recently had a discussion with a client about estimation. From the developer point of view it's hard to predict the date because of many things which might happen during development and we are not aware yet about them at the moment when we need to declare the date.

Now we are using historical data to predict how many calendar days the features scope may take with a specific confidence level.

From the client perspective dates are important because they need to coordinate release of multiple projects and dates help them release right stuff, in right order, in right moment on production.

Here is article about our approach http://blog.lunarlogic.io/2015/project-management-estimation...