dkoston

Team building is pretty constant across all the “flavors” of agile. Building good communication patterns and trust isn’t coupled to the “how we execute” details

This is a standard agile exercise called “Rules of Engagement”. Should be run with any forming team. I’m always surprised that this knowledge isn’t more commonplace.

It depends on the day but not such a high percentage usually. Our design phases don’t last as long (software only).

Virtual meeting fatigue is real because there aren’t as many natural breaks. It is important to make sure that people know they can excuse themselves for water/food/bio breaks.

I’ve also found that virtual whiteboarding is slower for me as I’m used to picking up a dry erase marker. It takes time to adjust to the different work stream but a huge benefit is the quality of the end product (no sloppy hand writing, easy to modify).

I don’t have any direct advice geared towards your situation as our white boarding and planning phases are quite different in my industry.

Also using Miro and enjoy it.

In addition to the software, make sure your team has good hardware. A screen large enough with high enough resolution is critical to see the whiteboard. What’s also important is that video from meetings takes up screen real estate and with multiple participants, having enough size and resolution to see them all helps you stay connected. External webcams that can be adjusted are helpful also.

Thanks for putting this together. I’d recommend adding some indicator that the results load async. I thought you had zero restaurants in Austin and left the page after a second because there was no loading indicator. In fact, until I got to Pittsburgh, I thought it was a site with no listings asking for my email (because Pittsburgh loads way faster).

(Adding an index to your database for city or however the queries are formed would probably also help).

Sometimes it’s nice to have a list view. Google maps only provides geolocated results in a small list so you’d have to move around the city quite a bit to find the whole list.

It can be useful to be remembered of places you like but haven’t been in a while as well.

This was essentially the story of how we built a successful software company at cPanel. Our customers would describe things like “I want a button that does this” or “a script that does that” and we had to make many assumptions about what they really wanted. Finally after determine we were spending massive amounts of time on unused features, we developed everything as a workable “beta” feature. The ones that we got a ton of praise or complaints on kept getting developed. The ones with silence stayed the same.

This was before heat map software and click tracking which can do a great job of augmenting customer feedback.

The key part that we found out is that if you take this approach, you have to couple it with high quality software support. We hired and trained folks who knew the entirety of our business and tech (Linux, DNS, HTTP, hosting, etc). Our tech support reps were paid much more highly than others in the industry but it gave us more benefits than just being able to support customers when beta software was ship to prod, it also gave us a training ground for future software engineers, QA folks, SysAdmins, and Product Managers.

What I think a lot of people forget is that software is one piece of the puzzle in solving a problem, it’s absolutely not the whole solution. If you design a process based around connecting with customers on a regular basis and having feedback flow throughout your company, you won’t have to “make perfect features”.

I’ve only worked with a small number of companies since that have understood this. Most I’ve worked with have created communication silos and sit in rooms trying to dream up the perfect set of requirements.

Do you not have frequent one-on-ones and regular conversations with your remote employees? How do you learn about their motivations, goals, current situation, or changes?

If you’re trying to replace human interactions and relationships with your employees with single question surveys, you’re going to miss out on all the important context.

Getting good output isn’t about rushing as fast to a goal as possible and not taking time to make sure things are going well. It’s about setting goals that are measurable, coming up with hypothesis about how to get there, and checking in regularly with an objective lens to see if you’re on track or if there’s a better approach.

Avalara and Vertex both have options that integrate well with major billing systems.

I bought a VW GTI 20th anniversary (a numbered car) and the transmission broke on my third drive (wouldn’t shift out of third) which was a catastrophic failure. This was about 50 miles deep. Took a few hours to get towed. VW said they would fly in an expert from Germany to diagnose the issue. I opted to get a different car (new number, same model) and that one worked fine. Did I have massive ill will towards VW? no. This was even a numbered car that technically “could not be replaced”. Errors happen.

Statistically, a few hundred thousand model 3s have been delivered and many are well past 6k miles. I think you should wait and hear what Tesla’s reaction is before writing them off.

Things you should consider are:

- they have sold so many cars in the past 2 years that service center growth is struggling to keep up and wait times are long

- Tesla vehicles cannot be serviced easily by third parties so you have no option if Tesla says something you don’t like or wait times are long

Choosing the one broken car out of hundreds of thousands delivered as a reason to not buy one doesn’t make much sense. All manufacturers deliver defective vehicles. Looking at statistics on how many are defective per capita from each manufacturer would be a more reasonable course of action.

Pretty nuts if this is reproducible. Would be similar to decades of progress in shrinking super computers down to pocket sized phones in a few years time.

That’s understandable. I’d imagine at some point you’ll need to decouple the monolith a bit in order to work effectively as you scale. Best of luck with the challenge.

I think I see where you are coming from. Being as we use different tools, we wouldn’t allow a pull to be merged if it wasn’t up-to-date with master which is similar but a different approach. You’ll have to check at merge time because getting up-to-date could take a while and master could have changed. Jenkins does this and it can be done in other CI/CD systems with a bit of custom code.

I’d imagine at 1,000 developers and with a monolithic codebase, you’re looking to minimize test runs both from a time and cost of runners perspective.

You may also want to look into Zuul or Bazel if cost of test suite runs is a factor in coming to this solution.

Good to hear, that’ll make change management less of a chore.

I think the main thing that was missing for me is the rationale behind building this system rather than building a workflow in one of the existing CI/CD tools. Was there a throughout bottleneck in existing tools? Was there something custom about your workflow that wasn’t supported elsewhere? I may be wrong but the workflow you landed upon seems pretty common so I’m curious as to why the need to build and maintain a tool in house for this?

Interesting.

Would you say this is more of a decision based around the constraints of using GitHub or more of the ideal process for Shopify’s needs?

I’m curious because the article doesn’t mention the core reasons that you chose to write your own CD tool versus the other options that exist. The workflow you describe seems readily available in most tools. Perhaps the throughput was causing other options to break?

Thanks, I was hoping for more of this in the blog post. Since tools are just an expression of process/policy, it’s more interesting to here about the process and why than it is about building “yet another CD tool”. Appreciate the thoughtful and thorough response.

The major pain point I agree with on develop is changing the defaults to merge to that rather than master. It’s a shame this is not easier to do in git/github.

I’m not sure I agree with “develop can still be broken” as an issue that supports a queue. Whether it’s a queue or develop, one should run CI on each change to validate that merging it to master will not cause issues. It’s possible for both to be broken via the same scenarios just as it’s possible for master to be broken. Since CI runs before the branch is merged to develop and upon merge, a failure would “stop the world” and prevent more code from being merged unless that code fixes the failure.

I guess I’m not fully understanding how a queue prevents this. Since you don’t have a full picture of the state of master until something is merged from the queue, how do the CI checks in the queue prevent things that branch-based CI checks wouldn’t prevent in a “develop” branch? With branches and develop, pull requests remain open until they can be assured they merge properly with develop as well.

For clarity, I’m not arguing that a develop branch is the way to go, I think CD is much better.

Maybe I’m missing something big here but using multiple branches is permissible in other setups also. You can cherry pick a bunch of commits to a branch and test permutations but only certain branches get deployed to staging and production based on rules.

I’m glad that Shopify has found tools and a process that works. Honestly, I’m just having trouble comparing and constraining this to the other tools that are out there. The article never speaks about other approaches and whether or not they were considered and why you decided to go with a queue. It’s not clear to me if this was a case of improving the existing queue system because it was already in place or whether or not the queue was specifically chosen again because it was better than other alternatives (and why).

From what I gathered in the article, that’s the case now but before the queue required manual merges.

That’s what trying to get at in a “more polite” way. Many people use a “develop” branch to stage changes before master so they can run through CI. I’m curious as to why they used an untested queue instead. It seems like they wanted to cherry pick what commits made it to master rather than going in chronological order.

What’s odd is that they seem to characterize this as a tools issue rather than a process issue. There are plenty of CI/CD tools that allow for a similar or the same workflow as what they created. It’s also kinda scary that there’s not an emphasis on the overall SDLC and how the specific attributes of a branch or commit should/shouldn’t affect the process. You’d think at 1000 developers it’d be very important to define what is “launchable” as well. Haven’t worked with 1,000 on the same project but even with 20+, the standards and practices around development were always more important than the tooling. Tooling was just meant to represent workflows that were already defined.

Interesting. It seems like you have a very flexible process of how to launch code which could contribute to issues with visibility and rollbacks.

I’m curious as to why you had a queue instead of a develop branch before moving to CD? Was this to allow arbitrary commits to be launched to production rather than getting them batched by time?

Rather than being pedantic about the definition, maybe you could share your experiences with why that’s superior to validating each branch? Being dogmatic about a definition rather than experimenting with works best in production at your business seems illogical.

Sorry if my comment was unclear. I consider the queue to be a “branch” as well. Many people use a “develop” branch instead of a queue in this instance. The queue appears designed to allow arbitrary selection rather than merging in order (though the new solution with CD seems generally in order)

Totally agree that CD is required with this many commits. It’s commonplace on teams with many fewer developers. Was surprised to see you folks roll your own workflows rather than using other systems.

Would also be interesting to see if you tag commits that go to master in instrumentation systems so you have visibility into production metrics and can correlate them with what code was running at the time.

It would be useful in this article to hear about what content is acceptable in a merge request. For example: can these all go straight to queue because they use feature flags? Are commits a "single piece of work", etc.

Not to sound like a downer but this is really an article about fixing a broken process because not running CI on branches before merging to master goes against best practices. Would have loved to actually hear about their work process as this whole article could be summed up as "not running CI on branches before merging their commits to master is a great way to ruin master".

In my experience, you don’t. With tons of IOPS you need EBS or crazy expensive instances.

Instead you use a cloud like google cloud where you can add NVMe SSDs to whatever instance type you need and configure custom RAM and CPU instead of picking from the super expensive AWS instances with no configurable options and almost always the wrong resource allocations for your workload.

Source: testing my infrastructure that requires 60,000 iops on both google cloud and AWS and it being 1/4 the cost and higher performance on Google. Of note: this was a very high throughput streaming data application. YMMV for other applications.

I've got about the same members and attendees. This will drive the cost up about 3x but honestly, that wasn't the part we were concerned about most.

A couple of things: 1. At no point in this change did meetup suggest they were offering new things for subscribers that would justify a 3x cost increase. 2. Asking our members to start paying would now incur so many logistical costs on us: we'd now have to spend a bunch of time discussing with our members why they had to pay for something that was previously free. We'd also have to create email copy describing the change to them. We'd also have to deal with customer service issues where people thought we were charging them rather than meetup and they'd want a refund if they didn't show up. We'd also have to explain how charging met or didn't met our core values. 3. $2 isn't enough in the part of the world we're in to "prevent people from not showing due to cost". That's less than a cup of coffee so it's not much of a sunk cost to not show up. We could of course charge much more than $2 but that's a radical shift for something that was previously free. 4. Do we now charge or sponsors on an "on demand" basis based on how many members that show up? Or do we overcharge every sponsors based on the fear of how many people will show up? What used to be a $X per event cost has now added more unknowns that make logistics harder.

#1 is a small expense but #2 is a radical shift in the values and relationship between our group and our members. The logistical costs of changing how we interact with our members are MUCH higher than the added subscription fees per year. These kind of changes make almost no sense to us as we can't explain how meetup needs a 3x cost increase to deliver the same product it did yesterday.

In addition, the communications about this were terrible. A total of 0 times did they explain why this was happening and what our increased costs would bring us. They also tried to frame this as "lower subscription fees" but the total cost of the platform went up.

This degraded all trust we had in the platform and kicked off a search for alternatives immediately.

FYI, I told the Meetup focus group the same thing.

1Password allows you to add family members with access to specific values either read-only or read/write. The system for adding access is multi-step so unless you add someone to a vault they shouldn’t see, you have the flexibility to share as little or as much as you want. Since you can name the vaults you can name them things like “Shared with M Toussant (Attorney)” or “Samir Martha and Paul” which can make it easy to determine where to store what secrets. Have been using Business for a few years with some of my companies and Family with my family and have had good experiences. You can initiate recoveries as the administrator as well which has been helpful in both cases.

1Password has a Families plan for this.

I agree that this stuff is common place and I definitely understand that in today’s world of noise that you have to market yourself and your papers to get attention.

I was glad to see them add the disclaimer as hopefully it’ll turn away people who see “hey, here’s this secret management project made by a bunch of professors and PhDs, I should use it because they are super qualified”.

My goal was to provide feedback to the students about how people evaluate code “in the real world”. It’s not helpful for them to be applauded for “sub par” code but you raise a good point that it can seem harsh to not provide feedback on the “meat of the project” which is their paper and research.

You can use a PRF for the domain to derive additional secrets. It simply depends on the scope of the project and who you think your users are.

For example: if your user base is cryptocurrency wallet holders who have multiple secrets for each wallet, will they construct a secondary library on top of yours to manage those additional secrets? Why wouldn’t they choose to derive each secret independently? If millions of dollars are at stake, would you risk any shared state from your secret derivation function?

It would be unnecessary to derive additional secrets with this library to prove the concept in the paper so I don’t think it’s necessary if that’s the goal of the code. However, if mass adoption of the techniques you’ve created is your goal, a more user friendly API which doesn’t require each end user to “roll their own code” to manage multiple secrets should be a goal.

One of the first major projects I worked on was essentially a wrapper around open source software that provided “ease of use APIs and UIs”. Because the average user was not technical, the convenience wrapper became highly valuable and is used by hundreds of millions of sites today (cPanel).

One of the biggest challenges as a technologist is to understand to what degree most people are not technologists, even fellow programmers. For example, I’ve worked with skilled programmers with impressive resumes who had issues troubleshooting CORS because they never learned how headers are defined and where to look up the RFCs.

As mentioned above, providing an API for multiple secrets could be out of scope for a bunch of reasons. If you’re looking for mass adoption by developers, I’ll wager an Omakase at the sushi place of your choosing that it’ll be required.

For #2, it seems unlikely that people would have a single secret.

I’m not sure if the codebase is more just to prove the concept from the paper or to potentially get adoption. If you are looking for people to adopt the project, I’d suggest a way to store and retrieve multiple secrets.

Best of luck with the research and project!

There’s no misconception. Until I commented, there was no disclaimer stating that the code was not ready for production use and was for research purposes only.

Very few people are qualified to review and critique this type of codebase so it’s dangerous to market a repository backed by multiples professors and PhDs as “it does X, Y, and Z”

It would be great to live in a world where people did a serious review of projects before downloading them but that’s not the reality we live in.

OP didn’t provide any context either that this was focused on reviewing the paper and concepts in it so I think what you mean is that “you think people should focus on the paper” but “we focused on the code”, that’s not a misconception, it’s a lack of clarity by both OP and the project website about what’s important.

If the code was irrelevant, it should be removed from the website so that the audience could focus solely on the paper.