We have a single sandbox environment currently, but different customers are asking for varying amounts of delay before feature release. How do you manage bug fix deploys in these cases as well?
Interesting, can you explain this a little more? Some of our customers chomp at the bit for particular features, and when we can we give them early access. We do not charge them more for arguably huge amounts of value add to the product.
I wonder how to have these types of conversation, internally and with users. I'm a PM but have a direct line to management. I'm frustrated with how much money I believe we are leaving on the table, even if it doesn't directly benefit me.
Someone can forget to have a separate set of terms and suddenly the support team are on the hook for supporting previews to a greater degree than expected. If features aren't locked then suddenly features get jammed in because the preview is a good hook for customers who haven't yet acquired your product.
If they say "OK, great", you've got immediate revenue to bring back to your management, and a strong case to continue doing more of the same.
Customers routinely over-and-under-estimate future value, compared to what they're then willing to pay.
They're often wrong in both directions, they usually won't pay as much as they think they will, _and_ they'll often pay far more than they think you will.
There are many, many reasons for both kinds of errors, many of which can be de-risked in advance and safely ameliorated.
*I'm willing to play ball too, e.g. for a staff position I might take extra vacation in lieu of more cash, tradeoffs like that. But basically I'm not gonna take on extra hassle without being incentivized — because I don't have to and don't want to, all there is to it
- this incurs real costs, so we should charge for this. (The charge doesn't have to be related to the costs! But if you're doing something extra there should be an extra line on the invoice)
- this customer wants something that other customers don't have. There's all sorts of real-world services where you can pay to board the plane first or get to the front of the concert. These may be several times more expensive than the regular service. That lets you phrase it in fawning terms ("you can join our Super Elite Ultra Platinum Preview Fellows") and thereby extract much more money.
I read this as “A customer that asks for advance notice of feature changes is likely to be a big enterprise customer that values stability and predictability, and the fact that they’re asking suggests your product is important enough to their process to matter, so you should take this opportunity to charge them a ton of money for an ‘enterprise support plan’ which includes advance notice of new releases, etc.”
As the GP said, it’s a signal that they’re a certain type of customer.
That's also engineering effort: you need to maintain feature flags or a different environment, which is not cheap in the long run
So to paraphrase Arubis, this is a business question,not a tech question. At what price point is the technical investment covered ?
We did once have a customer with their own environment with a different version deployed, as they were initially a major client with such demands but after a few years it became clear it wasn't worth the maintenence effort to special case this one client so we made it clear in contract renewal one year that this would be going away.
The roll out of feature flags is a separate process.
Also recommend making sure customers know that this is time-limited, otherwise you end up supporting too many different configurations of your product.
Also if these are customers you're in regular contact with, you can preview your roadmap or demo features that are coming soon. This helps them feel in the loop, and gives them a chance to provide feedback.
Fowler has a good write-up about different types of feature flags: https://martinfowler.com/articles/feature-toggles.html
Also, you might look at cPanel's notices that they send out where they explain upcoming feature updates well in advance of the rollout. They also have different release cycles, where the newest features are released to the EDGE cycle instead of the STABLE, and then the long term release cycle remains unchanged except or security updates.
This.
Problems they might be trying to solve:
* want to be able to jump on new features as soon as they are released, but need to plan with the eng team
* want to be "in the know" so not surprised by new features
* want to be able to give feedback and help direct the roadmap
* have been bitten by bugs in your software (or other vendors) and want burn in time, especially if you upgrade them without their choosing (as is typical in some SaaS models)
In my experience it’s because if you roll out a new feature (or non-trivial UX change), that can wreck someone’s day and cause internal support issues into the company that uses your product.
Consider your rollout of new features to be least impacting, don’t confuse someone who’s using your product to get work done by changing something that breaks a workflow that worked yesterday etc.
You can put new features in new sections, or need special users to activate them.
In some cases we would show screenshots of upcoming features to hear about any other feedback, but otherwise focused on new features not being disruptive.
Seems like putting as many things that might risk this under settings/configurable as possible? This does extend development to have to gate everything in this way though.
I’d dig deeper into the why with your customers, after we understood moving buttons around causes people’s day to be hard we did it only when really justified etc.
I wouldn’t hold up releases for customers, but do dig into the why more with your customers about why they want this notice, you may find it’s just an assurance something like a workflow isn’t totally different next Monday morning.
We have two special case preview environments. They're set up just like production, but with lower capacity and no SLA.
The first one is general previews of what we're planning on releasing; it's a normal part of our release process and every release is there for 2-14 days before going to production. All customers have access; some use it extensively, and some basically ignore it.
The second one is for specific in-development features; in theory we can spin up multiples of this, but in practice we don't want to diverge too wildly. Generally we showcase one feature at a time, and we gate the features by customers.
An example of this is a friend asking me how to weld a wire to a thin metallic plate. I digged further to find the actual problem for which his solution was to weld the two. He said it was a fuse's wire that melted and he wanted to weld a bigger wire to avoid that, to which I replied: don't. The whole point of the fuse is sacrifice to protect down stream equipment.
The same when a relative asks how to format a disk as their solution to solve a problem.
Generally speaking, implementing customers' solutions is a recipe for disaster, especially with enterprise.
Ask what it is they are really trying to accomplish, and figure out a way to solve that. Listening for problems instead of solutions or implementations.
There are always requests and negotiations in play. But those three channels sufficed almost all the time. At times we'd combine channels if there was an urgent need and agreement from customers.
The slowest customers were the most enterprise-y. Things almost couldn't go slow enough for them. When there were complications to the release process, they could be a release version or two behind and there'd be a catch-up release.
We also had customers outside all of this, and had custom releases, and we charged very high fees for this. Ultimately, we were able to bring them back into the slow release because the extra work wasn't really worth it. Part of the requirement for these kind of customers is to have additional engagement efforts to make sure their internal teams were really aware of the details and contingencies of releases. Big companies move slow and need over-communication, and you can't trust them to do it themselves. It was more effective to take that responsibility on ourselves.
I preferred to take the things that were difficult and do them more often. Keeping a cadence to things kept everyone together. The minute you break cadence -- and entertain special circumstances -- it's really hard to get back on cadence, and that cadence includes testing, notifications, documentation, feedback, approvals, releases.
Can you please help us understand the purpose of the advance notice/release delay is? What benefit does a process like that provide to you or your organization?"
---
I generally use some variation of that ^^. Having some customers on different release tracks is very painful in my experience, especially if it's done without properly planning for it in your release process. For example, let's say that right now, anything committed to `main` goes out to production via some automated process. What happens when you have a security bug that needs dealt with ASAP? In my org, this usually means that the security patch gets rolled out to everyone, and then when it's time for customers on a slower release schedule to "catch up", there are conflicts to resolve manually.
It's just not worth it. Figure out why they're asking, but say no. If it's a problem with buggy releases, that's the _actual_ problem to put some time into solving.
Remember that saying no is also an option. You aren’t obliged to say yes just because someone asks for something you don’t sell, even if they are willing to pay.
For most SaaS there’s no logical reason why having a feature for 2 extra weeks would be an enormous advantage to a customer. In my experience what this question really is is “can we make last minute change requests to a feature you thought was done”. I’m not keen to guarantee that, so it’s a simple no, sorry, we can’t do that. It’s yet to be a deal breaker.
If you frame it as early access, then people are apt to try to throw their wishlist at you, thinking it's still "not official yet" and easy to change.
But based on what OP asked, I'm not sure that's the primary reason. It seems more like his larger clients are finding the changes disruptive enough that they're wanting advanced notice to prepare for them. So integrating a lot of the suggestions here can very easily address that issue, while also providing leverage points for client management and sales.
@OP - don't think about it in terms of preview access or letting them into your internal sandbox environment(s). But rather, restructure your releases to decouple the release from the account migration. Using feature flags and such then allow end users to determine whether they immediately migrate to the new feature when it's available, or if they continue with the status quo and wait a few weeks to familiarize themselves with the changes before opting into them.
For end users – this gives them room to both continue with their day-to-day needs using the process they're already familiar with, while giving them flexibility to familiarize themselves with the changes and prepare (in whatever form they need) before committing to them.
For the tech team – there's an initial lift to adjust the app architecture to support this type of usage, but insulates them these sorts of client requests in the future since the app now supports these types of opt-in migrations within the live environment. If a user wants a sandbox account, it can be provisioned as a normal account in the live environment but with sandbox-like restrictions applied to it.
For the sales/account teams – this can be used as a negotiation point. You can position it as normal accounts get auto-migrated to new features upon release, but enterprise clients can pay for the ability to control their own release/migration timing via these opt-in feature gates. And a sandbox functionality (which provisions a new account using the current configuration of their existing account, but with applied sandbox-like limitations) allows them to both prepare for it and test it in a controlled environment before enabling it on their primary/production account. That sandbox capability is independent of the opt-in capability, and a second point of negotiation.
And since your application now supports both of these directly in production and doesn't require bespoke work or custom environments from your tech team to support, the incremental cost is pretty minimum to give it to a particular client. Which is hugely useful for a sales team when closing larger accounts. They can give it away without much hassle if they have to in order to close a deal or provide value-add during renewal, while simultaneously capturing incremental revenue from clients that are willing to pay substantially for these sorts of things.
I think some of the administrative pain can come from having a 'sandbox' or 'qa' tier beyond dev (which in this case would be customer facing) before production.
If data separation isn't required/essential, seems as though a a separate environment is actually unnecessary and feature gating could be used. But if data separation from other customers is required, then new infrastructure should be introduced via separate CICD if I follow correctly?
Pretty much. It's hard to guess at the reason why without knowing the product, but for whatever reason customers are wanting access to new features/releases before using them. Which likely means that they need to do something on their end every time you release (validation, update internal training/process documentation, etc). And it's causing disruption when things change outside of their control. So they're wanting more control over that change timing, to smooth out the impact of those releases as far as their usage goes.
If your release codebase can only support a single "production" version of a feature at once, you end up having to provision entirely new environments to hold a client back. And again, for every client wanting this. Because the timing they want isn't going to be convenient enough to be the same.
So instead, structure things so that you can run both the old and new version of "the thing" simultaneously in production, and give the client self-service ability to activate the new version when they're ready. That way your production environment itself is able to accommodate these requests and client's can opt-in when they're ready/comfortable to.
The biggest challenge to doing this tends to be architecting your application so you can push releases to the backend independently to releases in the frontend. Which can generally be handled by routing things through middleware that provides a stable response for a given feature version. If the interface is using the latest version, the middleware is basically just a stub that does nothing. But if the interface is using a prior version, then the middleware accepts the request signature of the prior version, translates into what's needed to execute on the new codebase, and returns a response that's consistent with the prior version. It's a few years old, but here's[1] an example of how we handled the data-model aspect of this sort of setup at a prior company
So in effect you're actually upgrading all customers to the new release at once, but from the customer perspective everything still looks/feels/act the same as they're used to until they "upgrade/opt-in" to the new release.
It's a pretty major lift to implement this architecture (particularly on an existing codebase), but maintaining it is easy to incorporate into the regular development process. And completely eliminates the operational and customer management headaches you're having now.
With this architecture, it doesn't really matter if you're running isolated deployments or a multi-tenant architecture. You only ever have a single version of production in both cases. The "sandbox" isn't a new environment, but a new production account that copies over the configuration/setup of their existing one while applying sandbox-like restrictions to it. This gives them a temporary space to prepare for the change so they're prepared when they flip the switch on their primary account.
Tried feature flags but it introduced complexity.
I wouldn’t charge more as others are suggesting- bugs you’ll address before general availability is worth the trouble.
If you keep something in preview for a long time then there can be a divergence from the main product. Make sure that data or work from users is migrated across when you finally release.
It makes sense to think about the opportunity for support to participate in the preview. Let them listen in to customer calls and learn about the problems. Providing a dedicated support person for previews is tempting, but long-term it's better to avoid this specialization and enable the whole support team to participate. This shares knowledge and avoids certain people being granted special privileges while others toil on average support tickets.
Microsoft publish some of their methodology at https://docs.microsoft.com/en-us/microsoft-365/admin/manage/.... The key point is that they have set out the boundaries for their program so that customers know what to expect and it can be repeated for every customer.
You will have to learn how to do your previews, but you should lay down some rules. Set it up so kudos from participants can be used for marketing after. You are the one to decide when the preview ends, but setting some minimum time in preview gives customers more confidence and time for you to process feedback. Make it clear if features are locked when you enter the preview and what feedback you are after.
Once the preview is over, thank the participants. Call them out if you have a public forum. Participation can become a mini case study for your marketing efforts.
Basically we had 2 production stacks with CI/CD for each. The only shared thing was the AWS ALB. Our customers each have their own subdomain so we used that to point to different EC2 ASGs.
We would continue pushing bug fixes to “prod”. Once a beta period ended, we merged to master and re-deployed master to both prod and beta channels. Rinse and repeat.
It works out pretty well. Occasionally there’s a really important bug fix that we have to cherry pick from prod to beta, but it’s rare enough to not be a huge concern.
Oh, and, charge them for it. I never got any pushback.
Was the beta channel that you had in the pipeline for prod? i.e. prod would have no chance of getting ahead of beta because prod was always deployed based on what was last on the beta release?
Hot or bug fixing is an interesting topic because you could end up with diverging histories if you're not careful. Did you try to maintain `master` as the single source of truth or did you make separate release and deploy branches for each environment?
Prod would be ahead of beta during hot fixes. Though we try and avoid it, it did happen occasionally.
You are correct that it took some extra effort to keep it all clean and manageable.
How would this work at scale? say if there were say 10 customers wanting approval, you'd have to have separate code release branches/versions for each customer? I imagine this could get error prone.
I suppose to your point on charging additional fees for this you can afford any sort of workflow.
That would be used for acceptance testing.
Bonus points for having up-to-date data in the staging DB.
I assume this is a multi-tenant implementation.
The other option is to charge the customer to disable unwanted features, in addition to enhanced features.
If it is multi—tenant, having the ability to ‘feature-gate’ could be worth the effort, if you can charge for enabling AND disabling features.
Ideally you have 3, and the third one is not used except to demo new features to customers AFTER they have been released (and perhaps, hidden behind a feature flag).
If you only have one sandbox, then it becomes a brawl between development, sales, and management.
More than three is overdoing it and assuming a technical burden for no benefit. (No, you don't need the capacity to spin up sandboxes willy-nilly. Three is enough.)
Alpha: Gets new features immediately and gets support within 24hours if something breaks.
Beta: Gets new features Every 45 week, and gets priority support if needed.
Stable: Gets new features quarterly, Mostly they are forced to upgrade from old features which you dont want to support, But they will be essential volume of your revenue.
alternatively, you can give them sandboxed demo read-only(resets every night at 12 o'clock) account for testing.
The schedule is totally in our hands, though. Once we say it’s time to go GA, it goes. At this point there still are feature toggles, though, just in case.
Every once in a while we merge new stuff in the customers branch so she has always the latest stuff in her preview.
1. Copies the production do 2. Removes sensitive data 3. Deploys the chosen build of that app (from Appveyor) pointing to that db
So in a nutshell CI/CD pipeline
If you are having trouble perhaps there are a lot of manual steps to your deployment.
I make the distinction so people don't just think they can put a single if statement in and be able to handle this sort of scenario.
* "Release train ahead" - code takes 2 weeks (or 4 weeks) to reach Production from merge, but it goes out to a Preview environment immediately or at the next release. So, you've forked your code into a Prod train, a Preview train, and mainline. You probably need the ability to hotfix to prod and preview fix to the week-ahead-of-prod preview environment, as well as roll normal development forward. You do gain "soak time" for code in the Preview environment, which is a slight advantage. And your QA is easier, because you have three discrete, testable things ("does it work in Prod, does it work in Preview, does it work in trunk"). But, you have another runtime environment which can diverge, which is a big disadvantage.
* Feature flagging by "release train," single execution environment. This is probably the most "Software Theorist Friendly" way to do this. You don't have to maintain/debug multiple environments beyond what you're already doing. Some customers are in the "preview ring" and get more features toggled on. Single runtime environment, single codeline. Fixes are just fixes. QA is a little harder depending on how you set up the possible "rings" and combinations, and the operational blast radius of your Preview code is now the same as your Production code (make a bad query that kills a backend service in Preview code, well, that's also Prod code. Oof.).
* Free-range feature flagging. Some complex arbiter system manages a huge range of feature toggles on a per-tenant/per-customer basis. Having this range of operational knobs is very fun and works great for customers who are particularly risk averse. It also lets you sell SKU-packs of flag combinations, roll specific features to specific "Early Access Programs," and so on. But, you get a combinatorial explosion of feature flag permutations, which depending on how coupled your code is and how intelligent your QA system is, can become a disaster: "Oh no, FooFeature only works with BarFeature on but our QA system always has BarFeature on so we didn't catch the regression! And only one customer has BarFeature disabled for legacy reasons!"
* Some combination of 1, 2, and 3 - for example a Preview environment that runs the same _code_ as Prod, but with the toggles set differently. A few feature flags which can be enabled for early access programs, but a general rule that code needs to live in a release train. This is my favorite approach. Having a separate Preview environment eliminates the blast radius concern of Preview-toggled code breaking Prod. Operational costs are higher, but these can be passed to the customer (making customers buy preview environments as a line-item is pretty popular). But by maintaining single codeline (rather than deploying a fork of last week's code to Prod), the code-theory is still easy, you don't have divergent release trains, and you don't have the combinatorial explosion that comes from a free-range feature flagging system.