Removing stuff is never obvious yet often better
gkogan.co
gkogan.co
> Visitors who didn't see the calculator were 16% more likely to sign up and 90% more likely to contact us than those who saw it. There was no increase in support tickets about pricing, which suggests users are overall less confused and happier.
Of course if you hide the fact that your product might cost a lot of money from your users, more of them will sign up. Whether they are better off depends on whether they end up getting a bill they are unhappy with later at some unspecified future date, or not. That's not something you will figure out from a short-term A/B test on the signup page. So this seems like totally useless evidence to me.
I see this dynamic frequently with A/B tests. For example, one of my coworkers implemented a change that removed information from search result snippets. They then ran an A/B test that showed that after removing the information, people clicked through to the search result page more often. Well, obviously, it makes sense that they might click through more often, if information they wanted which was previously in the snippet, now requires them to click through. The question of which is actually better seemed to have been totally forgotten.
How is that a function of the overly-simplified and often-wrong calculator?
If the user is never there to be happy/unhappy about it in the first place, then how would you test this anyway?
By closing the loop and increasing engagement, you are increasing the chance that you can make the customer happy and properly educated through future interactions.
The author was very careful with their words: they didn't say the calculator was wrong. They said it was confusing and sensitive to small adjustments. It's likely that the same confusion and variable sensitivity exists during usage. IMHO they should have bit the bullet and revised the pricing model.
Fair point. The author commented elsewhere here and stated that it's not the usage but the understanding of the variables in the calculator which are often wrong by more than 10x. From the response, it seems like the only way to know how much something will cost is to actually run a workload.
Edit: if the customer is getting a wrong-answer because of wrong-inputs, IMO, it's still a wrong-answer.
> IMHO they should have bit the bullet and revised the pricing model
I don't know enough to agree/disagree because they may be offering close to at-cost which might give better overall pricing than competitors. It's a complex-game :)
In this case, I do agree that the calculator is a bit daunting if you're not used to all the terms, but what should be done with it should have been an intuitive decision ("what can we do to simplify the calculator?") Not a fan of A/B testing culture that everything needs to be statistically analyzed and proved.
I'm not sure I'm following you here, so perhaps you'd care to elaborate?
The GP critique was that it was perhaps just creating a problem elsewhere later on. I'm not seeing the similarity to your case where the change is cosmetic not functional.
The issue of whitespace (padding) is subjective (see the conversation recently between the old and new windows control panels) but "scrolling down" does seem to be something that should potentially be avoided.
If sign-ups are increasing is that not the goal of the page? Is there reason to believe that the lack of padding is going to be a problem for those users?
My guess is yes, because consistency in design usually makes people assume better quality.
A/B tests, by definition, test between A and B. It is very likely that neither is the best option.
But how will you find the best option if you don't measure options against each other.
Firstly many businesses have never heard of A/B testing, much less apply rigorous application of it to proposed changes.
Secondly many businesses have found their niche and don't change anything. There's a reason "that's not how we do it here" is a cliche.
Thirdly a whole slew of businesses are greater changing things all the time. My supermarket can't seem yo help themselves iterating on product placement in the store.
Blaming testing in general, or A/B testing specifically for some companies being unwilling to change, or iterate, seems to be missing the actual problem.
Frankly, with regard yo web sites and software I'd prefer a little -less- change. I just get used to something and whoops, there's a "redesign" so I can learn it all again.
It has nothing to say about optimal solutions. Nothing about A/B testing suggests that you have reached an optimal point, or that you should lock in what you have as being the "best".
Now that the button position (or tighter layout) has been noted to have a material effect, more tests can be run to determine any more improvements.
A/B tests don't give you the best possible choice of all choices. They just give you the better choice between A & B.
The business shouldn't delay rolling something out that increases conversions *significantly* because a better design *might* exist.
You can A/B test "better" designs in the future until one converts better.
That doesn't sound like a signup problem; what was the goal behind the free plan? Drive more paying users? Raise company profile? Lock in more users?
Okay? The A/B test sought to measure which of two options A and B led to more signups.
> So, assuming that the increase in signups is driving more business is just foolhardy.
Your "A/B test enthusiast" was not testing for or trying to prove a causal relationship between increased signups and more business.
If he made the claim separately, then that is the context that is missing from now multiple comments.
I have no idea if that's a conscious or unconscious motivation for this business, but even if its not conscious it needs to be considered.
For me the principle is based on not exploiting the gap between the consumer and a better informed version of themselves. (“What would they want if they knew?”) There’s a principle of double effect: I don’t have to expend unlimited resources to educate them, but I shouldn’t take active steps to reduce information, and I shouldn’t leave them worse off compared to me not being in the market.
Essentially the failure is that we do treat users like this by relying on mass collection of data instead of personal stories. To be human is to communicate face to face with words and emotions. That's how you can get the nuanced conclusions. Data is important but it's far from the whole story.
I don't particular care whether their A/B test captures that potential aspect of customer (dis)satisfaction, but I am not sure how it would.
A silly amount of work but honestly lots of value. Experimentation optimising for short term goals (eg upgrade) is such a bad version of this, it’s just all that is possible with most datasets.
The problem with their calculator was that the users introduced slightly wrong data, or misunderstand what means some metric, and suddenly a 1000x the real price was shown. Their dilemma was "how to fix those cases", and the solution was "get rid of the messy calculator".
But they are not hidding a 1000x cost, they are avoiding losing users that get a wrong 1000x quote.
There's probably a bit of both, at the very least.
They started on vector search at a time when RAG in its current form wasn't a thing; there were just a few search products based on vector search (like a document embedding-based search engine for patents that I whipped into shape to get in front of customers) and if you were going to use vector search you'd need to develop your own indexing system in house or just do a primitive full scan (sounds silly but it's a best-case scenario for full scan and vector indexes do not work as well as 1-d indexes)
They blogged frequently and consistently about the problem they were working on with heart, which I found fascinating because I'd done a lot of reading about the problem in the mid ought's. Thus Pinecone had a lot of visibility for me, although I don't know if I am really their market. (No budget for a cloud system, full scan is fine for my 5M document collection right now, I'd probably try FAISS if it wasn't.)
Today their blog looks more than it used to which makes it a little harder for me to point out how awesome their blog was in the beginning but this post is definitely the kind of post that they made when they were starting out. I'm sure it has been a big help in finding employees, customers and other allies.
Probably our learning center is what you’re thinking of. https://www.pinecone.io/learn/ … The blog is more of a news ticker for product and company news.
But any attempt to address one source of confusion inevitably added another.I felt like that with elastic serverless’ pricing calculator which on the surface looks perhaps cheaper or more efficient than a normal managed cluster, because you think it would be like lambda. Except there are so many caveats and unintuitive hidden costs and you’ll likely pay more than you think.
They should have looked into this to see how to make it more obvious or more reflective of "regular use case"
Their sliders there are not too detailed. For example, what are namespaces, how many would a typical use need? Is 100 too much or too little? And if this is one of the variables that is too sensitive they would need to represent this in a different way
Companies can continue to monitor cohorts to compare retention to check the potential outcomes you highlighted.
> Would anything of value be lost if this or that chunk of it was removed?
In early stage projects I’ve seen this mentality backfire occasionally because it’s tough to estimate future value, especially for code and data.
For example, one time in a greenfield project I created the initial SQL schema which had some extra metadata columns, essentially storing tags for posts. The next week, a more senior engineer removed all those columns and associated code, citing the YAGNI principle (“you aren’t gonna need it”). He was technically correct, there was no requirement for it on our roadmap yet.
But the original work had taken me maybe an hour. And the cost of keeping the data around was approximately zero. It seemed he didn’t consider that.
Guess who needed the columns to build a feature a year later? Yeah me, so I found myself repeating the work, with the additional overhead of prod DB migrations etc now that the product had users.
I guess my point is, sometimes it’s also wise to consider the opposite:
Would anything of value be gained if this or that chunk of it was removed?
In this article the evidence is clearly affirmative, but in my case, well it wasn’t so clear cut.
It's all too easy to get caught either doing way too much defensive/speculative stuff, or way too little.
one person's "premature optimisation" is another person's "I've seen roughly this pattern before, I'll add this thing I wished I had last time" - and AFAICT there's no _reliably_ helpful way of distinguishing which is correct.
I presume this was not the only time such a migration needed to be done to add columns. Is it possible that as new features emerge, new columns and migrations will need to be done anyway and one more or less migration would make less of a difference on the grander scale?
i guess this is very illuminating - you have to predict the cost of adding YAGNI, before doing it. A low cost YAGNI feature might actually serendipidously become useful.
I feel this is the same principle as random side projects, not done out of necessity but out of curiosity and/or exploration.
We cannot perfectly predict the future, so when removing things there will always be false positives. The only way to get a false positive rate of zero is to never remove anything at all.
The real question is what level of false positives we find acceptable. If you can only cite this one case where it was painful, then I'd say that's evidence your colleagues approach worked very well!
(I think Elon Musk recommends a deletion false positive rate of 10 % as appropriate in general, but it will vary with industry.)
My point is - it's relatively easy to tell if something is used. Usually, a quick search will find a place where it's referenced. It's much harder to 100% confirm that some field is not used at all. You'll have to search through everything, column names can be constructed by concatenating strings, even if it's strongly typed language there could be reflection, there could be scripts manually run in special cases... It's a hard problem. So everyone leaves "unknown" fields.
The article is a little different, because "YES, they do need the calculator, people should know what your service costs up front". If they want to do away with the calculator, then they should restructure their pricing plans, so that it will no longer be required. That just not an IT problem.
If I recall correctly they have a measure at SpaceX that captures this idea: The ratio of features added back a second time to total removed features. If every removed feature was added back, a 100% 'feature recidivism' (if you grant some wordsmithing liberty), then obviously you're cutting features too often. 70% is too much, even 30%. But critically 0% feature recidivism is bad too because it means that you're not trying hard enough to remove unneeded features and you will accumulate bloat as a result. I guess you'd want this ratio to run higher early in a product's lifecycle and eventually asymptote down to a low non-zero percentage as the product matures.
From this perspective the exact set of features required to make the best product are an unknown in the present, so it's fine to take a stochastic approach to removing features to make sure you cut unneeded ones. And if you need to add them back that's fine, but that shouldn't cast doubt on the decision to remove it in the first place unless it's happening too often.
Alternatively you could spend 6 months in meetings agonizing over hypotheticals and endlessly quibbling over proxies for unarticulated priors instead of just trying both in the real world and seeing what happens...
When I want to know more about a number, I sometimes seek to test the assumption that an order of magnitude more or less (1.5%, 150%) is well outside the bounds of usefulness—trying to get a sense of what range the number exists within
The 5 Step Design Process emphasizes making requirements "less dumb," deleting unnecessary parts or processes, simplifying and optimizing design, accelerating cycle time, and automating only when necessary.
Musk suggests that if you're not adding requirements back at least 10%-15% of the time, you're not deleting enough initially. The percentage is an estimate initially based on experience, and now for several years based on estimates from manufacturing practice.
SpaceX probably has stricter processes than your average IT shop, then it isn't hard to calculate stats like that. Then when you have the number, you tune it until you are happy, and now that number is your target. They arrived at 15%. This process is no different than test coverage numbers etc, its just a useful tool not arrogance.
Taking that situation as read, the bare minimum I'd like from someone in a senior position is to:
a) invite the original committer to roll it back, providing the given context (there isn't a requirement, ain't gonna need it, nobody asked for it, whatever). At the bare minimum this might still create some tension, but nowhere near as much as having someone higher up the food chain taking a fairly simple task into their own hands.
b) question why the team isn't on the same page on the requirements such that this code got merged and presumably deployed.
You don't have to be a micromanager to have your finger on the pulse with your team and the surrounding organisational context. And as a senior being someone who passes down knowledge to the more junior people on the team, there are easy teaching moments there.
First of all, it wasn't guaranteed that this feature of yours would come. Even in this case, the feature came, you could probably add it with not too much effort, sure maybe a bit more than otherwise, but on a large project it's hard to guess the future. What if someone else would have taken that task, maybe they wouldn't even recognize why those columns are there and they could have just reimplemented it anyway.
Also, a year is a long time, and who knows how many times it would have caused additional work and confusion.
> A: hey why this thing here, it doesn't do anything / never actually used?
> B: I dunno, Alex added it because 'one day we might need it' (eyeroll), will get very touchy about if you try to remove it, and will explain how he/she can predict the future and we will definitely need this feature.
> A: and this thing over there?
> B: Same... just move on, please, I can't keep re-discussing these things every month...
And, if your team wouldn't have applied YAGNI, you would have 1 feature that was a year later needed, and probably around 20 that was never needed, yet caused maintenance burden for years down the road.
I dunno, YAGNI is one of the most valuable principles in software development, in my opinion.
In the case you describe, there are three possible outcomes, in broad categories:
1) The fields do turn out to be useful, in exactly the way you implemented them first.
2) The feature is implemented, but using a different set of fields or implementation.
3) The feature is not implemented.
Even if we assign equal probablility to all the options, creating them in the beginning still only results in a win in 1/3 of the time.
How much extra mental effort would have been spent making sure that all the other features implemented in the mean time work correctly with the metadata columns if they had not been removed?
Of course, you turned out to be correct in this case and that shows you certainly had excellent insight and understanding of the project, but understanding whether a decision was right or wrong, should be done based on information available at the time, not with full hindsight.
Don't get me wrong, I understand your point here — and if the feature is something that you know for sure will be needed later on, adding it in from the start instead of bolting it on later is absolutely the smart choice. You wouldn't pour a concrete foundation for a building after you built two floors — especially if you know from the start it is going to be a skyscraper you are building.
In fact my experience with software development is that good foundations make everything easier. The question is just whether the thing we're talking about is truly part of the fundament or rather some ornament that can easily be added in later.
Result: disgruntled senior employees quitted and were replaced with juniors. 3 month later, overtime was distributed across the tech department, killing the laid back 4-days week culture that was put in place to attract talents, the employee unionized, some more quitted, and about a year later, upon failing to hire back these productive elements, the COO was fired.
You're missing the time other developers will spend trying to figure out why that code is there. When investigating a bug, upgrading, or refactoring, people will encounter that code and need to spend time and mental energy figuring it out.
Recently, I've been modernizing a few projects to run in containers. This work involves reading a lot of code, refactoring, and fixing bugs. Dead code—either code that is no longer called or code that never was—is one of the things that most wastes my time and energy. Figuring out why it's there and wondering if changing or deleting it will affect anything is just tiresome.
Answering "Why" is usually the hardest question. It becomes especially challenging when the original developer is no longer with the team.
An interesting and pretty classic dynamic - I liked the article overall but I think this point didn't get the highlighting it deserved. If 30% of the people involved think that the calculator is a bad idea that signals a potentially huge problem even if the majority think it is fine.
Be alert to the politics here. Although it seems otherwise, people generally don't like to criticise other teams in the business unless there is a political gain to be had. By extension, if I polled the company and asked "is this thing my team did a net positive?" I'd expect the default position to be "yes" as people wouldn't want to stir the pot for no reason. 30% of people signalling that it might be value-destructive is much more significant than it seems because of that. It should trigger some fairly thoughtful consideration of why exactly they thought that.
In this case they seem to have indeed been alert to all that and the story has a happy ending, which is nice. But this poll result was always evidence of a serious problem.
Even better if you have mixed incentives: sales wants any and all dark patterns enabled, customer support is sick of issuing refunds because the cart auto adds extended warranty to the purchase, etc
I smiled in the article when they claimed that removing the calculator might be better for users because more sales are completed. Ignoring that maybe the users were getting the appropriate amount of sticker shock, and bailing was the correct choice.
Nobody is saying a feature should be automatically removed when it has 30 % detractors, just that it is a useful signal to investigate further.
The specific % threshold doesn't matter. Pick one that makes you chase false positives rarely enough. The exact number will vary from organisation to organisation.
Imagine the calculator code was a mess compared to the rest of the project, it uses outdated libraries and updating would break it, it may have some security vulnerabilities, uses an unreasonable amount of resources and breaks the build system. No one wants to work with it. Ask whether it is a good idea and most people will say "no", hoping to get rid of that mess. In that context 70% is a very good number. If on the other hand, it is a feature people love working on, then 70% would be very bad indeed.
This specific bit kinda gives pause though:
> One slight misinterpretation and wrong input and you'd get an estimate that's overstated by as much as 1,000x.
Does it also mean that in real world usage, one slight misinterpretation or misevaluation of your metrics and you're liable to 1000x more than you planned to ?
I totally see this as a reality of online billing systems. I've misconfigured GCP prototypes and ended with 100+ bills where I though it would be 2 or 3 at most and didn't care to watch for a few days.
But I'd understand a client bailing out when they realize slight changes to the sliders result in wild increases in the planned pricing. And removing the tool would sure help for registration, but not help the customer if they hit these kind of issues down the line.
> Does it also mean that in real world usage, one slight misinterpretation or misevaluation of your metrics and you're liable to 1000x more than you planned to?
Unlikely. You can see why in these two examples that really happened:
One user I spoke with said they assumed "queries per second" is calculated by (number of searches) x (top-k for each search), where "top-k" is the number of results they want back. I don't remember their top-k but let's say it's 10 -- so they were entering a value for "queries per second" that was 10x higher than it should be and they'd see an estimate around 10x higher than they'd really be charged.
Another user thought you get "number of vectors" by multiplying the number of embeddings by the embedding dimensionality (1,536 is a common one). So they were entering a value literally 1,536x higher than they should've. Their actual usage would be calculated (by Pinecone) correctly and not be that high.
Vector dimensionality is a basic concept for AI engineers and QPS is a basic metric for DB admins, but Pinecone sees lots of users who are either new to AI or new to managing DBs or both.
> where "top-k" is the number of results they want back.
Some systems will do that, so I get the confusion. I think the YouTube API for instance has a quota system that takes internal operations into account, so getting back 10 results in a query effectively weights 10+ credits.
I better understand the kind of issues you are facing, as these kind of subtilities are inherently hard to explain.
For better or worse, that's another advantage of starting with a small trial account and actually see how the operations are billed for typical operations.
I counted five obvious ways to subscribe on that page alone. Five! Do you really need to shove them in our faces and in the middle of the freaking content? Do you think you get more subscribers by interrupting people and being annoying? Are those the kind of subscribers you want?
Removing stuff is often obvious, you just have to take your mind out of the “more more more, make money, attract customers, more more” gutter mindset and think “what is respectful to users, how can I help them while simultaneously treating them like human beings instead of wallets to be milked”.
What’s respectful to users is a separate (but not wholly unrelated) question…
The goal is to get paying customers. We discerning readers are the high effort, low reward cohort that aren't worth losing sleep over.
Ignoring for now that’s 100% anecdotal and that “a few people” is far from enough to make definitive claims, what post history are you referring to? Do you have a link?
> The goal is to get paying customers. We discerning readers are the high effort, low reward cohort that aren't worth losing sleep over.
I understand that. I’m lamenting we live in a world where we find it acceptable to purposefully produce shit to exploit others.
Call it anecdotal evidence if you will. The matter of fact is that it seems to work well enough for people to keep doing it.
You call it asking, I call it nagging. And “yes” isn’t the only answer, there’s also “you annoyed me so much I’ll actively avoid you”. Have you never seen one-star reviews saying “the app keeps nagging me to review”? These have business consequences, it’s far from all positives as your responses imply.
> The matter of fact is that it seems to work well enough for people to keep doing it.
That’s like the old maxim that “nobody gets fired for buying IBM”. Just because “everybody does it” does not mean it’s the optimal approach. Things change and people get wise to common bullshit, even as this kind of “knowledge” and “best practices” keeps being shared by money-hungry pariahs. No one really tests these assumptions in depth, they just share them uncritically. If you’re so sure it’s the best approach, let’s see the data. Otherwise let’s just be honest and say we don’t know.
Do you have the data that you can share?
> It's good to remember that most businesses exist to make money, not to be pleasant to us HN readers.
HN does not have a monopoly on discerning users. We are not special. It would be unrealistic to think the number of people who care about this is but a subset of people who frequent one website.
>> It's good to remember that most businesses exist to make money, not to be pleasant to us HN readers.
> HN does not have a monopoly on discerning users
Businesses can and will take advantage of less discerning users and they are in their right to do so because making money is their reason to exist. That's a terrible mindset that dominates the (big)tech/startup sector and the reason for the Great Enshittification. Let's see how far this can go before it collapses.
This is a symptom of over hiring. Too many people removes agency.
When people lose sight of what's actually important and feel that they must reach consensus by committee then there are too many people.
Perhaps. It's also a symptom of bike shedding which can happen with as few as two people.
First: A lot of time was spent building consensus.
No individual felt they had unilateral power to remove the calculator. Instead the behavior was to seek approval. That's probably because of unclear ownership, which often happens because of too many people.
Second: Too many cooks in the kitchen.
At any stage of a company there's a limited amount of important work. Work that is mission critical and provides outsized value to the company.
When people feel their own work is not important they seek other work that appears to be important. So, you get the behavior of a lot of opinions on a pricing calculator.
Even design by committee is limited to the committee.
Also, I'm a big fan of "contact us for pricing." It's annoying for users who are window-shopping and want a quickie ballpark, but it helps you identify cases where standard pricing (or pricing which can't easily be described online) can be negotiated, and which the user would have otherwise overlooked. This doesn't work for things like most ecommerce, of course.
I have opposite feeling about them. They are like open invitation to give the sales guy a window of opportunity to look up your company website, and markup the price accordingly.
My biggest issue with these: when introducing a new tool/solution, we often don't know how much we want to use it. In particular, it will usually be introduced in a minor application first, and if it feels reliable and useful it moves to bigger systems and more critical roles.
Contact for pricing requires us to explain all our internal politics, budget management, which systems we have etc. upfront to random strangers, who are also incentivized to just onboard us first and push the sunk cost fallacy button from there.
I kinda feel this works best for companies that don't really care about comparing products and will buy whatever give them the best numbers on the contract (which is common for enterprise software, it's just not my personal cup of tea as a small fish in the game)
I make most of the buying decisions for tech tools for my company. And it is exceptionally rare for me to ever contact somebody for pricing. I usually move on to the next vendor with transparent pricing.
You can get away with it, if you are targeting a very small market with your product and none of your competitors offer transparent pricing. My own company does not offer transparent pricing and we can get away with it for the above reasons.
> The article states that the biggest factor was user misunderstanding of the options, not so much the number of different options.
(Emphasis mine)
It seems to me that you are in agreement with the GP :-/
When a significant portion of the target userbase cannot understand something presented by the software, the problem is rarely the users.
More developers should read Donald E. Norman's "The Design of Everyday Things"; even if you forget specifics in that book, the general takeaway is that the default position must be "It is not the users' fault!".
There must be significant evidence that the user is to blame before the user actually is blamed. The more users' that have that problem, the larger the body of evidence required to prove that this is a user problem.
More than 10% of target users have problem $FOO? Then you better have a mountain of rock-solid, beyond a shadow of a doubt evidence that the software/interface is correct!
Don't underestimate those kinds of customers.
For example, an ad for custom tile showers showed up in my feed. I just wanted to "window shop" the price, so I could get an idea if it was something I wanted to do, and plan when to do it.
I filled in the form with a "I'm just looking for a ballpark number, please don't call me."
No response.
Salespeople just don't understand how irritating phone calls are when you're collecting data: Whatever I'm doing at any given moment significantly more important than dropping what I'm doing to answer the phone. This is especially important if all I need to know is a ballpark number to know if I'm interested in having such a phone call.
Dev teams of those five are stretched thin, barely able to keep up with bug fixes, and struggling to add new, important features -- so much so that getting anything on the roadmap is an impossible battle, no matter who is asking.
There are thousands of devs at the company, most of them attached to the products that contribute relatively nothing to the revenue stream.
I don't know if it's not obvious -- it must be obvious -- that to move forward the right thing is to cut most of the products and refactor the teams to drive the remaining money-makers. Yet, this has not happened, and I see no indications, no murmurs of it happening. Corp politics are wild.
So we made a product advisor applet where the user would answer a few questions and it would suggest one or two products that would be best for them. Getting the applet right took a bit of work, but once it was done it worked very well.
We put it live on the site and.... conversions dropped precipitously. We A/B tested it and yep, it definitely hurt conversions. I still don't know why it hurt conversions, but it did. So we moved it from the homepage to a FAQ section of the site, and hardly anyone ever used it at all.
> I still don't know why it hurt conversions, but it did.
Maybe people were indecisive and you just saved them trouble of trying stuff to find out for themselves.
If it was a service, then maybe they were signing up just to try because they weren't sure, and then taking advantage of sunk cost fallacy by pulling out an Amazon Prime strategy. Or, maybe without the advisor applet they might sign up thinking they can get away with the cheapest version (e.g. like people who purchase 3-5 EUR/month VPS), but with the applet you deny those hopes right away.
If it was physical products, then you might have helped them out of a bad purchase.
> So we moved it from the homepage to a FAQ section of the site, and hardly anyone ever used it at all.
Yeah I wouldn't expect to find it there. In the footer, maybe. But not in the F.A.Q., so there's that.
I'm working on a tool that makes running wire inside residential homes easier. It requires carving a channel on the back side of the baseboard.
My original idea required a mechanical tool with a motor [2]. We prototyped a working version of this but always felt that the manufacturing challenge would be large.
We ended up adapting the system to existing routers. That meant our product was just a series of extruded parts with almost no moving parts [1].
I can say "I wish someone made a cheaper alternative to the Domino" and anyone in the space will understand what you mean instantly. But from an original analysis others might of told Festool that it was a bad name that will confuse people.
In other words it’s not much of a burden and they get much more reliable information.
Things we like, things we don't like, things that are totally useful, things that are somewhat useful etc.
There comes a point where we need to let go of the thing(s) we like - I called them the precious payload where we are most reluctant - and in this case the 'calculator' was the precious payload so many people in the company were unwilling to remove this feature except for one person.
In business, adding a completely new feature or subtracting an age-old feature is extremely difficult but oftentimes, this is where growth comes from.
I suspect that this often exists because people have tried this, and been summarily burnt by either politics, or some form of Chestertons-Fence.
Which leads to the question of: how and when do we discern the difference? How do we design our teams and engineering to do things like,
- ensuring the psychological or political safety to suggest slimming down systems without having every other team come down on your head.
- incentivise “svelte” systems at a product level, without falling into Google-levels-of-“lmao this is going away now thanks bye”
- engineer for slimmer systems. There’s lots of ink spilled on the topic of making things extensible, or able to have stuff added to it, but seemingly little about the opposite. Is it the same engineering practices, or are there other concerns you’d need to take into account, if so, what are they?
- would you as a customer pay for something that was better at its purpose but didn’t have every single feature under the sun? I sure would, how many other people/orgs would, if given the option? I semi-controversially think that too many tools providing too much flexibility mostly encourages orgs to paint themselves into wacky processes, just because they can. I doubt this entirely goes away, but if given less feature/flexibility bloat, how much “fat” (process/overhead/friction/etc) could be trimmed out?
Pinecone is just generally thought as something extremely overpriced:
https://www.timescale.com/blog/pgvector-vs-pinecone/
Until they can solve the feeling (or cut down on their own overhead), I don't see that great future for the company.
No pricing calculator or benchmark post (from a competitor, no less) will replace just trying the service yourself.
But when we handle them, we give ourselves a pat on the back, without thinking can I just ignore the edge case and shrug if someone runs into it?
In the case of the OP, the edge case is the occasional user who might want to project their exact costs.
1)Add an adapter that’s customized on both ends, or 2) subtract the differences so they mesh directly.
Always look for opportunities to subtract, rather than add. Your system gets easier to understand over time as it becomes the purest representation of itself, instead of a collection of gap-fillers.
i use this in day to day to life: making plans, buying things, managing others - reducing has led to more accomplished.
I believe shrinking the feedback loops is the most effective form of envelope-pushing.
Simply removing a feature without addressing the root cause of confusion will lead to other problems down the line, such as increased customer support demands or user dissatisfaction when actual costs differ from expectations.
And, yes, removing things should be rewarded. It's a cognitive bias to reward just for adding things.
I feel like it should have a lower priority on such business decisions. My guess is that calculator removal was approved before A/B testing have started.
I love his “algorithm” for getting shit done. The final step is, “If you don’t put back at least 10% of what you take out, you aren’t removing enough.”
People hate the proposal to remove the microwave ovens and free coffee from the kitchen, so it must be the right idea!
It would seem a common thread is our inclination to embed part of our ego into everything we do.
For anyone who liked the trick in the post consider checking out TRIZ: https://en.m.wikipedia.org/wiki/TRIZ
There are too many interesting ideas in this framework, but one of the first steps in the algorithm of solving a problem is to "Formulate ideal final result", according to TRIZ.
Now the "Ideal final result" has a specific definition: The part of the system doesn't exist but the function is still performed.
I'm having a lot of fun with this and other tools coming from TRIZ when solving problems every day. You might like it as well!
As for A/B testing and getting unexpected results: inside TRIZ there is explanation why it works - it is called "Psychological innertion". i.e. when engineer is getting a problem it is usually already formulated in a certain way and engineer has all kinds of assumptions before he even starts solving a problem. This leads to him thinking along specific "rails" not getting out of box. Once you have algorithm like TRIZ, it allows to break through psychological innertion and look at the problem with clear eyes.
Some other trick one might use to find interesting solutions to the problem from the post: "Make problem more difficult". I.e. instead of how to make calculator simple and unrestandable, formulate it in a different way: "how to make calculator simple and unrestandable, visual, fun to use and interact with, wanting to share with your collegues?"
"Turn harm into benefit". calculator in the post is treated as a necessary evil. Flip it. Now we have a calculator, but we could show some extra info next to prices, which our competitors can't do. We can compare with competitors and show that our prices are better, calculator can serve as a demo of how customer is always in control of their spending as the same interface is available after they become customer to control their spend etc.
Formulating this way already gave me some ideas what could be added to calculator to make it work.
Hope it helps someone.
Reminds me of the quote: "It is difficult to get a man to understand something when his salary depends on his not understanding it."
That applies to us as software engineers too, our salary depends on having projects to write code for so it's not so surprising that a very smart group of people don't often consider that doing nothing or removing something is a valid solution. I like the author's observation on this too: it would be nice if removing things were rewarded. I wonder of the employee who questioned the calculator's necessity got any reward.
Which may/not matter, but it requires more politics and negotiation to suddenly be dropped.
“Perfection is achieved not when there is nothing left to add, but when there is nothing left to take away”
“Less is more”
“The value of a creative product doesn’t lie in how much is there, but in how much has been discarded.”
rttm: Reduce to the max
"Kill your darlings"
...
"Any fool can make something complicated. It takes a genius to make it simple."
"I apologize for such a long letter - I didn't have time to write a short one."
--
As for the calculator, I think it points to a bigger problem. Customers need to know what the platform will charge and a way to compare platforms in general. If the only way to truly know how much something will cost is to run their code on it, then maybe that's the thing that someone needs to implement.
There are big issues with this in the most-naive implementation in that people can easily abuse the ability to run code. That suggests that perhaps we need a benchmark-only environment where benchmarks themselves are the only thing allowed out of the environment. This may require a fair-amount of engineering/standards effort but could be a game-changer in the space.
A framework for being able to run this on many platforms to compare performance and pricing would lead to customers generating packages for vendors to compete. Though, I suppose it could also hide some devilish details like step-changes in rates.
This same framework would be useful for other things too, like testing how implementation changes affect future bills. Also, how pricing changes between vendors might become more-advantageous over time.
Of course, the sales folks might balk because they would rather have a conversation with everyone they can. Maybe I'm just advocating for a more advanced and complex calculator? ¯\_(ツ)_/¯
How dare they think that a thing called calculator is accurate and not double check!
Up-front pricing is only good for the buyer and never good for the seller. And while being an advocate for the user is a good thing in general, it's not a good thing if the result is less revenue. And if you want to present a pricing estimate your incentive is always to lowball and underestimate.
Businesses that are successful tend to exploit information asymmetry. This is frustrating as a consumer but if you can guarantee that the user doesn't know their final cost you can always sell them a sweeter picture than reality and play off the cost of switching to a competitor, if one exists.
Total aside but this is why housing prices are insane, at the most micro level no buyer has any idea over the final cost until the last possible moment at which point the cost of undoing the transaction probably outweighs the risk of continuing - psychologically, at least.
(I may or may not have been the victim of this recently and thought about it from the seller's side)