Fission

I think there are a couple of great points here, and I'll try to break things down and address each point independently.

1. re: "real" user reviews — right now, we do do this, just in a different way vs. what I think you're thinking of. For all of our guides so far, we've conducted a lot of interviews with founders who've used each product. However, we ultimately synthesize this data, pair it up with our own findings from our testing, and draw our own conclusions.

We think this is close to the optimal way of doing things. We found that sites which rely 100% on user reviews usually hit a ceiling in terms of usefulness, as the reviews are typically on the shallower side, which compounds b/c high-level reviews typically end up talking about the same things, so they're often redundant as well. By getting founder feedback, we're able to get a lot of datapoints to guide our findings and to surface less common issues, but by testing and ultimately writing these guides ourselves, we're able to both spend the time to elaborate on important aspects in depth (e.g. caveats, background info, etc.), while also being able to present info from a bigger picture perspective (e.g. this is how X, Y, and Z compare, etc.). The main downside is that it requires a lot of additional work.

2. re: Gartner + incentives — I believe that you identified the problem correctly, but misattributed the underlying cause. Gartner's incentives are misaligned b/c vendors who pay have an edge up on those who don't, for reasons you correctly identified. Importantly, this is not isolated to Gartner — in fact, the incentives are just as misaligned for a lot of the 100% UGC review sites. For those sites, if a vendor pays, they're allowed to cherrypick their own users to invite for an "organic review." If the vendor doesn't pay, then their rating will be relatively lower b/c it becomes much harder for them to get their best users to write a review on the site.

So in general, the problem isn't caused by whether you're a UGC review site or write guides yourself — it's caused by, well, whether or not you're doing something slimy to make money.

Edit: Oh, and I just saw that you mentioned Capiche at the end. I just want to say that none of this comment is directed at them — I sincerely think that they're passionate, motivated founders who deeply care about doing good for the startup community.

Hey, thanks for the datapoint! I'd like to break your comment down into what I believe are the two main points, and address them independently.

1. On whether or not FRB should be in the list — we certainly think they should. Based on the founder surveys we sent out, we found that FRB was quite-well represented across the startup community (and I think that your datapoint adds to that finding). Our inclusion of a company/product is not an endorsement, but rather a result of what we think would be useful to test for the community. In this case, we evaluated FRB because we know a lot of founders (including yourself) have considered/are using them.

2. On whether our evaluation reflects your experience — this is the more interesting point for us. We have data from ~15 founders re: FRB, ranging from full-blown interviews to anecdotes. In general, we didn't hear anything substantially negative about their core banking service, but we aren't going to conclude that they're perfect from that data alone. That said, the founder data was informative enough to draw some directionally-accurate conclusions from, although additional data points help us both realign as well as shrink or expand our confidence interval.

If you're willing to email me about your experience with FRB, we'd very much like to look it over and take it into account when we update the guide.

It definitely is a bit confusing, I'd agree with you there — I'll try to explain it:

Put shortly, this is the exact same company that YC invested in back in S18. The product may be different, but the underlying company (in the legal sense) is the same.

When a startup pivots, they usually have two options: 1. shut down the company, return money to investors, and start afresh; or 2. continue with the same company, just with a different product. You'll hear varying opinions on the "right" way to approach it, but in general, YC startups usually take the latter option as YC is generally of the mindset of investing in the founders rather than the idea.

This is why Retool, Brex, Segment, GOAT, etc. are considered YC companies, despite completely changing what they were working on (the latter two substantially after demo day).

Thanks! I shared some thoughts on this earlier (https://news.ycombinator.com/item?id=23237147), but I also wanted to take some time to discuss the points you made.

Completely agree that The Wirecutter's up-front-ness and monetization is well-aligned with the end customer. Wirecutter's affiliate model works well with consumer goods b/c affiliate rates are mostly standardized (mainly by Amazon), which mostly mitigates conflicts of interest. This doesn't work as well in SaaS because vendors are so fragmented. This means that there would need to be a lot of checks and balances to avoid bias in B2B software. That's why any eventual monetization likely won't look like Wirecutter's.

Once we settle on a way to make money, we'll make it transparent. I'm just being vague right now because we don't know what exactly that is, so we're being... transparently vague :)

re: fonts — I booted up my windows machine, and you're right, it renders abnormally narrow and is hard to see. No CSS rendering options could fix it, so I think it just comes down to how Windows handles the font. We'll try to swap it out for something more legible, but big thanks to you for bringing this up either way.

re: screenshots — I think I see the point you're trying to make. We'll get something up and running and see how it works.

re: tooltips — just pushed something to add them

This is actually a really interesting comment, because it reminds me of a meta-observation that I've made over the past few weeks.

First off though, I want to clarify that the money market fund is typically not operated by the org. The GMMF choices for SVB are managed by BlackRock, Morgan Stanley, etc., for Brex it is BNY Mellon, etc. You really only get self-operated funds for larger financial institutions, e.g. Wells Fargo or Fidelity. Therefore, many of these products act as a software layer over some existing bank/fund/etc. I believe we explain this distinction in the article. There are some caveats, but this is the general gist.

But onto the meta-observation: given the current economic climate w/ depressed treasury rates, we've recently received a lot of questions about higher-yield options, specifically re: CDs which are currently hovering around 1.5%. These are from well-known, well-respected startups who presumably don't need the cash and don't need to deal with the high overhead and illiquidity of CDs. I think that this really demonstrates the survival instincts of later-stage founders being carried forward from their early days, and the fact that startups are functionally live optimization machines, not just at a product-level, but as an entire organization. It seems to us that the best startups eke out every advantage they can — with all respect to the nature of the power law, these small differences, in aggregate, might make or break the entire company.

Thanks. Focused on info for the earliest stages of a startup for now, but we're continuously progressing towards later-stage topics.

Pushed! Thank you for the suggestion — completely forgot about some meta tags.

Done, thanks again for the suggestion. It'll break the links in the main post temporarily, but hopefully dang can rescue it.

Edit: decided to throw in some redirects instead, should work well.

Appreciate the suggestions. To be honest, we spent a decent amount of time on table formatting, but we'll take another look. We think horizontal headings are out of the question though, b/c we don't see a way to preserve the table format going that route.

Thank you, appreciate the feedback! Would you be willing to go into some more detail into how these decisions played out for you in the past?

Thank you for the comment — feedback like yours helps us continuously find ways to improve!

I'm really interested in digging in a bit deeper, as you've expressed some thoughts we haven't heard before, and I'm hoping to improve the site more.

> I'm not getting any younger, and I find that the non-white background combined with the light weight of the font hard to read

This is interesting. The text color is #000, and the background color is #fafafa. This is a higher contrast ratio than the wirecutter, hn, etc. Since the purple background is easier to read on, it might be that the text is actually too high-contrast. It probably doesn't help that our font is a bit on the narrow side. I have a couple of ideas on how to fix it, but don't have a great idea on how to test it.

> I think adding screen grabs would help break up the monotony of long streams of text, and give people an idea of what the UI looks like for the product(s). One of the first things I look for on any SAAS landing page are screenshots, so I can see what the UI looks like. Adding screen grabs in your reviews would save some time for me.

This is particularly interesting for me to hear. Would you be willing to elaborate a bit more on how exactly you use screengrabs in your decisions? In my personal experience (emphasis on personal), screengrabs don't provide much value because they're mainly used as feature demonstrations.

We try to take an approach where we don't focus on the features, but rather the overall impact in aggregate (e.g. we won't focus on who has the best funnel analysis, but will ask who has the best core analytics functionality). We strongly believe that this synthesis approach is more useful, but it also means that features overviews are lower priority.

However, I'd be curious to know how you use them, as it sounds like you get value from them beyond just feature understanding. With a better understanding of how you use them, we can go back and get screenshots that best fit the kind of value that you get from them.

Appreciate the kind comment! To be honest, I didn't really think about that paragraph too much — I just wrote down my own qualms/experiences in a list, and eventually decided to consolidate it into a paragraph. It was just something that I felt strongly about.

> I had a quick look at your site. I wonder if targeting a smaller range of reading duration could help.

This is something that we tried to deal with through the overall summaries, the rating matrices, product summaries, etc. I completely agree that it'd be overkill for everyone to read a super long guide, which is why we added those components. However, we also wanted to ensure that if a reader wanted to go in-depth into the reasoning/background/justification behind something, we would have the information right there.

Granted, it's not perfect yet (I've spent a lot of time responding to emailed questions with "actually, we discuss that in this paragraph/section/sidenote/etc.", so there's clearly room for improvement on our end), but we're working on it! Any suggestions on this front would be super helpful, actually.

Hey thanks, appreciate the kind words! It's definitely something that we thought about (and are continuing to think through). There is also the question of service vs. product businesses, although there are certainly a lot of comparables for the former.

For now, however, we're mainly focused on making ourselves useful for early-stage companies. Certainly something we're thinking about however!

Didn't publish anything. Mainly, we focused on making 3D reconstruction practical for B2B use cases.

It's interesting, because I personally see the idea of commercializing academic research to be well-intentioned, but a bit more difficult than it sounds.

Some more thoughts from a previous HN thread: https://news.ycombinator.com/item?id=20753553

Totally! It's something we've considered, and it wouldn't be too hard to implement. The reason why we've deferred it is because we've found that most early-stage startups are pretty homogeneous in their requirements, so a self-weighing system like you suggested is a little excessive. That said, I think it's very much something that we'll revisit when we start delving into later-stage-startup topics.

Thanks for the suggestion!

Thanks for bringing this up. Just to clarify, the base version of GA provides an event-based model. GA App+Web is a feature of GA that's aimed at sharing/consolidating info between websites and mobile devices. We're focused on the web side of things, as we've insofar only written a web analytics article. We suspect GA becomes a bigger player when we take a look at mobile analytics, but we haven't fully delved into that yet.

We'll take a look.

Appreciate the feedback. Our goal is really only to be useful, which naturally ends up with us covering a lot of background (e.g. for store of money: what is a cash sweep, how is yield determined, what are the underlying incentives for various startups, etc.). Plus, by holding ourselves to a standardized set of criteria, it's a lot easier to stay objective.

I'll take a look to see if we can't address formatting in the mid-range of window sizes.

Appreciate the suggestions. You're right about dashes, will look to fix. Lack of metatags comes down to the lack of any social media accounts.

Actually, I totally agree with you. This is something that we thought hard about, because our initial three guides are dominated by YC companies.

I think the main explanation is that YC companies, to a much greater extent than other companies, tend to focus on early-stage startups. As our guides are currently focused on early-stage startups, we naturally see more YC companies represented in the guides.

If anything, I'd argue that we're so aware of this that we need to consciously avoid overcorrecting in the other direction.

Insofar we haven't outed ourselves as doing reviews, so we mainly used our own accounts or some burner accounts (we've got a few email addresses that will remain private). This allows us to get the actual customer experience.

We actually have a sidenote that mentions that GA can be used for event-based tracking, but we've decided to skip over it in v1 of the guide because it's missing some critical analytics functionality. We're focused on the big three event-based analytics tools for now, but are going to add more info on tools that can be used for event-based analytics, but aren't primarily designed for that purpose, e.g. Pendo, GA, etc.

This is going to be a bit of a roundabout answer, so please bear with me.

Our first task is to prove we there we can provide sufficient value through objective info, and in helping people figure out which tools to invest in.

This is a bit of a convoluted way to say that we aren't 100% sure on long-term monetization, but we do know it probably won't be wirecutter-type affiliate nor paywall. We fully subscribe to the pg thought process behind this: http://paulgraham.com/good.html. That said, if we can accomplish our goal, we believe we can make money somehow, whether by offering value-added services or something else. Keep in mind that we're software developers ourselves, so we're in a position to build something once it becomes clear where the opportunity is; we're not just review-writers. For now, we're focused on producing as much value as we can by writing useful reviews. Speaking of which, we'd really like to hear what would be most useful to you in evaluating SaaS products!

Will do. It might be a bit lengthy and take a bit of time to write, so I'll post this placeholder for now and update it throughout the day.

> I can't count the number of times I've ended up on one of those ranking sites that feels like an affiliate scam.

Totally agree. We took a close look to figure out what made some review sites slimy (and frankly not very useful). It mostly came down to what you'd expect: all reviews are normalized to be generally positive, lots and lots of recommended products, etc. That's why we made a deliberate decision to not go down that route.

> One suggestion: for your "ratings matrix" it would be easier to parse if you kept the "our take" section out of it so all of the ranking icons appear close to each other.

Appreciate the suggestion, I see what you're getting at. Most design issues come back to me, so I'll take a look at it and see if we can't make it easier to compare all ratings at once.

Yeah, startups are in the position of having to make high-velocity yet high-quality decisions, yet without too many resources to accomplish either. A bunch of our own decisions in the early-days mostly came down to asking some founder friends, throwing our hands up, and mentally flipping a coin.

Oh, and I'd like to add — one of the reasons why I'm so excited about this launch is because it's been a crazy journey so far. I entered YC as a solo founder fresh out of college, working on a near-polar-opposite product (mobile 3D reconstruction; highly-technical). We worked hard on that for nearly two years, and only recently made the decision to change directions. If it's interesting, I'd be happy to give perspective on this decision and my experiences so far.

100%. You won't even need any extra hardware to accomplish that.

However, I would encourage anyone going this route to think closely about the incentive structures behind the product they're building, and validate them with real estate agents, photographers, and sellers. It's a lot more complicated than what many of us may give it credit for.

I think there was a previous discussion on this article a couple of days ago at https://news.ycombinator.com/item?id=22707365

The link there is non-paywalled.