Fast or Slow | Hacker News Reader

Fast or Slow

fastorslow.com

359 pointsuptown121 comments

Hi all. We're really flattered to be on the home page of hacker news. Thanks!!

You caught us by surprise. As you can see we've built in the ability to gracefully handle a lot of traffic but it's not ideal because you need to wait in a queue until we have enough capacity. So I wanted to jump on here and give you an update of a release we're putting out next week.

FastOrSlow does actual browser simulations from around 12 locations around the world. When we first architected it, we wanted to provide as many locations as possible including places like South Africa. That meant in some cases we needed actual bare-metal servers. So we built it on bare metal in data centers around the world.

We experienced some moderate popularity about a month ago and realized we needed to do a better job of scaling on demand. So we re-evaluated moving on to AWS. Around the same time AWS launched in Cape Town and a few other locations, and so we were able to get the kind of coverage we wanted globally.

So we kicked off an AWS migration. We're days away from moving into AWS and using spot instances to ramp up and down fast based on demand. But as I said, you caught us by surprise.

So please give FastOrSlow a try today, but know that by around late next week we'll be doing a much better job of handling your requests FAST without having to wait in line.

Thanks for your interest and patience. Myself and the lead architect, Ryan, will be here to answer any questions and of course listen to your suggestions.

Regards,

Mark.

We'll also be expanding to 18 locations globally once the migration completes.

We have a lot of jobs in the queue right now. Just passed 1000 folks waiting. Just wanted to reassure you that your job will finish. Just leave the browser window open, take a break and come back. Even if you close your browser, as long as you come back to the same URL, your report will be there once the job is done.

If someone closes the browser, you could remove the user from the queue to save time for now. Lot of users would be closing the browser and might not come back.

This is a really cool tool, and seems much more polished than other similar sites I've used in the past. So first, thanks!

Second, though—this seems really expensive to run, based on what you're saying here. How are you monetizing or otherwise funding it?

FYI, if you don't want to wait for results (because we're slammed right now via HN) here are a few examples:

www.wordfence.com: https://www.fastorslow.com/app/profile/f5fcf22f-8daf-5567-b3...

cybersecurity.miami.edu: https://www.fastorslow.com/app/profile/5ad00348-8c5d-566c-8a...

apple.com: https://www.fastorslow.com/app/profile/a8f2f34f-b695-5ce9-b6...

facebook.com: https://www.fastorslow.com/app/profile/afffcc61-4eb9-5e6c-88...

Thanks!! I run Wordfence which does quite well. Profitable and the team is around 35 really great folks with a focus on solid engineering. So we fund it ourselves. For a going concern like us the costs are very modest and we can afford to run it indefinitely without charging.

In fact just the bit of brand awareness we get for our main product, Wordfence probably more than justifies the costs.

We actually created this as an internal tool because there wasn't anything that would give you a clear indication of site performance from DNS lookup to connect to TTFB to page render from around the world. Our own market is global and even a 1% drop in conversion really costs. After building it we realized that it's very little incremental effort to make it available for all.

As I mentioned we're about to migrate into AWS where we run much of our other infrastructure. The financial model and eng design we came up with keeps our costs surprisingly low. We can handle huge traffic spikes and it roughly doubled the cost, even with the spikes. So I'm confident we can run this for the long haul as a free product.

A few things I can commit to: Firstly, it's extremely unlikely that you'll see ads on FastOrSlow other than our own company name. That's never been our model and I can't stand when a great site or service gets smothered in ads.

In addition, our company has been around since 2006 and we've been made acquisition offers many many times by large companies that you know well. We've consistently turned them down and will continue to do so because we are profitable, love our independence and I think we all know what happens to many great products after acquisition. I'm telling you this because I want you to know that this won't be changing hands to a company that will try to monetize it in obnoxious ways.

No plans to offer a paid service right now. Our next release is for internal use and will allow us to do performance surveys across large numbers of sites and present statistically significant insights as research to the online community.

Happy to answer any other questions. Thanks for your interest!

Should have mentioned, but it's probably obvious, that we also don't need funding. So won't be raising VC with the pressures that come with that. Nothing against folks who do, but that's our situation.

You're saying all the right things :) thanks for the awesome level of detail, too! You guys seem like a really cool company. I'll definitely be making good use of this product in the future; thanks for making it and sharing it!

Thank you! We're hiring!! (Sorry, but I'd be remiss if I didn't mention that). We're currently hiring PHP developers and for QA. visit our company website at https://www.defiant.com for details.

Darn, if I were in the US I would have applied. Fantastic website!

A WAF specifically marketed at WordPress is a clever idea.

You mentioned that Wordfence has a team of 35 behind it. This seems like a large team, and I'm curious to know what the distribution of roles is like? Also interested to know how many users and paying customers Wordfence has?

We focus fairly heavily on engineering, particularly on QA. My co-founder Kerry Boyte ran QA at Symantec, eToys, BBC, and a few other well known companies. She actually built BBC's digital TV testing lab herself. Ran testing for the MS Exchange version of Norton back in the 90s. And ran the web QA team at eToys before the suits messed things up there.

I'm a dev (ceo these days) and I wrote the first version of Wordfence. Also worked at many big and small companies in South Africa, London and USA. We're US based now. And so we're both engineering focused, and our process is to only ship rock solid code via excellent QA.

Funny thing. Finding vulnerabilities is actually just QA. And a vuln is just a celebrity bug. So this culture really helps foster great security research too.

Another big area we invest in is customer service. Our CS team does a great job and we also focus substantial energy on community support in free forums, which really helps foster goodwill from the community and build a relationship with folks who use our free product, many of whom become paying customers.

I started my career in ops in the 90s before moving to dev and am passionate about high performance (as you can tell from fastorslow) - and so we spend a lot of energy and budget on solid operations to support our products.

Kerry is kind of a math guru and is our COO and CFO these days. So we run a tight ship in terms of financials, which is pretty much why we're still around. Cashflow plan updated daily. Awesome projections. Awesome granularity. Sorry if I'm over-sharing, but I love entrepreneurship and perhaps this will give other folks starting out some insight.

We have around 3 to 5 million free customers. Can't share how many paying, but it's a lot.

I haven't coded for some time now. In 2015 I completely handed off coding on Wordfence to a brilliant developer named Matt who is still with us. I've moved my focus 100% to leadership, strategy and somewhat of a CTO role, but I share that with the strong tech leads in our engineering team.

Hope that helps. We're a private company, so I couldn't just dump our internal data, but I hope that gives you some flavor of who we are, our focus, team distribution and culture.

Regards,

Mark.

Sounds like a good product, with a well thought-out process/vision, and a great team that enjoys working on it.

Congrats on the success!

Thank you.

I feel that half my job as WordPress developer is knowing or finding quality plugins. It can be a real jungle sometimes. Reading your posts leave no doubt in my mind that Wordfence should be my go-to security plugin in the future.

Thank you for sharing the history of the company Defiant, about the people who develop WordFence, and the guiding principles and approach in management.

The overview was insightful, to see what makes up a well-run operation: how the engineering, tight financial management, quality assurance, performance, security, customer support and fostering the user community - all tie together to make a solid, growing business.

It definitely earns trust and respect for the company's culture and decisions. As a dev learning the ropes in a CTO role, this kind of open communication about the internal workings of a successful company and product is great food for thought.

To decisively not seek acquisitions and venture capital, and instead value independence, profitability, good finances, engineering, QA.. This will stay in my mind as a role model.

If there's an opportunity where I work, I'll likely recommend WordFence or other services by the company. (In fact, performance profiling of large/heavy sites has come up a number of times this year.)

Elsewhere a possible blog post on migrating to AWS was mentioned. I love that style of articles, describing the thought process that goes into technical decisions, how it was planned and implemented. Hope I can catch it here if/when it's published.

Wordfence is a solid product. I 'inherited' my wife's business's website a few years ago from a small marketing firm. It was heavily infected with malware. I tried several tools to clean it up and Wordfence is the one tool that consistently helped me remove all malware and keep it off. I was using the free version for quite a while and just upgraded to a paid license yesterday as a matter of fact.

In the world of Wordpress plugins it is a gem in the rough.

Thank you, and thanks for your business.

> We actually created this as an internal tool because there wasn't anything that would give you a clear indication of site performance from DNS lookup to connect to TTFB to page render from around the world.

Out of curiosity, did you look into any paid services like SpeedCurve or Calibre? There are also OSS projects like SiteSpeed or WebPageTest.

Full disclosure: I work at SpeedCurve. We've had customers cancel their accounts in the past because they wanted to build their own performance monitoring tool. To the best of my knowledge, most of those customers have come back because building and maintaining a tool like that is no easy task.

...Which I guess is a strange way of saying congratulations on building a performance monitoring tool!

Nice job Mark! The site looks super clean. It's nice to see a post like this explaining what's happening behind the scenes.

Remindme! in a week ;)

Thank you for your kind words. The rest of the team is on this thread and it means a lot to all of us.

Wow, so cool! How does this compare to something like Google Chrome Lighthouse? I think there's a webapp: https://developers.google.com/speed/pagespeed/insights/

Is there any place we can sign up for updates on when the AWS migration is up and running?

I wish we'd thought of that! Once we're in AWS we'll be posting research because we've created an internal tool to do large scale performance surveys e.g. comparing hosting providers or performance of sites using certain libs etc. So I'll put that on the blog and HN might pick it up.

FastOrSlow is built using Lighthouse. :) We do actual browser simulations using headless Chrome and lighthouse to get the performance data. We do it programmatically from all the locations, bring the data back and build the report.

I did think of one thing: You can follow us on Twitter at: https://twitter.com/isitfastorslow

With the positive comments here, perhaps the HN mods wouldn't mind if you posted an update yourself once the AWS migration is completed? I know there's rules around that, but I think they are okay with the occasional self-promotion, especially for great, free tools.

We would be more than happy to chat about how we migrated to AWS, the differences, benefits, down-sides, surprises. Will be a fun post. I'll let rbritton know and we'll put it together, and of course you can try the product too if you like. FastOrSlow has a blog, so we'll post it there.

You could also think about throwing someone to the back of the list is their browser hasn't pinged the server in a while.

Thank you for the tool and the details you've shared! Is there any particular reason you opted to run your own instances of Lighthouse rather than use the PageSpeed Insights API provided by Google? API quota limitations, perhaps?

This is hilarious

https://developers.google.com/speed/pagespeed/insights/?url=...

Saved you a click, the result is:

Lighthouse returned error: ERRORED_DOCUMENT_REQUEST. Lighthouse was unable to reliably load the page you requested. Make sure you are testing the correct URL and that the server is properly responding to all requests. (Status code: 429)

Hi, great tool. I am in line (1,201) can you talk a little about how the dynamic spot instances work? Do you kick off spot instance based on CPU load or network usage of all other spot instances in the cluster? Are all regions created equal, or do some regions get more requests than others?

I have to imagine your bandwidth costs from AWS aren't gonna be cheap.

I think you understand this, but just want to be clear that this is only launching next week. We're currently still on baremetal.

It'll be based on sustained queue length averages rather than any kind of load number. We can also bump it up manually.

Our estimates for some fairly severe traffic based on historic data are quite reasonable for bandwidth costs. But we're a medium sized business, so we have revenue to support that. I think individuals and very small teams can get really surprised by this.

If you're just starting out and don't need dynamic scaling and the tooling that AWS provides, I'd seriously look into colo so that you can buy bandwidth using 95th percentile billing. We use a combination of servers that we own in data centers using 95th billing and AWS. Depends on use case and we have gotten very good at financially modeling future costs for a given config.

AWS doesn't charge for incoming bandwidth, so the site has advantages there.

One thing that might help a bit is to make the profiles you've already done public (well a curated list of popular sites maybe so you don't leak information). Have a list on the home page, so I can see what a report looks like, even if it's not the page I want.

Or maybe the page I want will already be there!

Definitely worth thinking about. Here's one from yesterday that I also posted below: https://www.fastorslow.com/app/profile/8b282a3c-db98-5b1a-98...

Oh hey Ryan. Didn't see you there. :-)

Ryan built this thing.

Oh brilliant. Not sure why I didn't think of this. Here is our company website profile for https://www.defiant.com

Profile: https://www.fastorslow.com/app/profile/8b282a3c-db98-5b1a-98...

If you get a 502 or if it takes a few seconds to load, we're busy adding another DB reader, so just reload and it will appear. Should stabilize as the new DB reader comes online. HN is slamming us right now.

Hey FWIW I'm not seeing the graceful queue. I'm getting a 429 on my very first request!

I think that's the queue ;)

I am 2889 in line, and it doesn't seem to be updating. I have no idea how long it will take, a time estimate would be nice. Is it 1 minute or 1 month?

It will depend a bit on how fast or slow the sites are in front, but a general estimate is about 10-15 sites per minute within the limits of the current architecture. Position 2889 will be a while, but it will be today (depending on time zone).

Oh good, I made it look up itself and was worried I broke it for everyone.

Be careful, Kurt Goedel once caused a crisis in the fundamentals of mathematics by doing something similar.

Also: If you type "Google" into Google - you can break the internet. Nobody has ever tried, not even for a joke.

Would you consider writing a post on your experiences moving to AWS and spot instances for dynamic scaling? I'd love to hear more about the architectural changes you made for that.

Absolutely. Will do.

This looks like a very cool tool, thank you! I totally understand the "getting surprised". This of all this interest (and the big queue!) as confirmation that this tool is of general interest.

Just heard the idea for this on a podcast a month ago! On the My First Million podcast

Does the tool detect when the site responds with a "we don't want you here" page from specific regions? E.g. the Cloudfront blacklist, Cloudflare captcha/proof-of-work challenge, etc. These pages are usually quick to render and may skew results.

They show the screenshot of the page loaded per region so if it did, you'd notice.

It seems you aren't caching results. Cos I typed in Apple.com and received a too many checks error instead of just returning a cached result.

* A slight tweak to your site allow you to save cash and handle surge traffic with your current server.

1. Returning cached results for sites that's have been recently checked.

2. Collect email addresses and mail the result when it's ready.

That's all.

It's a much better experience than leaving the tab open, frantically refreshing the page to see if the queue is reducing.

I collect sites like these! Here's my list of Every Web Performance Test Tool: https://www.swyx.io/writing/webperf-tests/ (will be updating with FastorSlow)

Nother one to add to your list perhaps? https://lighthouse-metrics.com/

added!

Would you mind adding Amazon CloudWatch Synthetics to your list? https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...

added!

Security Headers would fit nicely in your ‘improvement recommendations’ section: https://securityheaders.com/

Mozilla has a nice one as well: https://observatory.mozilla.org/

done, thanks!

I appreciate that it shows your place in line (I started at 53). This is really polished.

Commenting four minutes later: I started at 350! Hacker News bump I guess?

3 minutes later, I started at 483… definitely Hacker News bump!

And I'm in 661, 3 minutes later.

I'm in 429th place…wait, that's an error code.

One hour later and 2151 in line ahead of me, talk about exponential growth.

https://www.fastorslow.com/app/profile/244179fc-78d2-5cf3-8c...

And now just getting 500 errors :(

Working on it. Looks like a queue issue. Adding more workers. All kinds of fun this evening.

Edit: From our eng team:

"We're moving about 5000 events per second now. Definitely shrinking now, and memory usage is dropping too."

Should be good to go now. Sorry, but massive traffic right now.

Thanks, and we really appreciate your patience. See my comment regarding moving into AWS to improve scalability on demand.

Hi! I work on Lighthouse and Page Speed Insights (similar service to what you got here). Nice to see another player in this space :) congrats on the HN bump!

Wondering if you've had any pain points integrating Lighthouse that you wouldn't mind sharing. Anything more difficult than it should be, or were the docs lacking in some way, etc.

504 ERROR The request could not be satisfied. CloudFront attempted to establish a connection with the origin, but either the attempt failed or the origin closed the connection. We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner. If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation. Generated by cloudfront (CloudFront) Request ID: Ba3iKlMjJUrRy2BYUVqVLmm4hcOF70KSJgcfFnU2SUdQFLTIriKq8A==

Thanks. Seeing the same thing. Our team is on it. Just woke up our head of ops at 5am his time. Surprisingly Scott sounded awake and as if he was expecting my call. :-)

Will be a brief outage as we change the instance type on a server.

And we’re back.

Quick update: We were generating 500s for the last 15 mins. Our worker queue got overwhelmed. From the eng team:

"We're moving about 5000 events per second now. Definitely shrinking now, and memory usage is dropping too."

Should be good to go now.

Can someone post a completed report so we can peek at it while waiting in the queue?

https://www.fastorslow.com/app/profile/2bd978bd-f107-5761-84...

It seems cool but there are like 1k sites before mine, i wonder if you guys can implement a feature to email the report after the report is generated in the mean time (seems like a small feature).

Plus, they can collect e-mail addresses they can market and target to. Great idea.

That is spam.

How is it spam when you explicitly request to be emailed?

You explicitly request to be mailed the results, not marketing spam.

A checkbox to receive marketing emails as well solves the issue (and yes, I do sometimes click I want to be mailed news and marketing).

I don't expect the site to be fully crowded all of the time, but have you thought about adding caching for previously processed sites? Perhaps a choice to see either a previously generated report or create an entirely new one.

The reason I ask is that I did a report last night, and checked it on my phone in the early morning. I put my site's address in again from my desktop now, and I see that I am 1,000+ in line. Even though a recent report was made for this site already, I can't seem to access it.

Thanks. You can access previous reports but you need to save the url. And they are cached. But this is a good idea. Thanks.

Tip: if your site is going to redirect from the bare domain to something else (e.g. `https://example.com` -> `https://example.com/home/`) make sure you enter the final destination, letting it redirect means that the cost of the redirects will be included in the report, which is unlikely to be what you want.

Unless that redirect is something your users are likely to experience, in which case you shouldn't cheat and provide a URL that delivers a different loading delay than visitors will encounter.

Well...in many cases all it really tells you is the roundtrip network latency, which may or may not be something you are interested in.

In my case it actually was sort of interesting - it definitely suggested that the redirects don't have the correct caching headers, because they clearly weren't being served off the CDN. However, this redirect is a pretty unusual case for us, because most of our traffic is either from SEO or links from notifications we send users.

I agree that you should be looking at the experience most users are going to have, but if the first experience for most users is a redirect you should probably do something about that.

Yep. We try to indicate when you're being affected by redirects since `http` -> `https` and `example.com` -> `www.example.com` are common ones. Those can often mean hitting an origin server instead of an edge server first.

Yup! As I noted in a comment above, this is just not a common case for us - usually our inbound traffic is coming from SEO which sends you to the right place. But when I am typing it into a search box, it's real easy to forget.

I am going to look into getting those redirects cached though, there's no reason we shouldn't be able to serve them off the edge.

Usually you want to include that time, especially if most of your users will be going to the first url with the redirect.

Users will typically hit redirects like that when typing in a URL manually, but not in the much more common case of being linked to the site.

This is build by the company: https://www.wordfence.com

Thank you for creating this.

It was AHA moment for me when i looked at "Geography" and "Location" section. Pretty cool stuff.

Thanks!

How does this compare to webpagetest.org?

Does someone mind sharing his own result so that we cant check the tool without waiting 3 hours please ?

Here's one: https://www.fastorslow.com/app/profile/8b282a3c-db98-5b1a-98...

And another https://www.fastorslow.com/app/profile/2bd978bd-f107-5761-84...

Nice, my marketing site that's not even on a CDN yet gets over 90 score in all regions. :)

Where is hosted?

USA, CA, Fremont.

I've had some issues like "There is a more recent report available for this URL. Would you like to view it?" but then when I try to, it says "No reports found".

These pages use an awful lot of CPU when they’re active—in Firefox Nightly on Windows, a completed results page was consuming an entire core, which is not usual.

I'm getting "429 Too Many Requests"

Reminds me of a similar but completely unrelated phrase from the military: “slow is smooth, smooth is fast”

Shooter IIRC with Wahlberg. Love that old gunsmith in the Georgia forests.

I got a 429 Too Many Requests on my first try...

What.

I'm seeing an internal 500 for all pages. Will try again later or tomorrow I guess.

From our eng team:

"We're moving about 5000 events per second now. Definitely shrinking now, and memory usage is dropping too."

Should be good to go now. Sorry, but massive traffic right now.

Their own website is super slow.

Why do you suppose that is?

Intermittent right now but working on it. Should be back in a few mins.

We came back within a few minutes. Posted an earlier update above.

Yeah, let me know when it actually works. Started at about 1200 in the queue and 4 hours later I'm still number 81, and have been sitting at 81 for the last hour. Sounds interesting, but I think you launched too soon.

Oh great, someone is down voting me. Another four hours later, and I'm still showing #81 in the queue.

Your site eats up a lot of CPU for some reason

500 error

Looks like a queue worker issue. From eng: "We're moving about 5000 events per second now. Definitely shrinking now, and memory usage is dropping too."

Should be fixed.

Edit: I was wrong. But also check out https://webpagetest.org/ as an alternative (that you can host internally as well)

We're using Chrome's Lighthouse running in containers with a focus on normalized performance between locations, security and responsiveness. This is built from the ground up by us.

Ah, that make sense then. Most of the Lighthouse recommendations were copied from WPT.

Besides the underlying insinuation (what's wrong with tools copying others if the ideas hit all the right notes?), what recommendations did Lighthouse copy?

WPT is mainly a supertool—here's a ton of data, go dig into the waterfall and network requests with some visualizations. I think there's a page for recommendations but to be honest I've never used it nor seen anyone talk about it.

With Lighthouse's performance report, we aim to provide specific actionable advice with estimates for how impactful it will be.

Are you using https://www.sitespeed.io/ under the hood or is your tool custom-built around lighthouse reports?

Quote from the eng team: "Never actually heard of it. We're build around the raw data provided from the built-in Lighthouse audits plus some of our own custom ones"

Reply on news.ycombinator.com