Whatruns: Identify technologies used on any website
whatruns.com
whatruns.com
We are truly stunned to see us on top of HN today! :)
WhatRuns is a free browser extension that shows you what runs a website – from ad networks and developer tools to fonts and Wordpress plugins. You can also follow websites and get notified when they add or remove technologies.
We soft-launched a couple of weeks back and was lucky enough to be picked up by the Chrome team. We were featured on the Chrome Webstore, landing us 12k active users in one week. It was a huge validation and helped us tremendously in squashing bugs and making a finished product. We realise we have a long way to go, and our little team is working round the clock to make it happen. We also launched on ProductHunt today: https://www.producthunt.com/posts/whatruns
Would love to hear what you think :)
UPDATE:
Thank you for all the feedback!
Sorry about the occasional false detections. We are looking into this. This is largely because we detect a considerably large number of technologies/plugins compared to our counterparts. Lots of possibilities for false pattern recognition etc. Rest assured our team is working round the clock to improve accuracy and add more technologies/plugins.
Also, Our servers are going a bit cranky due to the huge traffic we are experiencing today. New websites (that was not loaded on WhatRuns before) are now queued up and might experience 2-3 seconds delay. This is to ensure best experience for our active users.
Thank you so much for such a great response!
We were featured on Chrome Webstore a few weeks back and got a great response (12k+ active users) helped us enormously in improving the accuracy and efficiency, and I'm sure HN and PH launch will be even more helpful in improving the product.
Also, Wordpress doesn't seem to be detected, either, on my other website: https://johnrockefeller.net
It's 2017 man, things are launched with kinks all the time now. Your definition of launch ready doesn't have to apply to everyone else.
However, there is a lot of manual labour involved in correcting detection inaccuracies, which our team is working full-time on. Rest assured WR will only improve from here on. Thank you for dropping by!
Try running it on www.example.com and let me know how many of those are accurate.
That is the assumption of broken hardware to be shipped 17 years avant la lettre?
IMHO it’s a very good thing we’ve "accepted things are going to broken at launch"; the "ship early, release often" model works a lot better than "wait 10 years before release so that everything is perfect but you’re 9 years late".
> Built With:
> - Ruby on Rails
> - Ember.js
> ...
From another comment I understand that it shows all detecting stuff for all subdomains, so this is the case.
1. WhatRuns detects fonts, Wordpress plugins and themes (tens of thousands of them).
2. Ability to follow sites (and know what techs websites started using/ditched).
3. Very lightweight compared to our counterparts, and arguably better UI ;)
4. More accurate data. BuiltWith can be very inaccurate as you might've already noticed. Wappalyzer is fairly accurate, but limited in technologies. WhatRuns is trying to be the best of both worlds.
It seems to only look at the second-level domain, and thinks that websites with the same subdomain are the same.
They are not.
Most users like to know the full tech stack of a website. If there is a blog at blog.company.com and if it is using Intercom, it can be a useful data. I hope this makes sense.
Anyway, we will definitely address this concern and think about adding an option for subdomain separation.
But all of them are evaluated as one, govt.nz. But they are all quite separately hosted and operated and use differing technologies.
Looks like this works fine for other similar situations e.g. .co.uk.
This is a really cool service, thanks for making it available for use!
Thanks, and glad you liked the product!
I tried a great deal of sites using similar tech, and an infinite spin is all I get on all cases.
Please drop me a line if you're still experiencing this issue: jijo [at] whatruns.com. Thanks!
Looks like it's not picking up Django! ;)
On a serious note, I understand your point and realise how new extensions can be dangerous. However, we have a very good team and is trying to solve all the concerns we had with our counterparts.
I hope you'll give us the benefit of doubt! :)
Just add a "insert url here" box and do it on your server.
I implicitly distrust anyone who insists on running code on my PC that they could run elsewhere.
Especially.. code they can remotely update
Also, our counterparts got a majority of their traction from browser extensions which made it our obvious priority (even though it wasn't the easiest of options).
We know 15k is not a lot (comment was pun intended), but it looks like a good start :)
I dont know much about extension development - can a finer permission like - reading browser url is not sufficient to achieve functionality ? or better - a button in extension options to read only current url ?
How do you defend yourself of not selling users data ?
Having said above - compared to extension wappalyzer (which I had) this gives so much more information!! Really cool.
To address your concern with the privacy, WhatRuns do not collect or log any visitor information – including IP address, location etc. We receive anonymous website data and match with our database to display the results. Hope this clarifies.
https://chrispederick.com/blog/web-developer-for-chrome-comp...
I doesn't, so this is more of a parlor trick thsm a practical way to police extensions.
Here is how to set up a separate user profile
Edit: Looks like I'm probably wrong since I see they have a tool named "Stack Scanner"
1. WhatRuns detects fonts, Wordpress plugins and themes (tens of thousands of them).
2. Ability to follow sites (and know what techs websites started using/ditched).
3. Very lightweight compared to our counterparts, and arguably better UI ;)
4. More accurate data. BuiltWith can be very inaccurate as you might've already noticed. Wappalyzer is fairly accurate, but limited in technologies. WhatRuns is trying to be the best of both worlds.
No, you are not.
I'll explain.
We use several signals like code snippets, filename, directory name, header info and several others to accurately identify technologies. However, there are many possibilities where this can go wrong even with few signals correct. Every time we detect a technology, we calculate a probability of its accuracy and filter out the rest. This system self-learns and improves the identification over time. Hope this helps.
You are most likely have a system that consumes content, compares that content against known hashed variants. If there is no match, you diff against known variants and check if the output matches any of the 'minimal' implementations.
If you can't match anything, you simply stage that content for a manual review.
On a serious note, I'm with you on how new start-ups go overboard with buzz words. As for WhatRuns, it was intentional that we do not use any jargons to advertise our product on the website, Product Hunt or HackerNews, so that it does not lose its charm.
For a new startup to achieve this scale in technology identification and accuracy compared to established players with more than a decade of development (and data), it is self-evident that manual labour would not yield such a result. In fact, technology breakthroughs and an excellent technical team were the reason why we decided to give this shot in the first place.
We plan to publish a comparative study on our experience with the effectiveness and superior prediction quality of deep learning vs normal pattern identification on our blog (which we will soon move to Medium). Stay tuned! :)
Your extension only works on Chrome, and it is for a feature that is not used commonly. There is no good reason to install it on a web browser.
Installing it in a browser is also a threat that the extension might do more than just scanning sites, and even if it doesn't affect privacy it still encourages installing extra junk on web browsers.
BTW WhatRuns works on all major browsers.
[edit] to be fair other options I tried don't detect the backend at all. This is a single page app with rails api so I get that might be harder than a rails app with server rendering and full page reloads.
If Whatruns had that feature, I'd seriously consider switching. But otherwise, there's just no way I could. It's way too convenient.
However, Builtwith is selling some plans which also include SEO related features like keyword reports. I understand that some might pay for latter but there is even more competition in that space.
What I don't get: Who should pay for your stuff? It's of course interesting to see other stacks but honestly it's not a crucial thing. My CTOs and I know what they are doing and of course we like to get inspired but yeah, at the end of the day tons of research, years of experience, debate and the individual use case decide our stack and not what some random website does. Same for design-relates stuff, btw to find a font-face is just a Command-Option-I away.
So no offense, but I am just wondering why you start a business which is already there, which is hard to scale and which is hard to get paid for.
Guess I missed something and happy to hear your view.
If you are selling a Wordpress plugin, you don't want to contact website owners that run on e.g Joomla.
Also, we are planning to introduce a predictive sales system which will suggest clients based on their technology adoption. For eg., if a company migrates to Magento, they are a potential client to Magento extension developers.
BuiltWith has been around for a while and has it's own chrome extension [0]. It correctly identified fonts.google.com as using Angular.
[0] https://chrome.google.com/webstore/detail/builtwith-technolo...
The latter is what I've been using and seems to have more users with higher ratings
[1] - https://chrome.google.com/webstore/detail/wappalyzer/gppongm...
1. WhatRuns detects fonts, Wordpress plugins and themes (tens of thousands of them).
2. Ability to follow sites (and know what techs websites started using/ditched).
3. Very lightweight compared to our counterparts, and arguably better UI ;)
4. More accurate data. BuiltWith can be very inaccurate as you might've already noticed. Wappalyzer is fairly accurate, but limited in technologies. WhatRuns is trying to be the best of both worlds.
1. All in all, this looks really tidy, so nice work!
2. Sadly, it looks a bit limited on detecting anything .NET/Windows. I pointed it at a few Umbraco sites running on Azure, and none of it was picked up.
3. It doesn't look like it works for subdomains.
4. Wappalyser does a good job of detecting Angular 2, whereas this seems to struggle.
These issues aside, I'll probably keep it running at work, and if these things can be resolved I can see this being my preferred choice.
Addressing your concerns,
1. Thank you ;)
2. Devs are looking into this. Neglecting .Net/Windows wasn't intentional. We will work on this.
3. Yes, WR currently considers subdomains as a part of the main domain. Most users like to know the full tech stack of a website. If there is a blog at blog.company.com and if it is using Intercom, it can be a useful data. I hope this makes sense.
Anyway, we will definitely address this concern and think about adding an option for subdomain separation.
4. Noted!
Might be network weirdness on my end, I dunno. (Or a HN "hug of death"?) Anyway, wanted to let you know.
Congrats on the project :)
New websites (that was not loaded on WhatRuns before) are now queued up and might experience 2-3 seconds delay.
I only have one extra UI recommendation that I think Wappalyzer got right, which you could enable as an option.
When a popular CMS/language/server OS is detected, Wappalyzer will use its icon in place of Wappalyzer's plugin icon. E.g. if Joomla is detected, Wappalyzer's icon on the plugins' toolbar will switch to Joomla's logo.
There's a specific order to this preference that looks to go from the CMS used (e.g. Joomla, WordPress etc.) down to the framework (e.g. Laravel), programming language (e.g. PHP), webserver (e.g. Nginx) and finally the server OS. In other Words, if Joomla is detected, it will be displayed first, not PHP.
The above is extremely helpful for anyone developing for the CMS communities (like myself).
Of course, to maintain your identity as a plugin, you could use a double logo (a mashup of your own and the dominant/higher-level technology detected).
* UPDATE: You should also consider providing a way for anyone to easily suggest new frameworks, apps, CMS extensions/plugins etc. to be detected, by providing a name, icon, description and the way to be detected (e.g. HTTP header, pattern in the HTML output or even HTML comment, linked source etc.).
Dynamic icon - I agree with you that it can be quite convenient to display the top technology (preferably the CMS). We will think about this as an option in the future updates!
Technology submission - That's a great idea. We are adding this to our roadmap. Thank you so much.
If you keep your icon the same I will switch from Wapalyzer.
Update: You've already won me over with the better layout (compared to Wapalyzer), separating them into technologies. The fonts is a nice addition too.
Once you've fixed this I'll be sure to use it regulary.
Check out the privacy policy before installing any Chrome Extension.
The golden rule of business: If you're onto a sweet money-maker, don't shout about it.
I'm currently working on a competitor to a site I read about that bragged about their business model, and if they'd have kept it to themselves they'd be facing one less competitor...
1. WhatRuns detects fonts, Wordpress plugins and themes (tens of thousands of them).
2. Ability to follow sites (and know what techs websites started using/ditched).
3. Very lightweight compared to our counterparts, and arguably better UI ;)
4. More accurate data. BuiltWith can be very inaccurate as you might've already noticed. Wappalyzer is fairly accurate, but limited in technologies. WhatRuns is trying to be the best of both worlds.
https://gist.github.com/9b/f5fe434bf9965d673963884b56d93d9a
On the privacy side, I could see concern from those using the extension. When the site is not found in their database, the full HTML of the page appears to be submitted to the servers and processed. This is a bit of what you would expect, but may present some concern for cases where a new site is submitted and PII is sent to WhatRuns servers.
Privacy side, I'm sure you noticed that we filter out any text content before sending html tags ‘anonymously’ for technology identification.
Unfortunately some sites that I am responsible for running in production are WP and we try our best to hide this fact and block all admin functionality to the public due to WP's less-than-stellar history of security vulnerabilities. This is the first tool I've seen that has detected it and now I'm stumped.
Can't really hide wordpress because any time there's a new vulnerability, scrapers spam every site on the internet attempting to use it it anyway, regardless of what tech they're built on
Also, most WP pages will be loading scripts from from the wp-includes directory. There are probably others I'm overlooking, and some WP plugins probably also drop recognizable script tags into your pages.
Since this is the first tool that has detected it, it's very possible you've already covered all of the things I mentioned.
- Rename paths to eliminate "wp-" prefixes and recognizable folder structure (wp-content, wp-include, etc)
- Remove or rename any common plugins that inject recognizable WP-specific code into the page
- Rewrite requests to bare paths instead of e.g. index.php
I assume you'd also try to do as much handling as possible at the Apache/NGINX layer instead of letting requests hit the WP application.
Seems like a HUGE amount of effort, and I'm probably not even getting everything. Is there a more efficient way of securing/locking-down a WP site?
Ultimately it isn't a huge deal for us though, since it runs concurrently with other build/deployment steps that in total (sequentially) take a similar amount of time.
One thought I had was perhaps it uses some cached batch parser and shows "No apps found" for all sites on first-run until it finishes analysing in the background? It doesn't seem to work at all on a few very obvious but small/obscure CMS sites but works fine on all well-known high-traffic sites.
That's not the whole truth - you are using Google Analytics to track visitors and you fail to disclose this in your privacy policy, despite this being mandatory under the Google Analytics T&C's.
Well done launching what looks like a very cool project, and I hope you can further improve it by informing visitors that you are using Google Analytics to track them (or even drop GA completely in favor of something privacy friendly).
Good luck with your thing. I am sure you did a ton of work; I am just naturally risk-adverse when it comes to installing extensions that that a potential to do things I might not want.
I'm not sure what extra tracking they do beyond that!
I would try it as a bookmarklet but I never install Chrome extensions that ask for all data on all websites. It's just an insane permission for what should only get URLs when I explicitly ask it to.
I wish Chrome would add a permission like this "website URL of the current page with your express permission every invocation".
I see comments like yours on this site pretty often, and it is tiring. There are many reasons people behave the way they do, and probably the most common reason is that their behaviors haven't caused them any harm as far as they know.
The warning "Read and change all your data on the websites you visit" is perhaps scary the first time you see it, but then it becomes insignificant as time goes by and as extensions get installed without causing any visible harm.
Which is exactly why it's dangerous. Granting access like this without a thought to the potential consequences is just asking for a bad character to take advantage of the blind trust people place in extension authors.
The core issue is the options Chrome gives extension authors. Offering the ability to grant permissions per-site and per-use would greatly reduce the threat. Even just a per-use "Are you sure?" confirmation would help.
LOL, what matters is the threat itself and not your waning level of apprehension over the threat. This is really a very, very strange comment. The point is there is no need for this to be a browser extension. Putting an input element and some AJAX on the page is trivial, so I really don't buy the excuse that they haven't had time to put together a web app yet.
It's disappointing you can't have finer grain permissions for Chrome Extensions. What's the alternative though if you can't make it a web service though? A Electron or native app for example would have even more permissions and could read any file on your computer.
Also, our counterparts got a majority of their traction from browser extensions which made it our obvious priority (even though it wasn't the easiest of options).
Not to worry though - we're working on something for the web as well!
I wonder if the mods would ever be interested in being interviewed or talking about some of the tech. The last bit of Arc info we got was https://news.ycombinator.com/item?id=11240681, which was awesome.
It's pretty unique. I don't think any other large website in the world has written their own stack from top to bottom. Even Facebook uses php.
Having to explicitly declare thread local access is a clever hack.
I also wonder what database they use (if any).
Did they also build their own http stack?
Originally they did build their own http stack but switched to nginx for a reverse proxy. On the other hand I'm not sure how much they lean on nginx's facilities.
Noticed that it doesn't report correctly for subdomains - one of the sites I built is at foo.megacorp.com, but the extension reports the results for megacorp.com which is a separate property.
Most users like to know the full tech stack of a website. If there is a blog at blog.company.com and if it is using Intercom, it can be a useful data. I hope this makes sense.
Anyway, we will definitely address this concern and think about adding an option for subdomain separation.
The spinner does not stop and it gives no results for my site (https://myhikes.org) - this is with both FireFox and Chrome extensions. Seems to work great for everything else though.
Just replying in case you're looking for new edge cases to debug!
Oddly enough I restarted Chrome and Firefox twice before posting the comment in hopes that it was just my machine. Thanks for the sanity check!
Most users like to know the full tech stack of a website. If there is a blog at blog.company.com and if it is using Intercom, it can be a useful data. I hope this makes sense.
Anyway, we will definitely address this concern and think about adding an option for subdomain separation.
Seriously people.
it is capturing my browsing behaviour because it sends any URL i browse in the background to the whatsrun.com server, even when i don't want to know what software the page is running (means clicking the icon), so Whatruns get's a full browsing history from me (and you even set a UUID cookie to track unique users!).
This is a huge privacy issue! Imagine Whatruns is starting to sell this data!
To replicate simply open the dev-console for the extension and click the network tab.
Hope this clarifies. Drop us a line if you still have any concerns, would love to clear it for you: hello [at] whatruns.com
https://www.whatruns.com/website/ycombinator.com
That sounds strange. Can these claims be backed up somehow? I cannot see anything in the source that would confirm these.
It also says Facebook uses Google Analytics \o/
Most users like to know the full tech stack of a website. If there is a blog at blog.company.com which is using Intercom, it can be a useful data. I hope this makes sense.
Anyway, we will definitely address this issue and consider introducing an option for subdomain separation.
If you're still facing the issue, please drop us a line with the URL in question so that we can take a look: hello [at] whatruns.com. Thanks!
I find myself getting slightly better at this as I spend more time in web development.
I run jQuery, nginx have google analytics and have my ssl certificate with lets encrypt. All stuff that builtwith.com found without any issues.
We truly understand your frustration with detection accuracy, but when there are tens of thousands of technologies to detect, the only solution is to break things and move fast.
We were featured on Chrome Webstore a few weeks back and got a great response (12k+ active users) helped us enormously in improving the accuracy and efficiency, and I'm sure HN and PH launch will be even more helpful in improving the product.
If you're still facing the issue, please drop us a line with the URL in question so that we can take a look: hello [at] whatruns.com. Thanks!
We were featured on Chrome Webstore a few weeks back and got a great response (12k+ active users) which helped us enormously in improving the accuracy and efficiency, and I'm sure HN and PH launch will be even more helpful in improving the product.
Our business model will be similar to that of BuiltWith's, i.e selling list of websites using a particular technology. For eg., list of websites using Drift chat (https://www.whatruns.com/technology/drift) will be a super-useful competitive intelligence for other live-chat start-ups.
Also, we are planning to introduce a predictive sales system which will suggest clients based on their technology adoption. For eg., if a company migrates to Magento, they are a potential client to Magento extension developers.
Most users like to know the full tech stack of a website. If there is a blog at blog.company.com and if it is using Intercom, it can be a useful data. I hope this makes sense.
Anyway, we will definitely address this concern and think about adding an option for subdomain separation.
1. WhatRuns detects fonts, Wordpress plugins and themes (tens of thousands of them).
2. Ability to follow sites (and know what techs websites started using/ditched).
3. Very lightweight compared to our counterparts, and arguably better UI ;)
4. More accurate data. BuiltWith can be very inaccurate as you might've already noticed. Wappalyzer is fairly accurate, but limited in technologies.
WhatRuns is trying to be the best of both worlds. :)
javascript:void(window.open('https://www.whatruns.com/website/'+window.location.hostname));The URL has to be publicly accessible from the Internet, right?
Addressing your question, all URLs once passed through WhatRuns will be publicly accessible. You will have to use the extension for new sites for now.