389 karma · joined March 19, 2012
Will be publishing a report on www.theweb.report soon - if you're interested.
That's a seed of about 13 million domains - pretty sure WordPress would be dominating the even longer tail.
( Also worth noting it's sooo easy to detect a site is WordPress, it screams it across every signal we gather, where-as some sites are just well made and have limited information leaks ).
Custom python crawler getting 240 sites a second crawled and classified. ( homepages and minimal probes, no headless browsers )
Be interested to test some false positives, if you have a URL I'll tell you what I see :)
I'm seeing about 6.6% block rate, but that does climb over time.
I've been doing the same bit wider scope, for the whole Crux list, pruned to apex domains, and looking for CMS signals - how's your throughput?
I'm not doing any headless browser stuff, or many requests, so hyper optimised for speed.
I do grab robots.txt - didn't really see much in llms.txt or humans.txt in the wild, does yours?
have found Shopify's AI implementation to be sane and really useful ( building flows and surfacing documentation correctly ).
I’m doing some really powerful stuff with the custom CLI commands, highly recommend the YouTube video on it. Makes bulk import / api enrichment and other things a breeze.
It’s licensed, and I’m happy to pay for a professional sustainable solution.
Replacing my iPhone was a nothing burger of choice, on paper the iPhone 15 pro was the best feature set for value vs buying a new iPhone 17, but Apple know that so don't sell the older models directly when the new models come out.
There's really limited impactful innovation when you get into the details.
Would show that off and zoom into them when selecting them from the list view!
Would be nice to have a definition of 'Turn-up' and 'Curtailment' in that FAQ for us non grid savvy visitors.
Works great though, well done.
Slop science papers is just what the world needs.
Block fields like ACF flexible layouts. Modern and fast with total control of the frontend.
Has been fun to switch and fast.
Or if you don’t need so many records we had good success with SearchWP up to about 90k records with a few thousand taxonomy tags.
How can you expect any developer to devote time to writing a plugin if the dictatorship of Matt can rug pull it at any time.
List of crawlers for those who now want to block: https://platform.openai.com/docs/bots
Are we going to end up with AI training on AI generated datasets.