6,553 karma · joined July 6, 2009
* Sometimes there's a pattern to the country. Oftentimes, not.
* User-agent, rotated between common, valid, current web browsers.
* Other headers, sec-*, accept, etc, generally valid and rotating.
* Bots will load the site to saturation in a denial of wallet attack.
The only thing that's specific is: * urls have a pattern.
* it's obviously invalid traffic.
(non-bot traffic on my sites does not go from 0 to 200r/sec on the search interface in seconds. It does not go away that fast either)The crawlers get stuck in a loop requesting the dataset listing page with every. single. combination. of. facets. At essentially as fast as it can be pumped out or blocked.
Had one bot super interested in a single organization to the tune of 1000 r/min for 24+ hours. Realistic rotating user agents, realistic sec-* headers, no ip address seen more than a couple times in 10 minutes. The only commonality was the route.
So we had a goal to make the train do interesting behaviors, like cover the entire track autonomously, pushing the switches itself. The biggest run I remember was a 4 bit counter.
Kid is now entering his last year of CS + Maths degree, so I guess it checks out.
If you redline at 20 searches a sec, and put in 4 more workers, suddenly you’re serving 100r/sec to the bots, paying 5x for it, and your users are still seeing shit qos. I've seen multiple cores of nginx saturated just dealing with one dos/crawl run on a somewhat high profile site.
At one point, other copies were both $300 plus on ebay.
Sadly the black one has an intermittent issue with some of the keys where hits are doubled, but the beige one is still a daily driver.
Haven't spent much time in Belgium, but I was disappointed in the average quality of the bread in small bakeries in France. There was one awesome organic bakery in Paris near the catacombs tour, and one hypermarche (forget where) that somehow had an absolutely incredible rustic baguette. But the little shops in towns, eh. And most of the hypermarches. Germany is also kind of average, the Reve bakeries are reliably on the good side of middling, but nothing that I'd consider incredible.
Assume you have 100g of flour at equilibrium 20% ambient humidity, and the same 100g of flour at 80% humidity.
I don’t know how different the effective moisture content would be, but measuring the weight of the flour to the gram seems like you’re including the moisture in the weight of the flour. Maybe one packs denser on a scoop. I don’t know. But I don’t necessarily think it’s more accurate.
On the other hand, it’s really easy to just pour in 540g of flour, mix in a shy tablespoon of salt, 280g of water, and a good glop of starter. Far easier than trying to get consistent scoops or measure to the meniscus in a liquid measured.
* fastapi + duckdb + parquet for the backend for a relatively high profile website
* wasm duckdb + react for a few visualization websites
* yaml driven ETL from lots of sources, principally ugly spreadsheets, into usable data. More T than E or L really
Mostly current valid user agents, lots of ip addresses, but the traffic patterns are not organic. I’m not clear if it’s bad ai scraping or dos, but at some level it’s indistinguishable.
How big is that fraction? In my experience being around a bunch of home schoolers (and adjacent, and school system), some of whom were more in or out of unschooling, I think it's small enough that it should be a rarely considered option.
There are some kids where it will work _really_ well. I've interacted with a couple. There are a lot of kids where it really doesn't work, especially in a distraction rich environment.
Next best was cheap tokaij furmint, distilled once and then mixed back into some of the undistilled wine. Basically the same thing as pineau de charante, but Hungarian and on the kitchen table.