mootothemax

>The UK census seems as good a measure as any.

Agreed, and it's a reasonable first reach for residential counts.

Problem is, it was last taken during 2021 - at the height of covid - and their estimates have been paying the price of it even since. Anything that touches on workday or employment they actively warn you away from, nowadays.

But then there's the elephant in the room: what if you're looking at somewhere like the City of London (the square mile one) where residential population is somewhere between 8,500 and 9,000 people?

EDIT: it slipped by me!

There is no single UK-wide census. NRS do Scotland's, last in 2022. NISRA handle Northern Ireland's census, last in 2021. And the ONS for England and Wales, last in 2021.

> town

That's where the problem lies. I spend too much of my life screaming into the void about UK addresses and what is and isn't a town :)

It's just about possible to create dataset of towns, cities, villages from mixing and matching and cutting up other geographies from the ONS (e.g. their Parished and Unparished data (though Unparished is its own nightmare), then the ONS Built-Up Areas dataset but... that one cuts out all the non-built parts so Milton Keynes is bunch of big blobs around the residential areas).

But if you frankenstein it, you lose all of the formal connections that define other links, and if you try to cheat via geospatial join, you will link police forces to areas they don't actually serve.

For Swindon, you're kinda stuck glueing together (going from memory here - please correct me if I'm off) the Parishes of Central Swindon, South Swindon and West Swindon. But these will overlap with some villages so it's not a true count.

And that's a natural moment to lead on to the next critical point: how're we measuring population? (I always get that Father Ted speech scene in my head thinking about this, start speaking - then twelve hours later "And now, we move on to liars")

For what it’s worth, once I figured out I could say something like “I’m going to buy this, would you like to call it my birthday present?” At pretty much well any point in the year, I found that the people around me really appreciated the chance to show care AND see me using something I like which they got for me.

Thanks for taking me in the spirit I intended; I wrote more aggressively than I’d intended, on reflection.

You’re not a dick and you weren’t being a dick, I totally agree about writing on the internet - it’s tricky to feel and to tell, and I imagine I must be insufferable at times.

I can see your point now and I think we might agree on more than either of us first thought. I can see how much you care for your family in every word you’ve written, and I’d be a bit surprised if one of your kids was at home, told you it’s their birthday, and your reaction would be to laugh about it, and then carry on with your day as normal with no further reaction or acknowledgement of it.

So - Ii’s not about remembering, or about the card or presents or cake: it’s what you do with the information in front of you. From the sounds of it you’re demonstrating your care brilliantly every day.

(I have the same type of brain wiring you’ve described, incidentally. My extremely accepting wife would ask me to buy her some flowers, nothing showy, just something to brighten up the house and remind her of me. And I’d hear that as “she wants me to remember to buy her flowers sometime” and file it away. And several years later we figured out that gap in our communications and nowadays, when it’s important enough to her that she asks me to buy her flowers - I buy the flowers. Turns out that that was all it took to demonstrate that I care.)

I also wanted to add: wow, as a parent that sounds rough. You’re doing better for her than you think you are.

If one of your kids said, hey, it’s ok, I love you and your chaotic mind, can you please buy me a card if or when you remember - and it pops into your mind after 10 or 20 or 100 days, or even 5 years later - would you buy them a card or not?

Honestly - and I hope this didn’t happen to you - it takes a surprising level of indifference for kids to pick up a “this is wrong” signal.

I’m not going to speak for anyone else, but if I forget a thing and “fixing” it is as simple as saying sorry and doing it the next day, I’ll do it. And I will keep doing it on a loop if need be _because I care_.

And that’s often the easy sign for this stuff. Did your parents forget and _not even try_ to show that they care?

It’s the sheer indifferent callousness that stands out starkest to me. It doesn’t take much, I’ve found, for people to state outright “you’re wrong to expect me to care about you.”

Cool, you do you.

Out of interest, why did you say “could forget” and not “did forget,” and why did you feel the need to reply to me if hypothetical?

Oh hey! I’m the same age as you, and while my kids are a bit older, they’re neurodivergent too. You didn’t ask for advice, and I won’t

I will say that my eldest, and otherwise wonderful and engaging daughter is having a hell of a time at her school right now, and schools and SEN provision and, honestly, competence, are highly variable, and I’ve been surprised at how large the differences can be.

I know someone whose parents didn’t forget their birthday, but consciously chose to downplay it because All Saints’ Day was the following day.

That is, they chose to make a bigger deal over celebrating the dead, than they did their living child.

Good grief. I read your comment and … the levels of parental neglect wrapped up in your single story are overwhelming me.

I’m sure you don’t need validation from random internet strangers, but as one with a similar enough childhood, I wanted to share that it’s a thoroughly shit club to be in.

I recently created a planning applications scraper for every council/local planning authority in the UK, very much with Claude's help. As you can no doubt imagine, it's the type of task where the mechanics are far less important than everything surrounding them.

Getting things like this to work well, for me, often boils down to finding out the things I don't know that I don't know in as little time as possible. I worked with Claude to create a basic debugging front-end as part of that - for my own use, inspecting the data, looking for patterns - when, largely on a whim, I asked Claude to create an MCP interface on top: last run status, what's looking good, whats not, where are the coverage gaps in the individual data points.

And I added the MVP to Claude and would check in regularly to ask how things were going.

And then - I'd ask for some nuance about why something doesn't look quite right. Or what happened to councils in general last night? And - out of nowhere, not being prompted or asked - Claude would query the MCP alongside looking at the code. Immediate production-grade insight while I'm debugging from the comfort of my chair.

And I thought "Oh."

And "this is really quite good," because the key detail is, I think, if an MCP is good and works well, and relevant to what you're doing every day at that point in time, it's yet another addition to your toolbox.

And that got me thinking about how I can do the same for my 9-5 users.

Is this resident population or ambient?

I've been building a UK-wide 24-hour population dataset lately, and it's really opened my eyes to how much population data is best-guess-estimate - including census figures - and quite how difficult it is to say with any certainly how many people "touched" a location on any given day.

Interesting. How do they tell the difference between legitimate and forged ip owner records?

> The anubis author has stated they recognize it's an arms race, but PoW scales.

The scraper wars are largely between script kiddies and people with both deep intimate networking and DOM knowledge. Yes greyhairs, I’m looking at you.

The problem is, you can’t PoW every page load and resource request because the user experience will suck and people will run away. And that window - the gap between what people will tolerate vs draconian enforcement - is exactly what the scrapers exploit.

And looking at the PoW options out there - I’ve seen at least one PoW WAF (honestly can’t remember if azure or amazon) have their PoW boil down to repeated trigonometric functions, ie very optimisable.

It’s a neat concept, but the answer and future to my eyes look bleak.

Can any LLM give you the rough pixel coordinates of an item it identifies in an image?

I found that while Claude, GPT etc could describe an image, there was no way to link the description back to specific pixels in the image itself. Not even to a bounding box or segment.

In your first comment, replace “until today” with “since then” and you’re good!

“Until today” is one of those English phrases that is particularly unfair on non-native speakers. You know “until” and you know “today” and so it’s completely natural to combine them in the way you did.

But as ever, English is dumb and annoying and hard work, all at the same time.

If you haven’t investigated storing in parquet format - and it doesn’t break other consumers that need your jsonl formatted files - it could be worth trialling for your use case. You’ll see vastly smaller file sizes (even more so if you use zstd compression), and querying time will shoot up.

Usual caveats apply, but as a general rule it’s held up well for me. Only downside is that inspecting the results moves from vi on the output file to duckdb and a select * from.

It’s great compression: Y sometimes a vowel, sometimes a consonant.

And while not encoded on a keyboard, it still blows my mind that English has a crazy number of past tenses - and a such a bad hack of a future tense that it’s hard to classify as such.

Linguistics is fun. The accents are alright.

Speaking from the scraper’s perspective, I like proof of work; a ten year old 96-core server will cost a couple of quid to run for a few hours and will grab an absurd number of pages thanks to the access granted by repeatedly solving proofs of work. Small slick codebases too!

Exactly. I’m constantly amazed at how little you actually need to bypass CF, Amazon, Azure WAFs and so on (Incapsula springs to mind too). When you look at the code you’ve come up with, it’s actually quite small and compact.

More to the point, these systems actually help scraping because proof of work unlocks essentially unlimited scraping, in my experience.

That said - from my experience on the other side, sure you can’t stop people like me or you, but you can stop 99% of the others. That’s more than worth it operationally.

I suspect that introducing the calibration concept might be a case of too much too soon for some people.

As far as I understand it, the various probability matrices boil down to: what token has the highest likelihood of coming next, given this set of input tokens. Which then all gets chucked away and rebuilt when the most likely token is appended to the input set.

Objective assessment of internal state - again, to my non-expert eye - doesn’t appear to have any way to surface to me.

Big-if my rough working understand is more or less correct - your calibration point makes a lot of sense to me. I’m not sure that it would make sense to someone who eg considers some form of active thinking process that is intellectualising about whether to output this or that token.

The irony for me being that when I was first learning Polish and looking for any and all mnemonics - “ah, that word is the number nine, and that one is ten because it has an s in the middle and that’s next to t for ten in the alphabet”-levels of desperate - the false etymology helped me set word, słowo, in my head, and the rather delightful dosłownie, literally / to the word, has remained ever since.

(tho while on the subject, it’s hard to beat wieloryb as a wonder that I don’t want to know the true etymology of ever because if there’s even a chance that the word for whale derived from the words great as-in-size + fish, I want to hang on to it forever)

I think I may have fallen victim to a GP midflight edit - I agree with you fwiw, it’s a stone cold fact.

> Germans are Nemci in Russian as well

I wanted to check; are you implying that Russian is not a Slavic language?

Recommend looking up the pronunciation of that there afon :)

Makes sense given Welsh’s evolution from Britannic. Much to my shame, I only started visiting Wales in later life, and there’s really something in the language that grabs me quite deeply. Once I’ve got my Polish down to pat, I tell myself.

It’s lovely isn’t it? There’re a good few of these things around: notably Torpenhow Hill (which killjoys dispute); and ones like Pendle Hill (which they don’t).

If you can find a way to combine this with local population to end up at pence per litre per thousand population, I bet you’d uncover some fun trends. Bet it’d also get interesting if combined with population within an X min drive too.

Tho really need some car population per road segment stats to drive the most out of it IMO.