HNHacker News
TopNewBestAskShowJobs

willvarfar

19,398 karma · joined July 1, 2010

willvarfar.at.hn

my old blog (dormant): https://williame.github.io/

submissionscomments
willvarfar··on The biggest heat pumps
A heatpump needs power to operate, but will generate at least 3x more heat than the electricity to power it.

The article describes how there will be a water battery.

So it can be thought of as a part of a bigger countrywide or europe-wide plan and grid?

willvarfar··on The biggest heat pumps
In the nordics we love heat-pumps! Something like 70% of houses are heated by heat-pumps, and 90% of apartment buildings are heated by district heating and that is often generated by huge heatpumps.

Apparently 95% of new heating installations in Swedish houses are heat-pumps these days: https://publications.jrc.ec.europa.eu/repository/handle/JRC1...

Heatpumps have been heating nordic homes for decades. Even in the countryside where many houses have small woodland attached, people I know have moved to heatpumps for convenience and because its affordable.

PS: shoutout to to the JRC, found their reports when doing a super quick dig for stats. Those reports were super easy to read :D

willvarfar··on Shai-Hulud compromised a dev machine and raided GitHub org access: a post-mortem
The amount of browser extension authors who have talked openly about being approached to sell their extension or insert malicious code is many, and presumably many others have taken the money and not told us about it. It seems likely there are IDE extensions doing or going to do the same thing...
willvarfar··on The Tor Project is switching to Rust
1 = no privacy from relay

2 = risk of collusion between relays

3 = goldilocks default

4 = ... actually, you have more attack surface and you are more susceptible to fingerprinting because everybody else is using 3, so you're timings etc help identify you

So the default is 3 and nobody ought change it! Use 3 like everybody else.

The exception is .onion sites. TOR actually deliberately defaults to 6 hops when accessing .oninon sites - 3 to protect you and 3 to project the site.

willvarfar··on The Tor Project is switching to Rust
Then the single relay knows both who you are (your IP) and where you are going. This offers no anonymity against the relay itself.

3 relays is the goldilocks number for speed vs privacy. Using less is not a tradeoff the usual user of Tor should make.

willvarfar··on Meta shuts down global accounts linked to abortion advice and queer content
All the american action movies we watched in the 80s and 90s with Rambo and Schwarzenegger etc were all about violence? Most American movies seem to work guns into them somehow too.
willvarfar··on A “frozen” dictionary for Python
The values in tuples cannot change. The values that keys point to in a frozen dict can?

But yeah I'd be in favour of something that looked a lot like a named tuple but with mutable values and supporting [name] access too. And of course some nice syntactic sugar rather like dicts and sets have with curly brackets today.

willvarfar··on Just 0.001% hold 3 times the wealth of poorest half of humanity, report finds
Is there a global movement to tax business where the customer conducts it rather than where the company chooses to base itself?
willvarfar··on Just 0.001% hold 3 times the wealth of poorest half of humanity, report finds
When you say "Americans" do you mean people in the USA? Or the continents?

Beyond the wealth inequality within countries, there is the wealth inequalities between countries which drives migration of people or jobs etc. Even the most closed off and isolationist country in the world - probably North Korea? - is not immune to relative wealth on the global stage.

willvarfar··on When would you ever want bubblesort? (2023)
your random number function might return the same number multiple times? So to choose k random but unique numbers you may have to call the random number function more than k times?

Of course my intuition would be that you can do a random shuffle and then take the first k, which is O(N). So I might be misunderstanding.

willvarfar··on How Much Wealth an AI Stock Market Crash Could Destroy
Does it materially impact people's pensions?

Globally, pension funds hold approximately $5–7 trillion directly in the top US tech stocks (the "Magnificent Seven" and adjacent AI infrastructure).

Also in the last couple of years many pension funds have moved money into Private Equity and Private Credit to chase higher returns and they're the backstop for all the off-books AI datacenter buildout debt?

willvarfar··on So you want to speak at software conferences?
The (in)famous astronomer Tycho Brahe died from a bladder infection after politeness prevented him leaving the audience on such an occasion.

https://en.wikipedia.org/wiki/Tycho_Brahe#Illness,_death,_an...

willvarfar··on Richard Stallman on ChatGPT
Are you saying that LLMs _do_ have a model of the human mind in their weights?

Imagine you use an ARIMA model to forecast demand for your business or the economy or whatever. It's easy to say it doesn't have a 'world model' in the sense that it doesn't predict things that are obvious only if you understand what the variables _mean_ implicitly. But in what way is it different from an LLM?

I think Stallman is in the same camp as Sutton https://www.dwarkesh.com/p/richard-sutton

willvarfar··on Microsoft increases Office 365 and Microsoft 365 license prices
I suspect my country is the same. I expect most countries are the same.

Is there basically any expectation that the US government doesn't know the internal goals and thoughts of all other governments just by reading the cloud?

willvarfar··on The Forgotten Roman Ruins of the ‘Pompeii of the Middle East’
When holidaying in Italy I had the luck to pick the day trip one day. Really glad I found Paestum (an ancient Greek city): it is every bit as captivating as Pompeii imo.
willvarfar··on Most technical problems are people problems
> I am pretty sure there would have been a small group (or at least one) of tech people in there who knew all of this and tried to get it fixed, but were blocked at every level. No idea - but suspect

I recall there was a whistleblower Richard Roll who said that engineering did know of the bugs and flaws

willvarfar··on Functional Quadtrees
A general quadtree implementation question that puzzled me when I was implementing it myself for hobby games was: do you store a rectangle in the smallest node that completely contains it?

Most code that I saw that used quadtrees were treating things as points and storing them only at the lowest level.

I also made mine auto-divide by counting items that are entirely in a quadrant as they are added to the node, with allocate and split triggered if a count went above a certain threshold.

Anything novel or oopsie?

willvarfar··on Accepting US car standards would risk European lives
I would guess it is a tariff thing rather than NATO. Is anyone in Europe really believing the USA still has our back?
willvarfar··on Accepting US car standards would risk European lives
The article says road deaths in USA are up 30% over last 15 years and links to https://www.transportation.gov/sites/dot.gov/files/2024-02/2.... That doc talks a lot about initiatives but what is the normal American's sense of what's going on on the street?
willvarfar··on Anti-patterns while working with LLMs
I have had both frustration and joy working with AI. Just some anecdata:

I have found it is far better at understanding - and, with prodding, determining the root causes of bugs - big sprawling codebases than it is at writing anything in even simple code bases.

Recently I asked an AI to compare and contrast two implementations of the same API written in different languages to find differences, and it found some very subtle things and impressed me. It got a lot wrong, but that was because one of the implementations had lots of comments that it took at face value. I then wrote a rough spec of what the API should do and it compared the implementations to the API and found more problems. Was a learning experience for me writing specs too.

I repeated the exercise of comparing two implementations to track down a nasty one-line bug in a objc -> swift port. I wasn't familiar with the codebase, or even remember much about those languages, so it was a big boon and I didn't have to track down people who owned code until I was fairly sure that the bug had been found.

Also recently I asked an AI to compare two sets of parquet files and it did sensible things like downloading just bits of them and inspecting metadata and ended up recommending that I change some of the settings when authoring one of the sets of parquet files to dramatically improve compression. It needed esc and prodding at the halfway point but still it got there. Was great to watch.

And finally I've asked an AI a detailed question about database internals and vectorising predicates and it got talking about 'filter masks' and then, in the middle of the explanation, inserted an image to illustrate. Of 'filter masks' in the PPE sense. Hilariously wrong!

willvarfar··on [dead]
OLAP and batch analytics is just getting so much more common. Thinking back 30 years ago, column stores were rare and SQL was all about OLTP CRUD. It still does all that, but it's also become mainstream as the data science and analytics becomes ever more routine and approachable too.

And the dialects of the language itself, SQL keeps getting more relaxed and interoperable and forgiving. With WITH and CTEs and things it keeps getting easier and cleaner to express things, so it's going steadily in the right direction. There are still a few slight differences in the syntax for window functions between bigquery and duckdb, for an example I fight often, but they are all a lot closer to each other today than they used to be back 30 years ago when you had to use SQL differently and construct complicated queries differently just to run on Oracle vs MySQL vs Postgres.

willvarfar··on Python is not a great language for data science
I think you can query and join polars dataframes if you call to_arrow() on them too.
willvarfar··on Python is not a great language for data science
Duckdb can see and manipulate dataframes too. Duckdb has it's own storage, but other table storage - e.g. the parquet files I mentioned or even csv files or even dataframes from pandas and polars - are first-class citizens. Duckdb lets you query them quickly and expressively.
willvarfar··on Python is not a great language for data science
My experience was that data science was doable but clunky and ugly with pandas. It got slightly better with polars. Only really slightly better. Then, for me at least, it jumped lightyears ahead with duckdb.

These days I run some big query on an OLAP database and download the results to parquet stored on the local disk of a cloud notebook VM and then mine it to bits with duckdb reading straight from these parquet files.

The notebooks end up with very clear SQL queries and results (most notebook servers support SQL cells with highlighting and completion etc), and small pockets of python cells for doing those corner case things that an imperative language makes easier.

So when I get to the bottom of the article where it shows the difference between Python and R, I'm screaming "wouldn't that look better in SQL?!" :)

willvarfar··on Using an Array of Needles to Create Solid Knitted Shapes
Are there croquet machines yet? Googling is really confusing with lots of forum people saying there aren't any true ones, and lots of webshops claiming to sell them.
willvarfar··on Trade Chaos Causes Businesses to Rethink Their Relationship with the U.S.
Trust takes a lifetime to build but just a moment to lose. The rest of of the world is not going to risk the deep dependency vassal state relationship that it had with the USA. Things don't reset like that.
willvarfar··on Trade Chaos Causes Businesses to Rethink Their Relationship with the U.S.
taking an example from the article, the USA currently produces 0.2% of coffee it consumes domestically (Hawaii and Puerto Rico).

Could coffee be grown in reasonable quantities inside the USA? I find some mention of very expensive high-end 'boutique' coffee grown in California but it is not generally a crop that grows well in the continental USA.

(until global warming reduces the chances of frost in Florida perhaps?)

Another example from the article was a tea grower. Again, niche growing is limited to just some regions of the USA, with less than 0.1% of consumption domestically produced.

And of course with these products they have distinctive tastes that reflect where they were grown, so tea from California is distinctive tasting and not a direct substitute for tea from Japan from the article.

The growers in the article had been heavily disrupted by tariffs.

willvarfar··on µcad: New open source programming language that can generate 2D sketches and 3D
I've been searching for a CAD language that supports iteration in a good way. I've been designing a self-build house and it occurs to me that, once I have the walls etc, filling in the details for the manufacture of those walls could be done by a programming routine. Such a routine could work out how many studs are needed and their placement etc and generate the kind of cut-throughs where you can see the construction inside the wall that you sometimes see in construction books. Anyway, without for-loops that kind of thing is really difficult and I've given up.
willvarfar··on 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark
Often, at the medium and large sized companies its not 'risk aversion', its resume padding.

Architects want to build big impressive systems that justify their position and managers want that too because success is judged by size of systems and number of staff under management, not its efficiency; its all about perverse incentives.

This is just a tax the scientists trying to use whatever the company settles on have to pay every time they wait for queries to run.

These days scientists can just suck down a copy of a bunch of data to their laptop or a cheap cloud VM and do their crunching 'locally' there. The company data swamp is just something they have to interface with occasionally.

Of course things go pear-shaped if they get detected, so don't tell anyone :D

willvarfar··on 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark
I often crunch 'biggish data' on a single node using duckdb (because I love using the modern style of painless and efficient SQL engines).

I don't use delta or iceberg (because I haven't needed to; I'm describing what I do, not what you can do :)), but rather just iterate over the underlying parquet files using filename listing or wildcarding. I often run queries on BigQuery and suck down the results to a bunch of ~1GB local parquet files - way bigger than RAM - that I can then mine in duckdb using wildcarding. Works great!

I'm in a world where I get into the weeds of 'this kind of aggregation works much faster on Bigquery than duckdb, or vice versa, so I'll split my job into this part of sql running on Bigquery then feeding into this part running in duckdb'. It's the fun end of data engineering.

← PreviousPage 2 of 34Next →