ctur

It’s fun to build things like this but if you want to nourish a user base you need to fully understand the landscape of similar tools and then explain your differentiating value. This is /particularly/ important for security related tools.

Specifically you should compare and contrast to tools like SOPS, Ansible Vault, pass, etc.

This is an unnecessary optimization, particularly for the article's use case (small files that are read immediately after being written). Just use /tmp. The linux buffer cache is more than performant enough for casual usage and, indeed, most heavy usage too. It's far too easy to clog up memory with forgotten files by defaulting to /dev/shm, for instance, and you potentially also take memory away from the rest of the system until the next reboot.

For the author's purposes, any benefit is just placebo.

There absolutely are times where /dev/shm is what you want, but it requires understanding nuances and tradeoffs (e.g. you are already thinking a lot about the memory management going on, including potentially swap).

Don't use -funroll-loops either.

Woohoo, one of the highlights of this time of year. I had to do mine from an eastbound flight over the pacific. This has become a fun tradition not just for me personally but for many friends, colleagues, and fellow HNers. Big props once again to wastl and his helper elves for making this!

I encourage anyone who gets value from this to donate to support it if they can. It is a passion project but nonetheless comes with real costs.

Architecture matters because while deep learning can conceivably fit a curve with a single, huge layer (in theory... Universal approximation theorem), the amount of compute and data needed to get there is prohibitive. Having a good architecture means the theoretical possibility of deep learning finding the right N dimensional curve becomes a practical reality.

Another thing about the architecture is we inherently bias it with the way we structure the data. For instance, take a dataset of (car) traffic patterns. If you only track the date as a feature, you miss that some events follow not just the day-of-year pattern but also holiday patterns. You could learn this with deep learning with enough data, but if we bake it into the dataset, you can build a model on it _much_ simpler and faster.

So, architecture matters. Data/feature representation matters.

But not all things you might do with a dotfile (or, more generally, per-user customization) are just replacing files. Things like cronjobs, brew installs, `defaults` in MacOS, etc. Viewing dotfile-based customization as strictly files to obliterate with pre-existing files is needlessly myopic.

For this broader problem, there are other more complete solutions that are more robust and flexible. Personally I like dotbot (https://github.com/anishathalye/dotbot) as a balance between power and simplicity, particularly when managing files across multiple OS homedirs (e.g. linux server, macos laptop).

Reread the story. The child wasn’t left in the car for an extended period (by a grandparent, not parent). The child had just been buckled into a car seat and the driver closed the door, walked around to the drivers side, and couldn’t get in.

Absolutely no indication of improper adult behavior.

We give away potatoes to trick or treaters on Halloween. They are immensely popular and we’ve become known as the potato house in our city’s Facebook groups. The weird delight on the faces of kids of all ages was hugely unexpected but surprisingly consistent.

Careful, your homedir has a CloudStorage folder and if you are using, say, Dropbox or Google Drive then that find will be incredibly slow (in addition to security software possibly slowing it down).

I find it very useful. I made a tool similar to mcfly (before knowing it existed) and use this workflow (`--here`) constantly. Also hostname context and shell session can be useful at times, too, to reconstruct something in the past.

https://github.com/chipturner/pxhist

While I doubt I'd quit my day job for it, over the past couple of years I've been poking at my own database-backed shell history. The key requirements for me were that it be extremely fast and that it support syncing across multiple systems.

The former is easy(ish); the latter is trickier since I didn't want to provide a hosted service but there aren't easily usable APIs like s3 that are "bring your own wallet" that could be used. So I punted and made it directory based and compatible with Dropbox and similar shared storage.

Being able to quickly search history, including tricks like 'show me the last 50 commands I ran in this directory that contained `git`' has been quite useful for my own workflows, and performance is quite fine on my ~400k history across multiple machines starting around 2011. (pxhist is able to import your history file so you can maintain that continuity)

https://github.com/chipturner/pxhist

What a great historical summary. Compression has moved on now but having grown up marveling at PKZip and maximizing usable space on very early computers, as well as compression in modems (v42bis ftw!), this field has always seemed magical.

These days it generally is better to prefer Zstandard to zlib/gzip for many reasons. And if you need seekable format, consider squashfs as a reasonable choice. These stand on the shoulders of the giants of zlib and zip but do indeed stand much higher in the modern world.

For those thinking about lessons to take from this, check out the EOL DR checklist: https://github.com/potatoqualitee/eol-dr

Mark is one of the world's top experts on practical MySQL performance at scale, having spent a huge amount of time optimizing MySQL at Google and Facebook. There's a question in this thread about whether this has real world impact... yes, if Mark noticed it, yes, yes it does. This will materially improve many common workloads for InnoDB.

This article seems like it is an AI summary of another document. It doesn’t offer anything new, just restating someone else’s analysis. And that analysis comes down to… storage demand will outpace SSDs price point enough that the demand will be met by both SSDs and traditional spinning platters.

Which isn’t surprising as even today there is demand for tape. The question is how long the demand will remain vaguely mainstream vs when it will become more niche. The 2028 estimate of storage being dominated by SSDs seems vaguely reasonable to me.

Pure’s bravado of staying zero hard drives will be sold after 2028 seems silly but inline with what a flash storage company would say. But from a directional standpoint it probably is right that many use cases will get further eroded by SSDs. One big challenge with hard drives is access speed (throughout and latency) compared to nvme, and hard drives being used more as cold and near line storage is definitely going to continue. Write once access never in many cases.

Sounds like hypnogogic hallucinations and/or the hypnogogic stare. Thomas Edison and Salvador Dali (an)used this state for creativity supposedly.

He was and worked on, among other things, Python while there. He's a trustworthy source.

In addition to the great advice in this thread, it’s worth being aware of something called “Rejection Sensitivity Dysphoria” (RDD). It isn’t a diagnosable condition according to the DSM-V but is often observed coincident with ADHD. Basically many forms of criticism or even observation can trigger a deep and intense fight-or-flight response due to challenges regulating emotions.

Therapy and medication can help. If you think this might be in play it is worth talking to a mental health professional.

https://my.clevelandclinic.org/health/diseases/24099-rejecti...

Unless the recipient of whatever you are compressing absolutely requires gzip, you should not use gzip or pigz.

Instead you should use zstd as it compresses faster, decompresses faster, and yields smaller files. It also supports parallelism (via “-T”) which supplants the pigz use case. There literally are no trade-offs; it is better in every objective way.

In 2023, friends don’t let friends use gzip.

I consider myself very fortunate to have had the opportunity to work with Yann. He is amazingly humble for someone who has made something so impactful. One thing also that sometimes gets lost is wins as substantial as Zstandard not only save the big tech companies money but also reduce energy costs overall as well as local costs such as bandwidth on mobile devices (which affects individuals as much as it does companies).

To be fair, for every one of these sub-30 wunderkinds there are dozens of 50+ year olds hoping to make a fortune by propping them up… who often avoid such public derision.

I find the iOS app BeOrg great for mobile access to my org mode inbox. It works well as not only a place to write quick notes but also to, say, "share to" links from Safari and other apps to create inbox entries as well to quickly capture URLs to view later.

I don't think you are anywhere near that line. Using human beings as part of a punchline is pretty much the opposite of funny. Trying to rub something in the nose of people who were laid off is rather tacky, too.

Being able to ask questions well, be it in public online or in the workplace, is a skill worth developing (as is answering such questions). I compiled a shorter and hopefully more actionable list of similar guidelines based on a few different jobs in the tech industry and public sources, including this one.

https://github.com/chipturner/asking-questions

My general flow is “do it, teach it, automate it” where the middle step is explaining it to someone else who then also does it then (hopefully) helps, or at least consults, on the automation. This way you can iterate a bit before codifying it and help flush out any bad ideas or assumptions.

The lz4 developers are the zstd developers :) In this case I suspect the IO reductions on the output are what matter -- while lz4 is faster, it also produces more output (of course) which you then have to write to disk. This can make the wall time take longer... basically you become IO bound on output, be it network or disk.

zstd also can tune itself and adjust ratios to saturate output bandwidth, which is pretty cool (--adapt).

Pretty suspicious; certainly dubious, potentially just a scam. Collecting sensitive information, no actual information on the site about who is behind it (bonus: it's a one page site whose backend is a third party), submitted to HN by a new account, hosted on a domain created yesterday, (ab) use of the Y Combinator name without any actual evidence of affiliation, and with logos of companies that aren't even links.

The Decoder Ring podcast did an episode[1] recently about the supposed death of the mall. It's a fun listen and covers a lot of the history and evolution of the mall as a concept and how it shaped, and was shaped by, transportation changes in the US. It also goes into the social importance they have had over the years and how that has evolved.

[1] - https://slate.com/podcasts/decoder-ring/2022/07/is-the-shopp...

In many ways you can think of large, long-living tech companies not unlike old cities like, say, London or Paris. The buildings and roads you see are built on top of older buildings and ruins. The streets are weirdly shaped and intersect at odd angles because they were made hundreds of years before and adapted over time as needs evolved. There are catacombs underneath sidewalks and no one genuinely understands it all nor does a single reference exist that explains it.

Literally everything predates everyone who lives there. Generations and generations of original designers, architects, and laborers have arrived, plied their trade, and moved away. There are people who are experts in certain parts, and who can build a new skyscraper at any given spot, but it is just layering and organic growth.

The emergent complexity of centuries of being lived in and adapted belies easy understanding.

Large tech companies are similar. You just can't understand how "it" all works. If you were to build it from scratch, perhaps you could, because it would be simpler and clearer, but nothing was made with the current state in mind. It evolved and adapted over time.

So reading this, I am not surprised. I think you'd get the same answer about many other aspects of data, code, system history, etc at any other 10+ yr old tech giant.

It feels obligatory to reference Technology Connection in a thread about heat pumps. TC is a super fun YouTube channel that digs j to everyday technology and has had a number of episodes on heat pumps and modern thermodynamic management challenges and technologies.

https://youtu.be/7J52mDjZzto

What? No, this is incredibly short-sighted. It overlooks the fact that much grid energy is renewable already and will become more so in the future. Go with electric sources for cooking and heating (home and water), then let the grid handle being renewable.

Heat pumps are amazing technology and moving heat, vs generating it, is incredibly efficient and clean.

Do not burn fissile fuels in your home.