adamcharnock

I also had good experience setting up PIs with read-only root devices. All data needs to be sent off-device (or at least onto external storage), but it wasn’t too tricky and should avoid the usual SD-card issues.

Additional thought: How competitive are the two markets? Does the USA have fewer second-tier cloud providers, and the market is just dominated by AWS/Google/Microsoft?

EU customers can naturally use those services, but there is a demand for local/EU-sovereign providers. Perhaps this demand provides enough fuel to maintain a robust & competitive ecosystem of smaller cloud providers?

I only have partial information here - and that is that I’ve heard connectivity is generally much cheaper in the EU. And I can see why too: denser population, smaller distances between population centres, close to other continents.

But I’d be interested in any other reasons people can give.

A quick google didn’t reveal much about pgstream’s delivery semantics. Do these webhooks get called at most once, or at least once? I hope the latter, but in absence of information I’d guess it is the former.

Thank you for the reply!

> it is about creating a searchable index of the web scale that has certain quality level that is simply impossible to do anymore for many reasons that would require another article to explain

I am both happy to take your word for it, and also very interested to know more. If you were to write that article then I would love to read it.

> The Google Search Index is a unique and irreplaceable resource within the digital ecosystem. Mandating fair access to it or treating it as an essential facility could address the core issues...

The article estimates the Google Search Index at 12.5PB. If Kagi thinks that is a big enough moat to be the primary target then, well, I suppose they should know. But I'm also skeptical. You could fit that on about 50 Hetzner SX295, so about $20k/month. Plus the cost of gathering the data. It is surely a huge resource.

But weighed against the combination of Google Search + AdWords + Android + YouTube + Chrome, all in a single company? To me a 12.5PB search index feels like small change in comparison.

NB: Happy Kagi-paying customer here.

A anecdote about Pavel in a HN comment from few weeks ago (not that I have a stance anything in this situation):

https://news.ycombinator.com/item?id=41149755

Same experience in a Costa Rica hostel! Except in that case my skin just gently tingled where the water touched it. It was a case of, “I know that sensation, and I certainly shouldn’t be feeling it in the shower”

In my experience, dedicated hardware has provided a baseline real-world 2x speed up over cloud instances (presumably down to no contention, local nvme). So that would be 8 hetzner instances.

I managed to squeeze out a 5x speed up total (see my other comment). In which case that would mean 3-4 instances.

Plus with shorter build times you may then find that having a builds queued up is acceptable.

Very much this. I’ve overseen this process for one of my clients and we’ve seen build + deploy times go from 5-10 minutes down to around 1-2m. This was down to increased performance, improved caching, and being about to cut out some minor workflow setup steps.

So 10x cheaper and 5x the performance.

Still using GitHub Actions, but now just using self-hosted runners.

Bare-metal Kubernetes deployment specialist

  Location: EU / Global
  Remote: Yes
  Willing to relocate: No
  Email: adam@adamcharnock.com

I specialize in bare-metal Kubernetes deployments for startups and SMEs, as well as any subsequent support requirements. This covers everything from requirements gathering, to initial setup, configuration, and ongoing maintenance.

I've been working with Kubernetes for 10 years, and in the tech industry for 18 years. Previous clients include Twitter (X) and the British Royal Navy.

Spilo has, in my experience, been poorly maintained for a while. There has been some slow progress, but it doesn’t seem to be anyone’s priority right now (happy to be corrected).

I was running my own Spilo builds for a while, which was hit-and-miss. For my new (Kubernetes bare metal) cluster deployment I’ve moved over to Stackgres. I also evaluated CNPG (promising, but still early-ish days), as well as one other IIRC.

I found Stackgres to work most reliably. And it solves the biggest pain with Spilo, which is building an image with the required PG extensions. Stakgres instead has its own repository of extensions that it can install from, which is a huge help.

I think that is putting it rather strongly. I use UUID7 (for its better clustering) for anything that both, 1) may end up having a lot of rows (1M+), and 2) where leaking the creation time in the UUID is acceptable.

Otherwise I use UUID4.

I agree with your concerns, but wow do I miss the 1990s/2000s era of the internet. Things were so much more fun then.

Perhaps a 'mirror to S3-compatible store' feature would address these concerns though. Sure it is extra cost, but it would be a nice de-risking option for early adopters.

While I don’t recall running into issues either, I can certainly see that a new index could cause the query planner to make a different decision. And that decision could - in some cases - end up being worse that the previous behaviour.

I definitely have seen the query planing make some peculiar choices in the past.

I once (accidentally) tried this when travelling around Australia many years ago. The gate changed at the last minute and I didn’t notice, so tried to board the wrong place.

As soon as they scanned my boarding pass all hell broke loose. Flashing red lights, sirens going off, the whole works. They were very nice about it, but yeah, that particular system made it impossible not to notice.

I see so many videos of (predominantly USA) carpenters using table saws without even the bare minimum of safety features (even just a riving knife, for example). Is there no way to just enforce basic low-cost low-effort safety features rather than just jumping all the way to a very costly commercial saw-stop-like solution?

I’m wondering why this is, and I can think of a few potential factors (broad generalisations follow):

- The US is a low trust society

- European culture typically places less emphasis on workplace friendships compared to the US.

- As an another commenter said, the size of US cities. But also the difficulty & cost of getting around

- There is something about the American demeanour that is surface-friendly, but also seems to discourage deeper/emotional connection. Whereas many European cultures can seem less smiley-friendly, but also more honest and genuine

- I feel like the very strong Protestant work ethic in the US may play a factor. When productivity is so important I feel that must detract from other aspects of life (stress, free time, relocations)

At the time I was working with either GPT 3.5 or 4, and – looking at my old code - I was limiting myself to around 14k tokens.

> Were you happy with the results?

It was somewhat ok. I was testing the system from a (copyrighted) psychiatry textbook book and getting feedback on the output from a psychotherapist. The idea was to provide a tool to help therapists prep for a session, rather than help patients directly.

As usual it was somewhat helpful but a little too vague sometimes, or missed important specific information for situation.

It is possible that it could be improved with a larger context window, having more data to select, or different prompting. But the frequent response was along the lines of, "this is good advice, but it just doesn't drill down enough".

Ultimately we found that GPT3.5/4 could produce responses that matched or exceeded our RAG-based solution. This was surprising as it is quite domain specific, but also it seemed pretty clear that GPT must have be trained on very data very similar the (copyrighted) content we were using.

Further steps would be:

1. Use other LLM models. Is it just GPT3.5/4 that is reluctant to drill down?

2. Used specifically-trained LLMs(/LORA) based on the expected response style

I'd be careful of entering this kind of arms race. It seems to be a fight against mediocre results, and at any moment OpenAI at al may release a new model that eats your lunch.

I did a hobby RAG project a little while back, and I'll just share my experience here.

1. First ask the LLM to answer your questions without RAG. It is easy to do and you may be surprised (I was, but my data was semi-public). This also gives you a baseline to beat.

2. Chunking of your data needs to be smart. Just chunking every N characters wasn't especially fruitful. My data was a book, so it was hierarchal (by heading level). I would chunk by book section and hand it to the LLM.

3. Use the context window effectively. There is a weighted knapsack problem here, there are chunks of various sizes (chars/tokens) with various weightings (quality of match). If your data supports it, then the problem is also hierarchal. For example, I have 4 excellent matches in this chapter, so should I include each match, or should I include the whole chapter?

4. Quality of input data counts. I spent 30 minutes copy-pasting the entire book into markdown format.

This was only a small project. I'd be interested to hear any other thoughts/tips.

> Now, there are extensions out there that implement versioning – temporal_tables being the most popular I think – but none are supported for managed Postgres instances on e.g. Azure or AWS. This means that if we want system-versioned tables, we’re forced to make it ourselves.

This really provokes a wild geek rage inside me. Not that it is untrue, but that this is the state of things. It's that cloud providers are now often seen as the only responsible way to operate infrastructure, and in the process we hand over so much control of our tooling. We let ourselves be convinced that simplicity was worth a 10x premium, and admitted we were not grown up enough to handle the tools we had made for ourselves.

Sure, I know it is more complex than that. But I also feel sad about it.

Is anyone aware of a streams implementation for Postgresql? I see we can do simple queues, but how about a stream with multiple consumers / consumer groups (much like the Redis streams implementation)?

This reminds me of the short story 'Ansible 15715'. I won't spoil it, but they had a use for us and it wasn't as slaves.

Not sure about multi-tenancy, but I’ve been using VerneMQ with the Postgesql auth backend.

Oh wow! Thank you for sharing this! I had no idea.

I highly recommend the book “nothing to envy”. It is a series of first hand accounts from N. Korean defectors/escapees. Truly unimaginably bad.

In that theme, I’d also recommend D-Day Through German Eyes (short), as well as Second-Hand Time (long). Both are also a collection of first hand accounts. The former is the experiences of the German soldiers during the D-Day offensive and their beliefs in their actions at the time. The latter is experiences of the fall of the Soviet Union (a fascinating read, but also the most harrowing and emotional book I have ever read)

If anyone’s looking for an alternative, I’ve been using Epicurian cutting boards for years. Some kind of compressed wood composite I think.

They’re great to chop on and dishwasher safe (although they may discolour a little, but that never bothered me). Not cheap though.

My (inexpert) thought is that it would actually be the releasing of the vacuum that would drive the marinade into the meat. Which makes me wonder: does the rate of pressure release make a difference? How about repeating the process several times? Hum.

Not all are, but the Simpliphi batteries are designed to be plug-in replacements more or less

When I read this is thought it was a cool idea. I suspect it involves using ZFS’s file deduplication/copy-on-write mechanism to maintain a lightweight copy of a database on disk. Memory will still be required for holding the indexes etc, but in theory minimal extra disk space should be needed.

ZFS can also perform compression, which could be a big win as long as you have the spare cycles for lz4 (which is normally the case but may be an issue if you’re maxing out your NVMe disks’ bandwidth, say 3GiB/s+)