pselbert

There has been some progress on that front. For index size there is a new rebuild command in PG12 which works concurrently. There is also pluggable storage now, which enables much more efficient row deletion. I can’t find a link to the new storage format, but here is the announcement for 12 https://www.postgresql.org/about/news/1943/

Sure, you can glue things together with lua. Here is a script that handles descheduling from a sorted set: https://github.com/sorentwo/kiq/blob/master/priv/scripts/des...

That definitely does the job. My point is that it is much more complex than a select/update clause in SQL.

> What exactly is fragile about this approach? We've supported scheduled tasks in TaskTiger (https://github.com/closeio/tasktiger) via Redis's sorted sets and haven't had any issues.

Using sorted sets for scheduled tasks isn't the fragile part, gluing it all together to prevent losing jobs by shifting into backup lists (or hashes) is the fragile part in my experience.

No, unfortunately not. It does use jsonb for job arguments to make interop easy. That’s entirely on the enqueuing side though and not processing.

I don’t think it is much of a problem until you are pushing 2-3k jobs a second. Only the message saturation is a bottleneck anyhow, everything else I mentioned is handled by how a library is architected.

It seems like PG has been pigeonholed as only suitable for “small-medium” size queues, but without numbers to define what “small-medium” means. A few million jobs an hour is entirely reasonable for PG based on anectdata (and I’ve load tested up to 54 million jobs an hour).

PG is an amazing tool that can handle more than most people think (or at least more than I thought).

Postgres PubSub is great for triggering job dispatch, but it isn’t good enough on its own. There are a few caveats that make polling an additional requirement:

1. Message compaction — notifications are deduplicated, so if two jobs are inserted in the same queue the system may only get one notification. 2. Message saturation — with high levels of activity you’ll need to start discarding messages, essentially denouncing, otherwise the database gets throttled. 3. Dedicated connections — pubsub requires a dedicated connection for listening, which requires a single connection with custom dispatching or one connection per queue.

Relying on PG for everything is awesome regardless!

Redis is brilliant for simple job queues but it doesn’t have the structures for more advanced features. Things like scheduled jobs can be done through sorted sets and persistent jobs are possible by shifting jobs into backup queues, but it is all a bit fragile. Streams, available in 5+ can handle a lot more use cases fluently, but you still can’t get scheduled jobs in the same queue.

After replicating most of Sidekiq’s pro and enterprise behavior using older data structures I attempted to migrate to streams. What I discovered is that all the features I really wanted were available in SQL (specifically PostgreSQL). I’m not the first person to discover this, but it was such a refreshing change.

That led me to develop a Postgres based job professor in Elixir: https://github.com/sorentwo/oban

All the goodies only possible by gluing Redis structures together through lua scripts were much more straightforward in an RDBMS. Who knows, maybe the recent port of disque to a plug-in will change things.

Learning ffmpeg can be daunting but it is a highly useful tool to know. The option intricacies and peculiarities make it extremely hard to “master” though. It requires so much experimentation and experience (trial and error) that achieving mastery in a year seems nigh impossible!

This is one of the many things I learned from the destroyallsoftware screencasts [0] so many years ago. Before that I used vim in a gui (MacVim). Along with moving to tmux this completely changed how I work.

[0]: https://www.destroyallsoftware.com/screencasts

They were a life saver when I was going to school for furniture. We would order everything from them. Screws, bits, inserts, random bits of metal.

Later I discovered that they provide CAD drawings for everything they carry. That was immensely helpful when designing things in say, Solidworks. If there were some screws that were called for it only took a visit to McMaster-Carr to get a 3D version of the real part.

They have a major release every year or so, the last one was June of last year. They will release features in point releases though, for example Counters was released in 21.1 and Persistent Term was released with 21.2.

The page says the latest release was 2017-08, that seems tremendously outdated. Am I missing something? Is it updating in the background?

The LISTEN and NOTIFY don’t shouldn’t be on the same connection. In fact, many libraries require a separate dedicated connection to LISTEN on.

One feature of PubSub that is also a caveat is that it is transactional. That is great for production code, but makes transactional tests impossible.

Every person I know that bikes in Chicago has been hit at least once. Fortunately none of them have been seriously injured, but it is a constant danger.

There are lots of bike lanes downtown but they aren’t protected or respected. We have a long way to go to make biking a safe form of transport like it is in the Netherlands.

I recall purchasing two of these in the late 90s for relatively little money. Both cards could work in conjunction and it radically changed how games looked—really amazing how much of a difference it made. I can distinctly recall the difference in crispness, and how disappointing it was that didn’t apply to everything I used or played (3DS Max, CAD, etc).

They were the first thing I ever sold on eBay, sometime around 1999.

My most popular open source library is a Redis backed cache for Ruby. It continues to accumulate stars, but I haven’t had any issues in well over a year.

Sometimes software is just finished, not abandoned!

Not the original poster, but I would recommend to anybody that they move to Fastmail.

Our organization uses four primary languages (Ruby, Python, JS and Elixir). The package management situation for Python is by far the weakest.

We’ve been using Pipenv, but it is atrociously slow and flawed at dependency resolution. An alternative is extremely welcome, E.g. Poetry which was mentioned above.

Digital Ocean has their own data centers, they aren’t built on top of AWS

https://www.quora.com/Does-DigitalOcean-have-its-own-datacen...

I went to school for furniture design in Savannah. None of the teachers were missing any fingers, but plenty of guys from shops in the surrounding area were. From what I recall all of them had lost fingers from jointers and routers, not table saws.

Loosing a digit from a table saw certainly happens, but some of the other tools are less obvious and equally dangerous menaces.

Is there a long term plan for the Codeship domain, blog, and the content that you've generated?

I ask this because I've written content for the Codeship blog before and have links to it from my personal blog. If the content is going to vanish I'd like to publish it directly rather than have it lost to the ether.

It was what stopped the British led allied bombing which had escalated out of the Blitz. Dresden was not a military target and its destruction killed so many civilians that they stopped night time bombing raids.

I must admit, for the past couple months I've been digging for status info about Redis Streams. They would fit a use case we have perfectly, but we use cloud providers for Redis so manual compilation with modules isn't possible.

Really pleased to see that I'm not the only one digging for info and that work is ongoing.

This article from Brandur does a great job of explaining a use case for Redis Streams [0]. As part of that he begins by explaining what they are, how they work, and a bit about how they differ from Kafka.

[0]: https://brandur.org/redis-streams

There was a Stuff You Should Know podcast about this just recently [0]. An impressive amount of engineering has gone into something that seems much more artistic than scientific.

1: https://www.stuffyoushouldknow.com/podcasts/whats-the-10000-...

Typically I design applications to be as stateless and tolerant of restarts as possible, as they do get restarted during deploys anyhow. That is a minor issue compared to the difficulties around connection restarts and communication between nodes.

It is wonderful that Heroku now offers exec, which at least allows using observer and other remote debugging tools. The tooling keeps getting better—but there is no denying that it was designed for Ruby deployment, and working the BEAM has significant production differences.

Surprising development. I wonder if this will complement Gigalixir or compete with it.

The ability to connect between nodes, avoid mandatory restart, avoid connection limits and keep websocket connections alive easily gives compute engine a significant advantage over platforms like Heroku.

There was a promo for this on the most recent changelog. The term “heroes” seems a bit hyperbolic, but I’m intrigued.

When you have children everything changes. Then your success depends on teaching them to adhere to the same countermeasures—there isn’t much chance of that until they are older.

I had managed not to throw up for 20 years. That all went out the window when I had a child that attended school.