znep

It also allows you to prewarm a read replica before sending queries to it. You can also do this by slowly ramping up traffic to it (and may want to do that anyway for other reasons) but depending on the app and query patterns, that by itself may not be enough especially if you are worried about your p99 latencies.

It can also be helpful when you have network attached storage (eg. EBS volume in AWS) since even if it is SSD you still have the network latency for each IO.

My phone line was down for 4 days because they accidentally disconnected it when hooking someone else up and it took them that long to fix.

Hopefully they actually finish this at some point to be useful by showing utilization and not just quotas. For the handful of things I looked at, other than lambda which they use in their example, utilization just displays "-". I'm sure the current functionality helps some use cases, but I'm bit disappointed as this just adds another place I have to look to try to figure out quota madness.

SSD on a 2016 macbook pro. The timing coincides exactly with the FS (and OS...) upgrade.

I wouldn't expect that level of disk IO to cause issues dragging existing windows around without some deeper lock contention or something, it does not behave like just IO bound. Stuff being slower, totally, but not to this level. It was never an issue before the filesystem upgrade.

Hopefully somehow maybe while they are in there they might do something to fix the nasty system lockups I have seen when starting both time machine and carbon copy cloner backups since switching to APFS. I'm assuming it is related to having >5 million files, but haven't figured that out for sure.

The symptoms are the system becomes almost totally unresponsive for up to 5 minutes to the point where you can't even drag and drop an existing window sometimes.

Reading between the lines of their status history is also interesting: https://status.docker.com/pages/history/533c6539221ae15e3f00...

Their message seems to be missing the "which Hub accounts are impacted" info unless I missed that, so not sure if they are only sending the message to the folks impacted or what.

Another interesting thing to watch in the electric airplane space is Harbor Air and magniX.

https://www.harbourair.com/harbour-air-and-magnix-partner-to...

The float plane business seems to make a lot of sense in this regard because, at least in the Washington state and BC areas, it involves a lot of small planes and relatively short flights that aren't easily replaced by other modes of transportation, especially for people who essentially use them for commuting and can't live the same lifestyle using a ferry boat instead.

I don't have enough knowledge to know if the technical aspects make sense from an aviation perspective but nothing stands out as unfeasible.

An interesting postgres oddity I ran into a few years ago that isn't about counting but is in a similar arena is how under very specific circumstances, "SELECT *" is significantly faster than "SELECT one_column" despite the conventional and normally well advised advice that you should only select the columns you need.

See https://www.postgresql.org/message-id/CAN1FPGN1ynBj3m1DMszc9... and Tom Lane's followup for details.

I would guess the pricing model is actually closely related to the main dimensions of their costs and is quite valid.

The key point is illustrated by this quote from their main landing page: "storage and compute are decoupled, allowing each to scale independently".

This suggests it is built on top of the Aurora storage layer, or something similar, as other comments have suggested. This means there is a real cost per I/O operation because you aren't limited by the physical hardware of the compute instances, you get "free" storage nodes underneath that do much more than traditional storage and thus have to be built into the pricing structure.

It is definitely not going to be the cheapest possible solution for all use cases, but do the math before you reject it. If it does follow the Aurora pattern, then the number of I/O operations you are billed for will be a lot less than you may think because, to use another quote from their product page, "Amazon DocumentDB reduces database I/O by writing only database changes to the storage layer, avoiding slow, inefficient, and expensive data replication across network links". I think that quote is harder to understand without background as it sounds like market speak, but lines up very well with some of their in depth Aurora whitepapers, such as https://www.allthingsdistributed.com/files/p1041-verbitski.p... Again, I haven't seen evidence this is based on Aurora but the details they talk about line up really well.

They don't have to fully maintain it on their own. Packaging their own openjdk distribution gives them control over timely targeted bug and security fixes even if they don't have "JVM experts" working on it. I'm not saying they don't have such folks working on it since, well, James Gosling does work there after all along with many other talented folks, but the move makes sense regardless.

At this point it appears to largely be a vote of support for openjdk as the project evolves.

From their announcement:

Amazon has already made several contributions to OpenJDK 8 and we look forward to working closely with the OpenJDK community on future enhancements to OpenJDK 8 and 11. We downstream fixes made in OpenJDK, add enhancements based on our own experience and needs, and then produce Corretto builds. In case any upstreaming efforts for such patches is not successful, delayed, or not appropriate for OpenJDK project, we will provide them to our customers for as long as they add value. If an issue is solved a different way in OpenJDK, we will move to that solution as soon as it is safe to do so.

The list of specific patches applied is here:

https://docs.aws.amazon.com/corretto/latest/corretto-8-ug/pa...

Looks to be mostly or maybe entirely backports.

Good to see another option out there, it will be interesting to see how many of these get folded back into openjdk8 as it transfers maintainership.

As far as I understand, Red Hat plans to take the lead on supporting a LTS version of OpenJDK 8 after Oracle steps down, with the intent to eventually do the same with 11. Various organizations such as AdoptOpenJDK will then be making releases off that.

Are you the Mat Keep who is Director of Product Marketing at MongoDB? If so then, FWIW, people don't often respond well to marketing links posted without identifying your connection to the company.

I'm not clear what you are trying to suggest with that link. The whole point of the claim in the article around sorting is that in Aurora, for using disk to sort, postgres only has access to a relatively small filesystem that runs out of space and can't use the Aurora storage that is actually backing the database.

The reasons I couldn't live happily without swap on my development machine:

- Macbook Pro's don't have a lot of RAM. Still. (Let's not get into that whole discussion as valid as it may be, that is far from the biggest issue I have with current macbooks...)

- Macbook Pro's have no user facing complexity to use swap or to have it encrypted. Heck, they even give you a middle ground as well by default of compressed RAM.

- I have to run a lot of services to run our development stack. Most of the time I'm only using a few but don't want to have to go manually start and stop services all the time. Also, even within a service typically access to most of the memory isn't required for every request. Swap handles this quite well.

- There are parts of our development stack that, simply put, are extremely bloated in terms of memory use. Four gig webpack process, I'm looking at you. Yup, that is ridiculous and should be fixed and maybe there is a fix out there we haven't figured out ... but I don't care, isn't my problem, and I don't have to fight it because it seldom accesses most of that bloated memory so it lives great in swap.

- I like being able to switch to working on something else without having to shut down and restart all the applications I'm using as required for the new task. Swap is a great fit for that. For example, if I have to switch from developing code to analyzing a large Java heap dump, which requires lots of memory, I don't have to go shut everything down, it just quickly gets paged out to swap and comes back when I need it. I don't care if it takes an extra 30 seconds for switch between these two tasks, it already takes much longer than that to load the heap dump regardless.

I think often people give swap a bad name because they never see what it does well for them unless they go looking for it, they only see it when they are asking their machine to do something where the working set is actually too big for memory and blame swap.

That said, no need for swap if you can live fine without it. I just don't think that is good general advice.

For an example of a somewhat similar project, see Austin's bluetooth traffic sensor data:

https://github.com/cityofaustin/hack-the-traffic/tree/master...

They used to have the code used to anonymize the data available, but I can't find it. I believe they re-randomized the MAC addresses daily so you could track trips but not a device long term. With enough sensor locations that seems like it could still be vulnerable to de-anonymization to some degree.

Exactly. I'd go further and say the burst capacity is actually a super useful and powerful feature that is very hard to get on your own hardware. Definitely can catch you unaware if you aren't on top of it and can get expensive for some needs, but as you say not hidden at all.

I have a hacky shell script I sometimes use for moderately sized environments that don't have better monitoring setup, it reports the minimum percent of burst balance remaining over the past day, requires the aws cli and parallel:

  #!/bin/bash
  while getopts "p:" opt; do
    case $opt in
      p)
        export AWS_PROFILE=$OPTARG
        ;;
      \?)
        echo "Invalid option: -$OPTARG" >&2
        ;;
    esac
  done

  if [ ! "$AWS_PROFILE" ]
  then
    echo "-p <aws profile> or AWS_PROFILE env var required"
    exit 1
  fi

  start=$(date -v-1d +%Y-%m-%dT%H:%M:%S)
  end=$(date +%Y-%m-%dT%H:%M:%S)

  aws ec2 describe-volumes --output text --query 'join(`"\n"`, Volumes[*].VolumeId)' | parallel -j 20 "echo -ne {}\\\t && echo \$(aws cloudwatch get-metric-statistics --start-time $start --end-time $end --period 86400 --namespace AWS/EBS --statistics Minimum --metric-name BurstBalance --dimensions Name=VolumeId,Value={} --output text --query 'Datapoints[*].Minimum')" | sort -n -k 2

Not any more, as of some number of months ago you can grow and change volume type on the fly especially if it isn't a boot volume.

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-expan...

Read the limitations very carefully, http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/considera... In particular if it was attached before Nov. 1, 2016 you need to do a one time stop and start of the instance or detach/reattach the volume and there are limits on instance types supported (but the definition of "current generation" is broader than you might assume at first).

This is quite an awesome enhancement, we were able to transparently convert a bunch of 15TB volumes from gp2 to st1 without downtime or impact to the app, and save a bunch of money.

Does anyone have any real world experience with how the Amazon DynamoDB Accelerator that AWS released as a preview a few months ago can help out?

The biggest reason I've found for using a database as a queue is that you can actually run queries against it, both writes and reads. This is very useful when different entries in the queue can have some relationship that matters when processing.

Do you want to wait 30 minutes after an item is queued then process all queued items for a given entity (eg. user) at once? Do you want to support more complex retry logic, such as efficiently handling some subset of jobs failing due to external dependencies being unavailable while others can complete? Do you want to be able to track what jobs are in progress (doesn't really work with skip locked), and even have the ability to update their status while processing for finer grained tracking? Do you want to be able to effectively monitor the contents of the queue beyond just the size (and some queues make it quite inefficient just to get that)?

Some of these are possible with purpose made queueing systems but, in my experience, even if they are they can be quite inefficient, complicated, and are often poorly tested and supported areas of functionality. For example the last time I tried to use JMS selectors with ActiveMQ (selectors let you filter messages based on a subset of SQL) it was a nightmare both performance wise and functionality wise and I wished I just had a database that was actually built to support SQL.

Other points about the cost of introducing a new technology (especially if you need strong durability, availability, etc.) and being able to update your database in the same transaction are also valid, but can often be dealt with by some extra effort.

Don't get me wrong, if you just need a queuing system the first thing to consider is a system designed for that purpose, especially if you aren't very knowledgeable about databases and have basic requirements. It can be nuanced to correctly and efficiently use a general purpose database as queue. At the end of the day, however, keep in mind that under the hood a queueing system is just a database that exposes a limited API that makes it easier to get basic operations right and hopefully easier to scale (sadly I have seen all too many queuing systems that utterly fail at that part) at the expense of flexibility.

Just setting https_proxy isn't going to give you any caching benefits for https requests and isn't going to MITM TLS in any way. The client will make a CONNECT request through to the destination and the encryption is end to end and the response not cacheable by the proxy.

From reading what you wrote, what I think you are talking about is statement caching where it can cache the "how do I run this" but still has to read each row (or indexed row) from the buffer cache.

The mysql query cache actually caches the results of the query, see https://dev.mysql.com/doc/refman/8.0/en/query-cache.html so it doesn't even need to look at the rows at all.

SQL Server may have a similar thing now, I'm not up to date on it, but wanted to call out the distinction based on how you described it.

No question, SQL Server has a lot of awesome qualities for the right use case and has come a long long way since the Sybase days.

True point (disclaimer: don't know how mysql has evolved in the past 5 years).

Fits with my general feeling of "if you want to optimize for a super specific use case, sometimes mysql can provide some cool tricks for that use case that can be amazing" but if you want a more general solution if you need more flexibility or don't have the knowledge or don't know your future needs, postgres shines.

While that is true and the mysql query cache definitely became a scaling issue as machines got bigger with lock contention etc., postgres doesn't even have a similar feature period.

Suppose you have a query that is doing a group by on a low cardinality field (eg. state) on a hundred million row table. Large amount of data in, small amount out. postgres has to actually look at all the rows to re-run the query. The mysql query cache only has to pull the tiny number of results out of the cache for that specific query.

This is a dangerous feature because there is a super slippery slope as traffic increases... any change to any of the tables invalidates the cache when you need caching most, and hits all queries against those tables at the same time. But if used well in an environment with many copies of the data (replicated or otherwise) where you could manage when updates happened, it can be a lifesaver.

InnoDB also has its own buffer pool that caches rows and is in general terms similar to the postgres buffer cache.

It is fair to say that mysql has a lot more features like this that are "sometimes life-saving hacks that can bite you hard if you don't understand them" and postgres is much more restrained in ensuring the features are a little more general and thought through.

Absolutely. The only thing that demanding total transparency gets you is hidden private groups. Private conversations will happen one way or another. From the outsider's perspective, the important thing is to make it explicit that there is that level of communication outside the public sphere when conveying decisions that were legitimately made by the core contributors that aren't directly observable from the outside. The goal is the reasoning and thoughts can be shared as much as possible, but when that isn't possible just acknowledging it is important.

At least that was my takeaway from my experience almost 20 years ago in a popular open source project.

This is a horrible interview question for all but a tiny handful of jobs:

  Is 300.168.0.123 a valid IPv4 address?

...but it is a great late night fun trivia question about "how can adding one character to that address make it into an address accepted by many tools".

This has been in beta for a while, the key thing lacking for our uses, at least with postgres, was full support for DDL.

The lack of "true" replication into postgres RDS instances is one of the things blocking us from fully using it.

If you didn't catch it, AWS now supports upgrades between so-called "major versions" (eg. 9.3 -> 9.4). https://aws.amazon.com/about-aws/whats-new/2015/11/rds-postg...

As long as you don't have too many microservices... the polling and templating doesn't scale too well.

This is related to https://github.com/hashicorp/consul-template/issues/165#issu... although that is even more extreme.