Fixed just for you :) I always forget that <meta> tag.
1,526 karma · joined October 28, 2019
Despite it being an awesome feature I've been itching to use, I've never actually found a use for it beyond messing around. Most places where S3 Select might make sense seems to be subsumed (for my uses) by Athena. Athena has a rather large amount of conceptual and actual boilerplate to get up and running with, though, S3 Select requires no upfront planning beyond building a fancy query string (or using their SDK wrappers)
Where S3 Select is likely to become fiddly is anywhere multiple files are involved. Athena makes querying large collections of CSVs (etc) straightforward, and handles all the scheduling and results merging for you.
Given modern Python means type annotations everywhere, the convenience edge between it and modern C# (which dispenses with much of the javaesque boilerplate) is surprisingly thin, and the capabilities of the .net runtime far superior in many ways, making it quite an appealing alternative especially for perf sensitive stuff.
The only thing I'd reserve judgement on is the tendency to throttle. I haven't got far enough to characterize it, but it's not clear how much value those extra cores will add over the n100 with TDP settings tweaked down in the BIOS, and if leaving the n305 to run at max TDP, heat/noise/cost/temperature-related instability may start to become an issue, especially when packing other hot components like a decent SSD into the tiny cases they come in.
> pickling
Sounds like if this is the tooling and the task at hand, about the most complex things that should be passing through the pickler are partitioned lists of filenames rather than raw data. E.g. you can have each partition generate a parquet for combining in a final step (pyarrow.concat_tables() looks useful), or if it were some other format you were working with, potentially sending flat arrays back to the parent process as giant bytestrings or similar
This is not to say the limitations don't suck, just that very often there are simple approaches to avoid most of the pain
Depending on how you look at it, you could view large chunks of DOM state through this lens, as intermediate data that only exists to cache HTML parsing results. What's the point of allocating a hash table to represent element attributes if they are unchanged from the source document, and reparsing them from the source is just as fast as keeping around the parsed form? etc. These kinds of tricks only tend to present themselves after optimization work is done, which is annoying, because it's usually so damn hard to justify optimization work in the first place.
There's a simple broad-spectrum fix for that. For the odd occasion I want to view high definition photos in Linux I just switch screen mode temporarily, for the rest it is sooo not worth the hassle
Have often wondered about this in terms of some of their control plane APIs, a read-only IAM key used as part of C&C infrastructure for a botnet might be interesting, you get DNS/ClientHello signature to a legitimate and reputable service for free, while stuffing "DDoS this blog" e.g. in some tags of a free resource. Even better if the AWS account belonged to someone else.
But certainly, ability to serve an unlimited URL space from an account with only positive hits being billed seems ripe for abuse. Would guess there's already some ticket for a "top 404ers" internal report or similar
In any case 2 weeks seems like an impressive turnaround for such a large service, unless they'd been internally preparing to acknowledge the problem for longer
No skin in this game, but looking at it from the respondents' perspective, 20k split between 4 is incredibly attractive: 5k and an apology to put an end to any threat relating to the issue, given the alternative of potentially unbounded costs to demonstrate innocence with a tortuous and uncertain outcome. From the claimant's perspective it is more confusing, triggering such a heavyweight process then settling for so little given the claimed harms suggests all kinds of things, not least including the kind of advice he may have received given the strength of the case as his solicitors understood it.
Obviously example above is contrived, but same principle applies to a pool of 1000 disks as it would 1. You also don't escape this issue with regular hot storage either, there is still a (((iops * replication count) / average traffic) / max latency) type problem lurking, which would still necessitate either limiting density or increasing redundancy according to expected IO rate. This is one reason why some S3 alternatives with weaker latency bounds (not naming names, they're great but it's just not the same service) can often be made substantially cheaper, and why at least one of S3's storage classes may be implemented entirely as an accounting trick with no data movement or hardware changes at all
For the rest of us, another half-decade+ wait to get useful levels of compatibility especially on mobile
Say 1 chunk lives in Germany, Ireland and the US each. Client races GETs to all 3 regions and cancels the request to the slowest to respond (which may also be down). Final client latency is equivalent to that of the 2nd slowest region, with substantially better availability due to the ability to tolerate any single region being down
Still wouldn't recommend using E2 for anything important, but ^ was one potential approach to dealing with its terribleness. It still doesn't address the reality of when E2 regions go down, it is often for days and reportedly sometimes weeks at a time. So reliable writing in this scenario would necessitate some kind of queue with capacity for weeks of storage
There are variants of this scheme where you could potentially balance the horrible reliability storage with some expensive reliable storage as part of the same system, but I never got that far in thinking about how it would work
It wouldn't matter if he claimed his 5.25" drive shot lightning bolts the first time he tried to compile that dialog, the stories are still great and it's personally enough for me having grown up with these systems to hear them told (not much different from listening to a pseudo-senile grandparent exaggerate war stories for entertainment's sake)