Unless it can die. And understand death. And adapt behavior to avoid death.
Of course, that raises a whole new set of issues.
506 karma · joined May 19, 2020
Of course, that raises a whole new set of issues.
I initially escalated to support. Similar thing - they said there was nothing they could do, and that I needed to cut a ticket to another team... one which I couldn't contact. They simply closed the ticket.
I did a chargeback for that portion of my bill. TBD whether they will nuke my account or not.
I'm also a little sad at this defeatist attitude. What you said might be true, but those things are solvable problems. Just requires a coordinated force of will from a few dedicated individuals.
However, FP's benefits can be overstated, especially for complex real-world systems. These systems frequently have non-unidirectional dependencies that create challenges in FP. For example, when component A depends on B, but B also depends on a previous state of A, their interrelationship must be hoisted to the edges of the program. This approach reduces races and nondeterministic behavior, but it can make local code harder to understand. As a result, FP's emphasis on granular transformations can increase cognitive load, particularly when dealing with these intricate dependencies.
Effective codebases often blend functional and imperative styles. Imperative code can model circular dependencies and stateful interactions more intuitively. Thus, selectively applying FP techniques within an imperative framework may be more practical than wholesale FP adoption, especially for systems with complex interdependencies.
We can continue to debate whether or not I (who worked on that team and generally understands what goes into building scaled cloud services, having built several myself) understand how a cloud provider responds to customers using your service in a way you clearly didn't intend, or we can move on with our day.
As others have pointed out, it's probably not a noticeable cost and, in fact, the fixed costs associated with setting something up yourself would far outweigh what you're paying to use S3 for this purpose.
Part of me just dies inside when I think of all the stuff needlessly happening behind the scenes, given it's not actually being used for storage. I mean, it's called Simple Storage Service.
If it were strict C in stead of C++ I would probably be in love.
(Source: I was on the S3 team. Opinions my own, etc.)
But the ecosystem that exists around the JVM also comes with both a bunch of awesome libraries, and a whole bunch of cruft and ceremony.
If, like Clojure, you're trying to pull in compatibility with JVM libraries, then you pull all that in. So now if you want a little C in your Clojure, you have to go through JNA or something.
I would like the option of going native with a Clojure dialect, without passing through either JVM (Clojure) or JS (ClojureScript).
By way of example, a little over a decade ago a famous online streaming company used to upload all of their video masters for transcoding. This process involved a huge upload and a huge workload, followed by a significant drop in activity.
The problem was that AWS had to provision for peak usage instead of average usage. This resulted in a situation where the peak-to-average ratio was very high for just one or two customers.
To address this issue, the solution was to incentivize these customers to spread out their workload more evenly over time, at least until they were no longer the largest driver of peak/avg.
This is also why things like Reserved Instances and Spot Instances exist.
Small objects (especially small hot objects) are actually problematic in S3. They cache them in the keymap instead of retrieving from block storage. But many small objects can quickly blow out a keymap shard. The keymap is designed for high-performance lookups. Because of this, it's also more expensive to scale out this class of 'storage' at the keymap layer than to scale out block storage. And you still have to do quorum writes to block storage and then cache eviction at the keymap.
If you're doing this for slow-moving leader election in a small cluster, fine. But if, for example, you started using this for leader-election among end-user-defined workloads that could scale up/down, you might find yourself on the other side of a call from AWS.
I wrote Lisp (Clojure) professionally for several years. It was the most fun I ever had.
My only problem was it was an ongoing battle to convince people - especially management - that I wasn't crazy.
Sure, I was crazy-productive, but they also thought that there was some nutjob in the corner that was quietly adding risk to the business.
If someone could give me the ergonomics of Clojure, but without the JVM and the ability to easily target 1/ native (incl easy bidirectional C iterop), 2/ wasm, and 3/ support both in aot and runtime + JIT modes, I would switch in a heartbeat.
Is that too much to ask?
This is because you're essentially pushing the problem down to S3, which does its own leader election in a way that is waaaay overbuilt for what we're trying to accomplish here.
But... that doesn't mean it isn't cool. :)
The emotion that AWS helped overcome was frustration that individual developers faced when trying to build something new. Suddenly, hardware was in their control from their keyboards.
That was a magical experience, and it definitely filled me with emotion the first time I uploaded an object to S3.
(I loved it so much, I later worked for that team!)
Engineers are driven by emotions like:
- Desire for intellectual respect: Choosing innovative products to appear forward-thinking.
- Risk aversion: Preferring established brands to avoid project failures.
- Professional pride: Selecting high-performing solutions for personal satisfaction.
- Peer validation: Making choices they believe colleagues will approve of.
- Cognitive bias: Favoring solutions that confirm existing beliefs.
What looks like logical decision-making is often an emotion-driven choice justified with technical arguments. This is evident in online discussions where product critiques are framed logically but stem from emotional responses or biases.
Effective marketing to engineers should recognize these emotional drivers while providing the technical depth needed to rationalize decisions. It's not about ignoring emotions, but addressing the specific emotional needs of a technical audience.
One interesting tidbit is that during the period this author writes about, AWS had a roughly 4-day outage (impacted at least EC2, EBS, and RDS, iirc), caused by EBS, that really shook folks' confidence in AWS.
It resulted in a reorg and much deeper investment in EBS as a standalone service.
It also happened around the time Apple was becoming a customer, and AWS in general was going through hockey-stick growth thanks to startup adoption (Netflix, Zynga, Dropbox, etc).
It's fun to read about these technical and operational bits, but technical innovation in production is messy, and happens against a backdrop of Real Business Needs.
I wish more of THOSE stories were told as well.
It's so incredibly brilliant that it spoils me for mainstream tech, but it's never actually "done enough" for mgmt to feel comfortable adopting it in any capacity. (See also: Unison https://www.unison-lang.org/)
Or, you can get them to adopt, but then you hit hiring limitations.
I say this as someone who built a large team of Clojure developers in a Fortune 100 company, and lived to regret it.
https://www.thedailymeal.com/1386775/crisco-facts-why-people...
I can't believe someone decided, "Hey, we should take this stuff we built as a submarine lubricant, and market it to housewives to make fluffier pie crusts."
Evil genius.