640 karma · joined June 28, 2019
The main one for me is typing props for a component feels extremely wack. In react I have a function, that function takes a type, I export that type and can import that type. In svelte I'm exporting the props individually? It's annoying to get the type declarations in one spot and make that easy to import for other components to use as intermediate typing. This is one of those things that I didn't even think twice about until my project reached a medium size and had multiple developers building out different portions -- it's inconvenient to scale and none of the workarounds feel good.
I feel like this statement alone demonstrates a wild misunderstanding of what OTP does and is. Kubernetes is at best a replacement for VM-level scheduling, but without a slew of additional functionality and application-level interaction with the kubernetes runtime, individual servers are basically running blind in a container, totally unable to do anything that OTP provides to gen_servers for free.
I could be totally wrong, but maybe combined we can make sense of it. From my read it feels like DD misguided this third party which then advised the author to spike.
That being said, I think it's a little funny because it epitomizes APIs that I see people build where ACID compliance is a huge "must" for their database choice and then they expose resources which can only be independently updated in a manner that fails to expose ACID beyond the scope of individual resources. I think this is funny because it really makes you question if they ever actually needed full mutli-table transactions at all or if they could have used a different database all together.
Anyways, it's a really cool tool.
It likely failed due to the nature of its transit (in my pocket, with my keys) but that's kinda the point. I think it's worth pointing out to those who haven't really investigated your "always plugged in" is the known failure of flash drives. Flash storage does has a fairly well documented write/erase cycles in the realm of like 10k-100k cycles. While the data should still be readable, its pretty easy for bad controllers and bad filesystems (most flash drives use fat32 which is a bad filesystem) to fail a write and corrupt data.
I think the main issue I have with a "do it yourself because you can't trust the cloud" mentality is that this advice usually comes from people who have done their homework as you have. It'd be dangerous to apply "just do x" advice without those conditions as it's a bit more nuanced and it can really put people in bad situations. I've seen this same thing with people who use a NAS over cloud storage and then it fails and they discover just what the monthly cost of cloud storage actually paid for. It'd be like telling home owners "just don't have a fire" and instead of paying for insurance, but not talking about the money you set aside and fire mitigation systems you invested in.
Just to offer a counter anecdote, I had a flash drive fail with my kdbx file on it and it was a monumental pain in the ass to recover from because I didn't have backups. Have backups. Especially for critical passwords that lock you out of everything. Flash drives do fail. Statistical failures SPECIFICALLY mean that some people will not fail, but that doesn't mean failures don't happen or that they're unlikely/uncommon.
So a quick aside: SQS is one of the oldest async "serverless" services in AWS. Historically, web applications ran into issues with large processing jobs so they would offload this work from the web server, which would synchronously handle requests, and instead do the work in an async manner that aligned with capacity. Job Queues or were designed to have workers pull messages in a FIFO manner and then capacity for those jobs could be planned separate from web capacity. Major contender in this space was RabbitMQ and something like Rails' ActiveJob, which is an ideal user of SQS.
So SQS is valuable but why isn't it here? Well, the short answer is: it is valuable, but in the conditions laid out (use with lambda) there are significantly better options. So lets look at the pattern they're using and talk about it. They're calling SQS a queue here, which is an easy mistake to make since it's in the name. I'm going to just toss an asterisk on it for a few reasons; first, because lambda will consume more quickly than most producers so it won't really sit idly for the retenetion window; second, because SQS (low scale) and single instance MQs like Rabbit will handle messages in a FIFO manner, but SQS is a service and abstracts scale away. At some point, the FIFO aspect of it becomes "best effort" unless you configure it to be FIFO (really avoid doing this). To me, this is a *major* design consideration and arbitrarily calling things a queue when really it's more of a pubsub is a bit more than just pedantry. At scale you *will* see out of order, potentially duplicated messages with SQS without FIFO on. Handling duplicates should be at the front of your mind.
So... we're using a pubsub pattern which is still maybe pedantically different from a queue, you're still asking why is that so bad? It works with lambda right? Yeah it does. Here's a list of services that work with lambda in a pubsub manner and would allow a user to publish a message and invoke an async lambda with that message:
* Kinesis Streams
* SQS
* SNS
* Event bridge
* Lambda (use Invoke and set "async" to true)
* DynamoDB / S3 (this one is abnormal but still possible -- there are more that I'd categorize here but these are common)
A lot of people look at this list and quickly eliminate options. DynamoDB table entries and s3 object drops are clearly misuses or special case jobs (good call). We want a DLQ via config not writing it ourselves so async invoke lambdas are out (bad call - normally DLQs are /dev/null but more expensive). Kinesis streams has an idle cost (true but lower than most think) and SNS and Eventhub aren't reliable or don't hold messages ** (this is false). The assumption is that SQS is the only remaining option that has:
* Reliable delivery
* Dead letter queue
* Roughly ordered delivery
* Isn't a "write it ourselves" or "misuse" of service (sorry dynamo/s3).
But... that's not the only service that does that. SNS and EventBridge both meet those qualifications. That surprises a lot of people. Both of these services have at-least-once semi-ordered delivery (same as SQS) with lambda and feature retry and DLQs. They also have a really really really huge feature for any operator who runs big serverless pipelines:
* They're emitter/publisher focused, not subscriber focused. You create a queue for every notified party, you create an SNS topic for every notifying item. This means you can add a secondary consumer (like... a network tap?) and see messages flowing through the system without changing the system. This is huge for debugging.
At this point you might say but what if I want disparate services to be able to consume from the SQS queue but steal each others work? Well... don't use lambda -- the lambda is going to steal all the work -- it can scale up to your account limit without adding more settings that aren't set here. Mostly SQS and worker queues aren't what you want if you want lambda. The real limit for burst-ability is set at a fairly large account limit level. There are *some* instances where queuing might be right, but at those scales you'd actually be asking if maybe Kinesis Streams is a better choice (usually it is).
TLDR: SQS is usually not the tool you want with lambda as a consumer. It works, but SNS or Kinesis are much more common. SQS comes with a lot of knobs and whistles that can dramatically run up your bill when using lambda with naive settings (I've seen this a lot, where the DLQ is setup for redrive and it just fail loops).
Hope this helps a bit. Note: I have about 5 years experience working with serverless/event driven application architecture.
SQS with minimal config into lambda. Alright, good luck.
Suppose you store your ids as a 64 bit number in the database, that's plenty of room, we shouldn't need to put an alarm for when we're reaching the end (although operationally you might want to still) - it's 9,223,372,036,854,775,807 when signed and 18,446,744,073,709,551,615 unsigned. That's a fine, sensible decision to make at the database level. Now suppose you have an API that interacts with these ids in the database and exposes them down to users in JSON. Javascript (and JSON by extension) numbers aren't integers they're double precision floating point numbers -- quick pop quiz, without looking, what's the maximum representable integer before you lose integer precision? Do all of your APIs work with auto increment numbers (which should still be safe) or are some of them vendor supplied, like UPCs or ISBNs? Will your developers remember this when they make a new API or will they look at an old api and think "it's a number there, I should match it"? Do some vendors front pad their numbers with 1s (this happened to us)? There are a lot of variables here and one big juicy thing developers want and know other developers want - consistency of rules. So I'm offering an easy, consistent rule:
Store it however your database lets you store it compactly, know your limits, set alarms, stop apologizing for database lack of features, but MOST IMPORTANTLY: insist to your developers that its type in code is a variable length utf-8 string, regardless of it's contents. Overflows happen, you need to accommodate them at business logic levels all the time, but stop letting developers trust numbers as being self-bound. They're not numbers in the URL, they're utf-8 letters in a string. They're not numbers in JSON, they're utf-8 letters in a string. Give them an a function that verifies the format of a string and stores it however you want in the database, but stop treating IDs like numbers even if they're numeric.
That's my opinion. Not that ints are a bad db row type.
I think this is where a lot of IP proponents would argue that IP is a way to form capital where it doesn't exist - around formalizing ideas and processes into something that you are legally entitled to rent. I think this really dangerously damages ideas that can help people and I think that it's a terrible crutch for a bad system.
For your second point, maybe that's okay? I think you're right with the tribal nature of people -- so if teams disagree, maybe the org isn't healthy and it's time to form two healthy orgs as part of a codified severance that doesn't result in layoffs or bankruptcy. I think the reason they don't is its expensive and difficult because you have to hire lawyers to figure it out after something is wrong. If tribes are mature enough to disagree and coexist, I'd call that healthy and worthy of continuing if both sides value each other.
I'm not sure many orgs codify severance. Typically this is something that partnerships account for but not corporations - corporations just fire people.
I think a more important concern is how does an org ensure that people remain tolerant and accommodating (both in legally required ways and in ways that expand your views and ideas) of people that may not mesh perfectly? I don't think a "divorce" of the org should be frowned upon and I think many orgs ought to codify how they sever to account for the possibility that people don't agree. I don't, however, believe that it's good to avoid conflict and sever every org over every meaningless spat.