Examining the covidtests.gov architecture
adhoc.team
adhoc.team
Perhaps some good citizen hacker created a bot that automatically submitted a request for every address in town.
https://analytics.usa.gov/commerce/
You can get a JSON of the data also and see they have “ZZ” and “not set” as two cities excluded from the front dashboard. https://analytics.usa.gov/data/live/top-cities-realtime.json
I assume Ashburn traffic (#8) comes from AWS us-east-1.
edit: And another interesting thing, there are no EU countries in the geolocation table.
My guess is that’s what’s happening here.
So, what's probably happening is that some IP address in Graceville happens to end in the octect 0. And a bunch of other high-volume addresses, when you drop their last octect because of IP Anonymization, look like the IP address from Graceville.
As a more general note, the accuracy of geography reports in GA are suspect to begin with (Amazon and Google only agree at the state level about 75% of the time), and with IP Anonymization enabled it takes a substantial nosedive.
USPS Informed Delivery / Daily Digest is awesome and underappreciated
Anyone that works on this -- please try to recognize those of us with "unusual" characters or a suffix as our given name! =D
0. https://github.com/18F/analytics.usa.gov
1. http://analyticsdash-46.s3-website-us-east-1.amazonaws.com
A quick aside - I feel most folks on HN can say that they have been, in some way, shape, or form, a part of this journey for the industry. Take pride in being part of a community that has made this a possibility.
Until we measure our services in transactions / kW, things will continue to get more redundant.
Hopefully it'll catch on more, and encourage the use of productive AND efficient approaches.
No? Maybe we could get cost, energy and other efficiencies by aggregating disparate smaller and larger workloads over a system that scales up and down with demand, so every instance is processing X million TPS.
Cool idea. We could label that scalable, because it’s vastly more efficient!
Wed be better off letting people buy masks on amazon and deducting on their 1040
https://special.usps.com/testkits
without submitting anything, it returns this error code:
QWNjZXNzIG1haW4gcHJvZ3JhbS4gQWNjZXNzIG1haW4gc2VjdXJpdHkuIEFjY2VzcyBtYWluIHByb2dyYW0gZ3JpZC4
which base64 translates to: "Access main program. Access main security. Access main program grid."
Which is a JURASSIC PARK reference: https://jurassicsystems.com/
https://azure.microsoft.com/en-us/pricing/details/cdn/#prici...
It is amazing that every single thing I've read said more or less the same thing.
https://bugzilla.mozilla.org/show_bug.cgi?id=1719846 ("Bug 1719846: Add UA override for https://covid.cdc.gov/covid-data-tracker/")
I remember I had to install a UA spoofer addon when I used FF to access some sites. Absolutely obnoxious.
People on this site tend to think that any attack on the horrific spending efficiency of these agencies is motivated by an antipathy of the government in general, or a view that the government CAN'T do it right.
I'm not of that mind at all. My gripe is that the people who are most invested in expanding the government's role in the US for positive goals seem to have zero interest in even discussing the rot within these institutions that has crippled their ability to execute on the funding they already have. I can confidently assert that, if I'm being generous, 3/5s of the current Federal (and yes, contractors too!) work force is what my old chief called "furniture". I have no doubt that many, many private corporations have the exact same problem. (I've seen the inside of the average insurance company, and my god are those places bloated).
I once interacted with a private government contractor. They did an adequate job on a project. It worked, and it wasn't terrible. But they billed $5 million for a project that could have been built for $50 thousand, or—once you put up with a drawn-out sales process, endless client meetings and the occasional change request—more like $500,000.
The simple fact is that many large organizations, public or private, couldn't do a $50,000 software project. It costs more than that to close the sale and negotiate contracts. As Joel Spolsky pointed out, once you need to put salespeople on airplanes, even shrinkwrap software costs $75,000: https://www.joelonsoftware.com/2004/12/15/camels-and-rubber-....
> I have no doubt that many, many private corporations have the exact same problem. (I've seen the inside of the average insurance company, and my god are those places bloated).
Yeah, that has often been my experience. Sufficiently large companies tend to be deeply inefficient about small things. Partly this is because they're optimizing for different needs than you might think, and partly because $1 million is basically pocket change if it actually solves a problem.
But also, once a company grows beyond a certain size, it operates internally like a centralized command-and-control economy. Internal politics matter far more in the short run than any market feedback.
If anything, governments can have a slight advantage if they keep things in-house. Government employees are often slightly more idealistic than their private sector counterparts, and they're definitely paid less. So an organization like the US Digital Service can outperform the original private contractors who built healthcare.gov. The USDS was staffed by idealistic FAANG developers working for the common good at below-market salaries. They did a better job than CGI Federal, the original private contractor.
Obviously ideally they should be limiting the orders by person, but without national id and with scalpers ready to make a penny wherever they can we cannot have nice things.
It is. But there are definitely a lot of assumptions that happen. The USPS occasionally reports to HUD that my home is unoccupied because it has no mailbox, which leads HUD to report such to my bank, which makes them report to my insurance company, and then I get a call telling me they cannot insure a house that is unoccupied, and I have to explain it to them... again.
Anyway, yeah, I get the reason why they'd limit it to residential addresses. But its the "Every home in the U.S. is eligible" that is a bit of a slap in the face. Don't say it if you know it isn't true, and they know it isn't true because they address it in their FAQ.
I would guess that delivering to residential addresses only is the only way to ensure a limited number per household. Otherwise, you are going to have people trying to get a bunch for resale.
Unfortunately, mail theft went through the roof over the past year. 90% of our packages were stolen. So, we won’t ship anything via USPS anymore and got a PO Box for things that won’t ship UPS/FedEx (which happily deliver to my door).
But, it looks like the covidtests site correctly recognizes at least some of these zip codes and is allowing non-residential addresses, which is impressive.
(In theory you could vote for a legislator who wants to make that web site a priority -- but nobody runs on that. Why would they? It would put most voters to sleep.)
{ "error": "QWNjZXNzIG1haW4gcHJvZ3JhbS4gQWNjZXNzIG1haW4gc2VjdXJpdHkuIEFjY2VzcyBtYWluIHByb2dyYW0gZ3JpZC4=" }
To further the learning, can anyone guess which part of the stack is likely generating the error?
Alrighty then.
But how lame to give such an opaque error to so many people.
I can't leave my home to get a COVID test because I am on house arrest, so I guess I am just out of luck.
https://www.fda.gov/consumers/consumer-updates/why-you-shoul...
https://academic.oup.com/qjmed/article-abstract/114/10/721/6...
> Based on very low to moderate quality of evidence, ivermectin was not efficacious at managing COVID-19. Its safety profile permits its use in trial settings to further clarify its role in COVID-19 treatment.
We wouldn't have sent her back to school as early if she wasn't showing negative.
This is false? It was definitely false pre-Omicron and my understanding is that it's still false.
https://www.medrxiv.org/content/10.1101/2022.01.08.22268954v...
> sensitivity was 97.6% (95% CI 95-100%) for persons with symptoms.
If you're not saying that, I'm not sure what claim you're making that is contrary to my own.
With omicron the gap between them is a day or two smaller IIRC.
So, by default, where I live in Chicago you can't do anything at all. I was told by the Sheriff not to even leave my bedroom for any reason when I first got here. I live in a big commercial building, so I ignored their advice. I have an ankle monitor which uses GPS and cell network triangulation, so they know my rough position at all times. This means they knew I was wandering around, taking my dog for a walk around the building etc. The Sheriff would send people over constantly to harass me. I did a FOIA request to see their logs and it was funny to see that they have police constantly tagging my movements with quotes which basically say "Look, this guy isn't going anywhere, he is just walking around his building, we don't need to bother him" - but obviously there are dozens of different people working at the HQ and some aren't so kind.
I think it took over two months before I persuaded the judge to let me get groceries and go to the laundromat. So I have two hours on Mon/Tues/Thurs to get what I need. But I am supposed to go straight there and straight back.
There is a new law in Illinois which says I get two days a week to do what I want, but apparently because of poor coding in their computer system I can't get it because my jail ID ends in an odd number.
It is also a new law now that they can't charge you with "prison escape" for going outside your house unless you are gone for more than 48 hours. [people ended up with years in prison simply for taking their trash to the Dumpster behind their house]
tl;dr: I can now, after some hassle, get groceries. I cannot go to the doctor. When I needed my booster I timed the appointment so I could run into Walgreens on the way to the grocery store so they didn't notice I'd taken a detour. It might take me several months to get the judge to sign off on a doctor's appointment. If I'm dying I can go to the hospital, but I better have every scrap of paperwork and be willing to prove I was actually dying as I've known a couple of people who have ended up going back to jail after going to the hospital. [one guy took his daughter in an emergency - the judge told him he had no excuse for taking her as he could have just called an ambulance to take her and stayed at home]
I'm sorry this isn't working for you. I've had luck ordering them on amazon though.
S3 is nasty expensive for outbound bandwidth. I’m surprised the usps landing page is there. No reason that couldn’t have been on lambda as well.
A general rule of thumb is CloudFront is $2500/mo for every 100 Mbps of traffic you have, not counting origin or per request fees.
I would expect they are pushing multiple Gbps with every household in the US trying to order tests.
The most expensive piece by FAR is Opensearch (previously AWS Elasticsearch) using 6 instances x 2 environments (12 total).
The cost for Elasticsearch in two environments came to ~$371 out of a total $400 bill in my last statement.
The compute cost for Lambda is very low but if you scale out to potentially millions of users it's going to be much more than $100.
Not sure if they're using Opensearch but if they are it's likely a huge part of the bill.
I'm sure there's a better alternative but it's not my particular area of expertise and I haven't been able to focus on that yet.
I saw your comment about anvere and looks like you have a background in search/ES? I'd be interested to chat but didn't see a way to contact you!
What kind of traffic numbers are you seeing? Because I can't figure out how one would ever end up with spending more than a few dollars to host a site that rarely gets a post.
Some of it is on Akamai as the article describes, but if we had everything on CloudFront and wanted to estimate the cost of 1M orders: 2.38 TiB out + (say) 1 KiB/request * 67M or 67 GiB in: that's $275.50 per 1 million orders (approximately $207 out + $67 in + $1 requests cost).
If we add 1M Lambda executions, we still get $0 since that's the entire free tier. 10M Lambda calls at 50 ms each and 128 MB of RAM cost $1.8, so that's pretty much free as well.
Assuming that CloudFront caching is configured correctly, there would be very few requests to the S3 origin so their cost would be negligible.
Someone please correct me if I've got this completely wrong. Source: https://calculator.aws/ (CF: 2.38 TB + 67 GB + 67M req).
There will be 420 million orders according to sources online, which gets us to $115,710. Very interesting - thank you for the analysis.
They're probably on a contract with a yearly minimum (multi-million) with top tier engineering support, virtually eliminating any concern for individual resources.
This is the government we're talking about. A lot of factors are going to push that pricing up (high profile of the project, risk considerations for AWS, sales team seeing blood, etc)
But with the SLA and support requirements probably involved, I definitely don't see this running on a basic account.
Static just means it's already a file, and that gets delivered directly to the user's browser with no server-side substitutions or especially database lookups.
There's some ambiguity there, but that's the basic idea. The alternative is pages that are created on-the-fly when requested, often by looking up a bunch of crap in a database, or the server running a bunch of code to compile together separate files.
"static" sites absolutely change, it's just you usually have to deploy something when you change them, it's not just "a different person viewed this".
A bit of an aside, but since this is an article about architecture, wouldn't the more common pattern here be APIGateway->Lambda->MessageQueue->BackendServices rather than APIGateway->Lambda->Database->BackendServices? Or does DynamoDB have something like a queue that backend processes can subscribe to? (Non-AWS user here).
0: https://docs.aws.amazon.com/amazondynamodb/latest/developerg... 1: https://docs.aws.amazon.com/amazondynamodb/latest/developerg...
Yet, most government IT projects run very high costs to begin with anyway. Indeed, for all we know, they probably saved money, even with the high costs of running this at scale on Lambda. And it'll be worth it if the service is actually functional.
Just interesting to think about the cost of introducing stability as a standard before eventually the pendulum will swing back and get people to think about cost optimization.
If you saw 300,000,000 requests in the month on a 256MB lambda function and 500ms of runtime. It would only cost $678 for the order capture. Maybe my math is wrong here, but not huge in my book.
We also don’t have a national postal service, it’s a deregulated market. There’s an operator owned by the state, but only 50%, the rest is owned by Denmark, so it’s unlikely it could participate in something like this without it being an open contract for all delivery companies.
What would that even look like? Do they expect a unified UX language from the Gov t?
To be fair, the little documentation and blog posts here and there that i have read on the US Digital Service is that those folks reaslly are attentive to design considerations for always including accessbility...which is awesome!!
There are all sorts of turnkey solutions for frontend and backend infrastructure, but none guarantee good user experiences - only a team that knows their users' needs and is empowered to meet them can do that.
You cannot order to any address, only to residential addresses that USPS ships to (as indicated by their Residential Delivery Indicator). Specifically, you cannot order to commercial addresses and PO boxes.
Resale is not prevented but (intentionally or not) limited by the distribution of only 4 tests per home, regardless of how many people live there. This means that poor folks where typically many people share a household get fewer tests per person to begin with, while more affluent folks (who are less likely to resell the tests) get more per person.
It's a tricky balance to strike!
That being said, while this site is already extremely fast, they could cut their first paint times by a further 90% by just reducing their font file sizes.
Since this is a static page, and we want things really fast for all Americans (99 percentile?), maybe subset characters to what's shown above the fold, base64 encode that subsetted font into the document and load the remaining characters async as a fast follow. Bang, great CLS and ~0s FCP.
EDIT: US Postal Service's fault not gov
So, calibrate your expectations accordingly.
I don’t see where it says anything about apartment-dwellers being out of luck in the FAQ.
Seems like a reasonable compromise to prevent scalpers from using Apt 1, Apt 2, Apt 3, etc. at their house.
https://emailus.usps.com/s/the-postal-store-inquiry
https://www.cnet.com/health/usps-offers-solution-for-people-...
Having an entry on a support contact dropdown simply means they want to route those requests differently than general support questions. It does not indicate that the problem is widespread.
and what are scalpers going to do, compete with a free service?
Yes. A human to review 500,000,000 addresses. In a timely manner. We'll just tap that massive skyscraper complex full of surplus government humans sitting at computers not doing anything at all.
and what are scalpers going to do, compete with a free service?
Yes. They already do it with the free in-person COVID testing, by setting up fake COVID testing services and charging immigrants and people with poor reading skills who don't know any better.
No, many thousands of humans to review a couple hundred at most a piece.
"Huh, I'm supposed to deliver ten of the COVID test packs to this house. Nope."
It's reviewed right before delivery... by the mail carrier.
"Hey, that's just one house, there's not 20 appartments here! I'll leave the other 19 test boxes in the truck and they'll get reassigned to legitimate addresses".
If you have no division at all in your address then you're going to have trouble.