makmanalp

> Actually, the closest natural viruses to covid were sent to Wuhan for study

Please don't "actually" something with an uncited statement without backing it up with any sort of detail. A news article? Anything, honestly.

> that is what the Wuhan institute of coronaviruses was built to do right?

Really? It's "Wuhan institute of virology". Not "coronaviruses". Right?

I have no specific horse in this race but I feel like I'm losing my mind every time I read any discussion on this subject because there's always folks throwing around random statements as if they're established fact and I feel like at this point, on this specific subject, on this website, we should know better, geez.

Just off the top of my head I can think of a few cases where we easily have 100-1000x that on a single table in shared node on kube, with an ebs volume (point being, bog standard, if a tad large, commodity hardware). If you properly index things and your workload isn't too crazy in a way that needs access to all of this at once (e.g. point lookups with some recency bias), classic RDBMSs have this use case covered - and it's achievable by anyone!

> But... didn't Congress pass some corrupt legislation years ago that made it illegal for the government to set up a tax-filing Web site? What happened to this: https://www.propublica.org/article/congress-is-about-to-ban-...

Thanks for pointing this out - looking at the responses, folks blaming congress might have a point but I'm also surprised that this thread isn't directing more of the frustration to intuit and others involved in the deliberately confusingly named "free file alliance" - an industry coalition that's nominally providing a free filing option that's hidden / limited / difficult enough to use that very few people actually do despite it being free.

The answer to your question is that initially yes, it was banned as part of a mutual MOU, but it is not anymore. The rest is in here: https://www.propublica.org/article/irs-reforms-free-file-pro...

Here's more on the subject: - This one has internal slides from intuit https://www.propublica.org/article/inside-turbotax-20-year-f... - This is a report prepared by a left think tank (pointing this out as a disclaimer) that has a lot of details from earnings calls and such: https://big.assets.huffingtonpost.com/athena/files/2023/05/1... - This one details on the influence of the "revolving door" of lobbyists taking governemnt positions on the IRS' policymaking https://www.opensecrets.org/news/2023/05/turbotax-owner-intu...

The likeliest answer is that it's too hard to tell - it's too small of a populace (1-2% or less likely) that's hard to reach in an unbiased way to sample properly for a poll (without it getting expensive, I guess). People are mostly speculating based on his attitudes and positions.

Admittedly I could be missing something but isn't this pretty sensationalist? Seems like an expected result of a big new venture that doesn't have time to faff about with scaling up slowly and is dumping money into large up front costs that can't be recovered by sales just yet.

But it seems a bit dubious to say that Ford, the company that could fairly claim that it invented modern mass production, is going to have the trouble scaling anything up. Comparing this to Tesla in 2017, by all rights a young company at the time that for years that suffered basic quality control issues while trying to scale, seems disingenuous, right?

Story time: I went to a small and pretty nerdy tech school (WPI), and there was an IRC channel at the time. One day I noticed that there were messages coming from a user named, IIRC, "compooper". Later I learned while visiting one of the regulars' student housing that they had an upstairs bathroom with an old vt-100 or somesuch wired to it. Not quite the same but got a laugh out of me :-)

> Real estate prices going down does not affect the costs to the landlord (assuming it is an investment property), their mortgage expenses are fixed

Silly question but the fact that their mortgage expenses are fixed works to their benefit unless they bought at the peak, right?

Talking specifically about the big US city I live in - most landlords who already owned property since forever still jacked their prices up many hundreds of dollars in the past couple years because the market would bear it, not necessarily because they needed it to cover costs.

I'd imagine that the renters paying the largest rent (a few thousand, let's say) would be the likeliest defectors to buying a house. So buyers in the the top end of the market collapse, and those landlords begrudgingly start lowering prices - they can still make a profit. Maybe the low end stays the same.

I have zero understanding of the economics involved here, just pontificating.

> I think this idea can apply to "speaking with someone" as well. Again, I'm not saying that if there is a crises people shouldn't seek help. Just that speaking to someone, as far as I can tell, ought to have a purpose other than "I just need to talk." What does that mean, to "need" to talk? What are you looking to get out of the conversation? Do you need advice because you don't know what to do? Do you feel alone and isolated and need evidence that you are not?

I think you nearly lead yourself to the answer here - it can and should have a goal of learning and self improvement.

Personally, what I thought was most helpful about therapy was getting a "second opinion" on the things that I think and the way I think about them. While I used to think of myself as a highly rational, evidence based, logical etc etc person, I discovered over time via a helpful outsider that in fact I was subject to biases and disordered thinking in the same way that most people are, all the while thinking that I was being perfectly reasonable. Especially when it came to things that I had bad experiences with in the past, that I was afraid of and / or was trying to avoid unpleasantness around.

I found that I could rationalize (meaning: reverse engineer seemingly reasonable reasons for) my worry about things that hadn't yet happened and were unlikely to happen, often involving complicated lines of reasoning which hid various fallacies. I figured out why I was worried about those things in the first place (which funnily enough wasn't clear to me at the time), and how experiences bias thought and perception of likelihood. All this involved eating a lot of humble pie, and resulted in a greater understanding of people around me and why they behave they way they do too.

The second most helpful thing was learning to cope. There's probably stuff that's out of your control (e.g. stuff that happened in the past that's still affecting you, or just plain old facts of life) that you can't change. It's not that therapy magically fixes your problems per se, but perhaps you're making it worse for yourself: e.g. shame, embarrassment, self-blame, over-worry. At the risk of sounding woo-woo, Buddhists talk about the "second arrow" which is the unnecessary suffering you inflict upon yourself on top of the suffering that you already have to live through - that rings true for me.

Like any skill, you practice and learn through failure how to apply these things. I still worry occasionally, but I can now much more easily take a step back, evaluate, separate out what I can do and can't do about it, formulate a plan, ask for help, etc instead of turning into a nervous wreck who cannot do a thing. I now mostly worry the correct amount, which is the small amount that pushes me to do something productive about it :-)

I imagine this all sounds very abstruse if you don't have this problem and already worry the correct amount, but I'm writing this so that others in a similar situation and have an inkling that something's off (e.g. "why am I always so angry"), but haven't yet discovered what's going on can benefit. If anyone has questions about this, feel free to reply and we can chat privately.

Don't know why this comment was downvoted - it's the truest take here I can spot. The fact is that the factors that make one versus the other more preferable (the state and quality of frontend / backend tooling and environments, compute power and rendering capabilities of servers vs clients, round trip time cost vs responsiveness requirements etc etc) are continually changing over time and that's what's causing the back and forth swing, but lessons from previous iterations are generally learned.

I wouldn't be shocked if we sooner or later saw language-level support (think of something like Elm, improved) for writing "just" code and then later marking up which parts execute where, and the communications and state synchronization crud and compiling down to the native language is just handled.

Noticing that the above stream of consciousness might have come across as crotchety - that was not my intention! A better way to phrase this might have been "the benefits of memory safety and human conveniences of a modern language can be worth giving up a bit of control, even in cases where it might be legitimately beneficial", which was supposed to be praise for the Rust model :-)

> If you’re using a higher level language, you’re not going to have access to how the memory is laid out. A simple change, like removing indirection in our list, was an order of magnitude improvement in our latencies since there’s memory prefetching in the compiler and the CPU can anticipate which vectors are going to be loaded next in order to improve the memory footprint.

This is a common experience and I'm still surprised by the choice I constantly see to use a managed-memory languages to build a database - one of a very small set of special cases where having full control over the memory layout might just be a reasonable thing to want. In this universe (absent doing something completely absurd) it's not algorithmic complexity but managing data locality in the cache hierarchy (e.g. reading things from L3 vs main memory vs disk) that makes things orders of magnitude faster, especially if you're in the realm of doing things like SIMD operations to speed things up.

Perhaps there's some level of suck we're willing to tolerate for all the other benefits you get, but I've been noticing a pattern of "align things just so at the higher level and hope they mostly turn out the way you want at the lower level" (e.g. also with the Apache java-y databases like hadoop / hbase / cassandra which I guess were mostly supposed to derive their total throughput from massive scale rather than per-node performance) which is a bit funny.

But also it seems like part of Rust's promise was "low level but make it high level" which seems to be succeeding (zero-cost abstractions and whatnot), so I imagine this will get better over time - having not attempted a project like this myself, I'm not sure what the limitations you'd run up against are in terms of laying things out in memory in a favorable way - I imagine the kind of massive manually managed arena allocations and ad-hoc pointers going everywhere that one normally does doesn't really fly.

Funnily enough I have a pet peeve with Grafana's picker (which is overall quite nice) - I can't copy and paste the whole date range from one window to another in one go. I'm often punching in varies specific datetime ranges across multiple dashboards so this comes up all the time every day. Sure I've added data links for the cases that I use most frequently but I can't do that for everything. The "recently used ranges" (which is a neat feature) doesn't seem to work across dashboards and seems to require a page reload which baffles me a bit.

MySQL approximates a hacky version of a similar idea with the bufferpool (newest additions don't go to the top of the LRU so they age out faster if not re-accessed) to avoid random full table scans from thrashing the cache:

https://dev.mysql.com/doc/refman/5.7/en/innodb-performance-m...

You are of course correct that those features are extremely useful and that the JDK authors foresaw these situations long before they were a big deal. But also you do note that it's not widely used which is also correct and that's the important bit. I think it would be more accurate to say that python and javascript are discovering (perhaps not fully intentionally) ways to implement these ideas that are suitable for mass community adoption.

IMHO it's like the difference between basic research (mostly not very useful on its own but extremely important) and public health in medicine (the thing that actually makes a massive impact on the world that leverages the former but at a slower and more controlled pace). To stretch the analogy even more any doctor worth their salt will tell you that a treatment plan is only as good as the likelihood that the patient will stick to it. Other things like let's encrypt (https everywhere) and signal/whatsapp/telegram (strong privacy enabled by default) come to mind as implementing ideas on things that experts had been handwringing for decades that people "should" be doing that are now done so well that it feels almost silly not to do them the good way.

Is this the one with the Devo soundtrack?

chiptunes: https://www.youtube.com/watch?v=Ib8IYfcOnWU original: https://www.youtube.com/watch?v=Nf8OmnRIBvA

> Of course this has nothing to do with what system design questions (most) interviewers care about. But this is the system design that matters on the job.

Sigh.

I just think it's so absurd that "system design" is now an interview category that's expected of new grads. I'm talking about the fanciful form of this, where the bright eyed and bushy tailed new grad is somehow working on a problem that is the equivalent of the Large Hadron Collider or the Apollo space program.

I've sat on the other side of these interviews, and what I'm hoping for, especially from beginners, is just some common sense, and some reassurance that they can identify at least the main design tradeoffs and make appropriate decisions based on the information available. And then we look at what should happen to the design of our system if I modify our assumptions, or add new use cases that create new forms of stress on the system.

What I get instead is stuff like "I'll use a key value store", and when I ask why, I get "because it scales better", and when I ask "compared to what, why, under what conditions", I get blank stares, because they've barely used either. Then I learn that they said a thing because they read some github repo with 100 system design questions that said so. Or they read a prep bible that told them this is how Google does things. Or they read an actual book about a subject (a bit better) but didn't really absorb the ideas (which is fine! happens to me too!), but wanted to use the thing they learned about anyway.

What I actually wanted was "because the data is simple and non-relational and this will suffice for the foreseeable future, and features XYZ help remove complexity from the app" or "this is a single-purpose system designed to do one thing very well, and we can pack a lot more storage / throughput per node this way" or "Never mind, the requirements and the future are fairly vague, I'll just use something I understand well, is flexible, well supported and easy to migrate out of" or ANY sort of sense that we do things because of reasons, and we don't do things if we don't have good reasons to do them.

I remember being a beginner too - not knowing things and inexperience is perfectly fine, the point is we want you to do your best with what you have. It's fine to be honest about your knowledge and capabilities. You can look silly otherwise.

What are fairly nuanced, philosophical or "it depends on ..." questions to me after two degrees and almost a decade and a half into my career, for some of these folks is "of course it's X", and I'm just flabbergasted at the level of confidence that's projected. Please just a) exercise judgement b) based on the facts available c) leveraging your existing knowledge of core software concepts. Try a design, say why you did each thing this way, say what you like and don't like about it, say what you're uncertain about, what a few alternatives would look like, what you don't like about those, and what you'd change if you had a bit more information. This is the kind of "how do I do this" thing everyone building anything should be doing every day.

It's not impressive for you to be able to regurgitate the X types of cache policies according to wikipedia. It is, however, impressive when I point out a potential problem with the way you set up your caching scheme, and so you look at what you designed, actually think about what that means, and are able to tweak it to better fit the requirements. I don't mind whether you're aware that you just re-discovered a thing that already has a name.

My plea to whoever is posing and judging these system design interview questions in this way causes young interviewees to think they need to participate in a kind of veteran staff engineer cosplay tech-braggadocio circus: for god's sake, just test what's required for the job, please and thank you.

Yeah, that's caching, and it's great when one can do it :-)

edit: kidding aside, this is approximately what a database view is, at least in some implementations (though probably not as common as the more simple implementation): a view is created as a SQL query that pulls some data. It's stored as is. As data is inserted to the database, the views (a.k.a. queries) that would include that row are updated too.

This is a great example of how as technology changes, it changes use cases, which can prompt a revisiting of what was once considered a good idea. You'll often see the pendulum of consensus swing in one direction, and then swing back to the exact opposite direction less than a decade later.

2010s saw REST-conforming APIs with json in the body largely as an (appropriate) reaction to what came before, and also in accordance with changes around what browsers were able to do, and thus how much of web apps moved from the backend to the front.

But then, that brought even more momentum where web apps started doing /even more/. There was a time when downloading a few megabytes per page, generating an SVG chart or drawing an image, interacting to live user interaction was all unthinkable. But interactive charting is now de facto. So now we need ways to access ranges and pieces of bulk data. And it looks a lot more like block storage access than REST.

---

These are core database ideas: you maintain a fast and easy to access local cache of key bits of data (called a bufferpool, stored in memory, in e.g. mysql). In this local cache you keep information on how to access the remaining bulk of the data (called an index). You minimize dipping into "remote" storage that takes 10-100x time to access.

Database people refer to the "memory wall" as a big gap in the cache hierarchy(CPU registers, L1-L3, main memory, disk / network) where the second you dip beyond it, latency tanks (Cue the "latency numbers every programmer should know" chart). And so you have to treat this specially and build your query plan to work around it. As storage techniques changed (e.g. SSDs, then NVMEs and 3d x-point etc), databases research shifted to adapt techniques to leverage new tools.

In this new case, the "wall" is just before the WAN internet, instead of being before the disk subsystem.

---

This new environment might call for a new database (and application) architectural style where executing large and complex code quickly at the client side is no problem at all in an era of 8 core CPUs, emscripten, and javascript JITs. So the query engine can move to the client, the main indexes can be loaded and cached within the app, and the function of the backend is suddenly reduced to simply storing and fetching blocks of data, something "static" file hosts can do no problem.

The fundamental idea is: where do I keep my data stored, where do I keep my business logic, and where do I handle presentation. The answer is what varies. Variations on this thought:

We've already had products that completely remove the "query engine" from the "storage" and provides it as a separate service, e.g. Presto / Athena where you set it up to use anything from flat files to RDBMSs as "data stores" across which it can do fairly complicated query plans, joins, predicate pushdown, etc. Slightly differently, Snowflake is an example of a database that's architected around storing main data in large, cheap cloud storage like s3, no need to copy and keep entire files to the ec2 node, only the block ranges you know you need. Yet another example of leveraging the boundary between the execution and the data.

People have already questioned the wisdom of having a mostly dumb CRUD backend layer with minimal business logic between the web client and the database. The answer is because databases just suck at catering to this niche, but nothing vastly more complicated than that. They certainly could do granular auth, serde, validation, vastly better performance isolation, HTTP instead of a special protocol, javascript client, etc etc. Some tried.

Stored procedures are also generally considered bad (bad tooling, bad performance characteristics and isolation, large element of surprise), but they needn't be. They're vastly better in some products that are generally inaccessible to or unpopular with large chunks of the public. But they're a half baked attempt to keep business logic and data close together. And some companies had decided at a certain time that their faults were not greater than their benefits, and had large portions of critical applications written in this way not too long ago.

---

Part of advancing as an engineer is to be able to weigh the cost of when it's appropriate to sometimes free yourself from the yoke of "best practices" and "how it's done". You might recognize that something about what you're trying to do is different, or times and conditions have changed since a thing was decided.

And also to know when it's not appropriate: existing, excellent tooling probably works okay for many use cases, and the cost of invention is unnecessary.

We see this often when companies and products that are pushing boundaries or up against certain limitations might do something that seems silly or goes against the grain of what's obviously good. That's okay: they're not you, and you're not them, and we all have our own reasons, and that's the point.

FWIW the scientific computing community (who often deal with petabytes of geodata) has been thinking of ideas like this for a while, e.g. techniques around file formats that are easy to lazily and partially parse, (ab)using FUSE to do partial reads using http RANGE requests, some combination thereof, etc:

http://matthewrocklin.com/blog/work/2018/02/06/hdf-in-the-cl...

> Problem is that academics are rarely experts at programming or have knowledge of computer architectures as much as someone in the industry. There are various tricks that are never taught at college, therefore academics have no idea some stuff even exists.

I want to push back on this generalization a bit. The academics that are focused on pushing the mathematical boundaries of discrete optimization are focused, no surprise, on only that. If theoretical improvements is not what you want, don't read those papers, read ones what people more focused on applications. Read literally any databases paper, or stuff like hyperscan, simdjson. I'd argue that a non-trivial amount of these are vastly /ahead/ of what's standard in industry, but industry is slow to adapt new ideas and catch up because of legacy software and existing codebases. Very similar stuff in the programming languages sphere, it took ages for industry to adapt ideas that were popular in academia for a long time (eg: Rust, LINQ). The idea that academia (at least in CS) is an ivory tower, far from the concerns of industry, is not very true as of recent. There's a lot of cross pollination and a revolving door of people going back and forth from one to the other spreading ideas.

Most of this article tracks quite well with my experiences! It's nice to see stuff like this written down.

> If the application’s code is in our control we can cache data in a memory store like Redis or Memcached to avoid querying the database.

On this point - what I've seen over and over again is caches being used as a band-aid for overall throughput limitations. Then the cache goes down (sometimes partially in the case of a clustered cache), the database gets slammed, and there's an outage soon after.

So caches help as an improvement on latency to some degree, but using them to avoid scaling your capacity can get you in a pickle depending on your usage patterns / hit distribution. You usually need some amount of capacity planning and consideration there.

Also to spread out over time the amount of people needing medical help. Perhaps a city can't handle 10k new patients in a day but it could handle it over the course of 2 weeks.

I'd ask the opposite question - at what scale would you want to have your own custom setup rather than RDS? Managing your own database infrastructure for workloads other than "a few queries a second" is hard work with a lot of pitfalls, and you better be at a size that there's some benefit (high levels of customization, use case specific tuning, economies of scale, etc). As a person who does exactly this for a living, I'd rather shell out for RDS or a similar offering than my own setup most of the time. Especially at first, before you discover what exactly you /don't/ like about it or what you'd want different.

(disclaimer - worked at Kayak a long time ago)

A big part of what made Kayak good was that it was (is?) a solid data product - basically a gigantic farm of scrapers and crawlers glommed together with direct-from-provider (airline, hotel etc) data ingestion pipelines and coherence protocols that flowed into massive unified local cache databases. All this to ensure that you, as a customer, saw the most up to date and widest variety of information possible.

When Kayak IPO'd, it was around the time that ITA software (massive airline data provider you've never heard of) got sold to google, and there was a lot of hand wringing over whether this'd spell the end for Kayak. But even then it looks like ITA supplied only 42% of flight data (check out the S1 filing https://www.sec.gov/Archives/edgar/data/1312928/000119312510...), the rest from other data providers (Amadeus etc), online travel agencies, or individual airlines.

So given that, Hipmunk's focus on "we're gonna make the UX so much better" was admirable (especially given the state of a lot of the competition), and it pushed everyone to do better in that regard, but always felt a bit naive as a business strategy to me, since I think most people will pick the site with the slightly worse UI and better / cheaper / non-stale flights rather than the other way round. Especially in a non-premium race to the bottom industry like online tickets. And getting quality data takes a big team and lots of money and infrastructure, as well as lots of business relationships.

But I always thought they were a cut above a lot of the other competitors which were bad UIs mangled with dozens of ads thrown together in a slapshod way on top of the same 1-2 data providers.

Fellow WPI grad, in 2012 I paid 275/person/month for an admittedly crappy apartment that I shared with 4 people, and later on 475/person/month for a room in a nicely renovated 3 story house right next to campus shared with 5-6 people. This one didn't have that many remnants except for an ironing board in the kitchen that plopped out of a wall closet, and a button next to the stairs indoors that said, ominously, "Sedgwick". It seemingly did nothing. We never figured out who or what Sedgwick was. We made sure to press it every so often, to "feed the Sedgwick" :-)

"Can I finish before you cut in?" directed at the interrupter seems to work for me - it's a reasonable request, it explicitly calls out what the other person is doing in a way they won't / can't reasonably get defensive about, and if they keep talking over you it telegraphs to other people that they're being a jerk.

It's annoying and probably unfair to have to do this of course, and honestly takes more practice than I thought it would to recognize this situation is happening and remember to deploy it.

More generally, this is now something I try to pay attention to while interviewing at companies. I don't need to be in a struggle every time I have a conversation or make a point.

I like to say that Matlab was designed by applied mathematicians who don't want to care about software engineering, and python+numpy+scipy was designed by software engineers who don't want to care about applied mathematics :P

I don't know that that's strictly true but it gives you a general idea for why things are the way they are.

I have two questions:

1) From the POV of the patient, how bad / invasive is getting a biopsy? Wouldn't you always want the low-risk biopsy over having discovered a problem too late? From a public health / insurance perspective, I get that you're trying to lower costs and time wasted, which are sadly conflicting goals :-(

2) It's always been so shocking to me that the reaction to "we get false positives" is not "we need to pour money for research into much better diagnostic techniques yesterday" and instead it's "well I guess we'll just not look then".

I think this is a very neat tool for educational purposes - sort of like python tutor!

I've got to say that I've always wanted to love hangouts but it always makes my laptop (reasonably powerful macbook pro) immediately max out CPU and make fan noise like a spaceship taking off, and for comparatively worse AV quality than the competitors. The only thing that kept me using it was that I didn't have to install crap. Just used meet for the first time yesterday, reserving judgement there, but honestly I didn't need yet another rebrand from google.

Zoom is the exact opposite: You have to get the client but it client works with minimal resources, on many platforms (phone / laptop is seamless for me) scales perfectly with large meetings / screenshare and video at the same time, laggy mobile connections etc etc. Most importantly it ALWAYS works. I never have to futz with it ever (whether in terms of connections, or "your link doesn't work", "you have to invite me", "user is from a different organization than you" etc). Just get the meeting ID and connect. This is how technology should work in 2018.