dilatedmind

They are the same, there is also some functionality for managing firebase auth users in the gcp console.

philly school district schools are 14% white. the school district i went to in the philly burbs is 79% white.

philly school district is in a bad place financially. https://www.inquirer.com/news/pa-school-funding-trial-philad.... A district with more money will have more resources for gifted programs.

I imagine this has it's roots in the demographic and population shift the city has seen starting in the 50s. Philly's population in 1990 was 75% of what it was in the 50s. I'm not an expert in this area, but I'm sure there was overhead in maintaining infrastructure, paying pensions, etc as the population shrunk.

At this point, maybe the federal government should just bail out city school districts in this situation. Why should an underfunded school district be paying a chunk of its budget on debts?

for this specific example, I think the shared library is not the correct approach. Queues work for simple fifo behavior, but in this case they also need fairness (and perhaps other logic, like rate limiting per client, different priority for certain clients etc)

For example, "Customer-A is clogging up the work queue and starving other customers out". The solution to this could look something like linux's completely fair scheduler, where every client is allocated some amount of messages per interval of time. This means messages need to be processed in a different order then they are enqueued, and queues are not good at reordering messages.

I would suggest implementing the queue abstraction as a rest or grpc service, backed by a database like postgres, which holds message payloads and everything related to message state (in progress, retry after times, etc). Now we can implement all the necessary scheduling logic within our queue service.

Thoughts on a couple of your points:

- we don’t need any kind of backwards compatibility, we just update everything.

if you don't care about backwards compatibility, then you can stay on v1 forever. Have you considered a monorepo? That would simplify updating packages and give you the behavior you want.

- For the client to update, it’s not a simple path change in go.mod

if a package moves from v1 to v2, there are breaking changes in either api or behavior. I think this implies more than a simple change to go.mod. This also allows importing both versions of a package if necessary.

depending on how you define transaction, this doesn't seem possible?

my approach has been to make all operations idempotent and ensure they are all ran at least once.

I worked on a project which required some medium scale web scraping (less than 100 million pages), and went with node primarily because of puppeteer.

The system had a couple dozen worker processes doing the scraping, and one coordinator which maintained a queue of pages which needed to be scraped. There was some logic to balance requests between sites, so we weren't making more than a request/s to any in particular. The coordinator just had a rest api endpoint, which the workers would hit to get their next job and to return w/e data.

Each worker process was ran on a separate aws instance, believe it was a t2 with unlimited cpu enabled. These are only a few dollars a day, and it was necessary to have as many ip addresses as possible (at least 5% of the sites we were scraping had some preventative measures in place, but they all seemed to be ip based)

I think he's making fun of his past self, thinking he knew everything when he was younger.

I agree this can be awkward, especially if you let these constructs propagate through your codebase and database. However, if a string or int can be null, then all strings and ints are essentially pointers, so you've just introduced this construct everywhere.

A couple things I have tried:

- hope default values align with your business logic, eg an empty string isn't a valid name and 0 isn't a valid age.

- for partial updates, populate the existing values before unmarshalling, then unmarshal on top. Missing fields in the json won't overwrite the existing values

- unmarshall into a map[string]interface{}, which gives you the semantics you want.

i would suggest bias your implementation against false negatives. They can always come back and update it if it's wrong, and their url could just as easily be "valid" but incorrect, eg any typo in a domain name.

if it's really important, you could try making a request to the url and see if it loads, but that still doesn't validate its the url they intended to input.

might be cool to load the url with puppeteer and capture a screenshot of the page. if they can't recognize their own website, it's on them.

I think this article misses two important points on siloed qa teams

1. Qa doesn’t know what can be covered by unit or integration tests

2. Since they treat our code like a black box, they may create permutations of tests which which cover the same functionality

Maybe this is part of the draw of having a qa team. Feature coverage rather than code coverage. The downside is this can create a huge number of expensive to run manual tests which may be hitting the same code paths in functionally identical ways.

The tooling for automating manual tests of web apps is almost there: puppeteer, recording user inputs and network calls, replaying everything and diffing screenshots.

Since qa tests are tied to features and not code, There’s also the problem of having to run all qa tests even if you’re releasing minor code changes. My build tools are smart enough to return cached results for unit tests whose dependencies didn’t change, but there’s no equivalent for qa tests.

cool, so this is like warby parker's app for renewing prescriptions?

that really saved my ass recently, when my glasses broke and I found my prescription was expired. Apparently, they let you use an expired prescription up to 18 months, and mine had expired like 20 months ago.

Their app worked, but was pretty painful. Required holding my phone 14 feet from my monitor- had to move my desktop to a different room to get the app to approve.

how significant is 1 if you are using cloud sql proxy? My understanding is that the proxy tunnels traffic over an encrypted connection, so there is no benefit to adding an extra layer.

where would you buy this chocolate?

google shopping and amazon don't turn up any results

maybe i did something weird last time i installed ubuntu, but my user is 1001:1002 and the default ubuntu user is 1000:1001

would have been useful for something we ran into at a previous job. For legacy reasons large csv files had been zipped instead of gzipped (or some other streamable format), and were causing some process in our data pipeline to consume too much memory.

I wound up coming up with a quick solution that appeared to work. Taking advantage of being able to get s3 objects at arbitrary ranges, I implemented seek and read on top.

my intuition is you can save cpu time when compressing before sending over the network (and certainly wall time).

a quick test copying a 24M file (with similiar compression ratios) to s3 showed a 6% decrease in cpu time when piping through gzip.

did you use any kind of message framing? Got bit by this at a previous job where we needed to change the message format to improve compression. Wound up figuring something out, but would have been easier if we had reserved a byte for versioning.

The best use case I've had for partitioning tables is when there is a lot of churn that would otherwise result in significant vacuuming. For example, a table holding the past 7 days of data would require deleting 1/7 of its rows every day.

Swapping out a child table and dropping the old one is free in that sense, but does require an exclusive lock which can cause issues if you have long running queries.

interesting, how did the spend breakdown between cloud run and firebase?

did you have any limit to how many req/s you made to an individual site? It seems this would be difficult to implement with this architecture.

how did you deal with following links in circles/ avoiding scraping the same page multiple times?

I had built something similiar at a previous job, recursively scraping ecommerce sites. The first thing I noticed was some of the sites we were scaping couldn't handle more than a couple requests a second (in particular as we scaped uncached pages by sites running php). Other sites were quick to ip ban.

I kept things simple, a few dozen micro instances on aws (think they were like $3 a day) running puppeteer. A single server acting as a controller, keeping a per site queue and allowing us to set per site request limits if necessary. All the state of which links were already seen just kept in memory. Of course everything was also persisted to a db, and if the controller process needed to be restarted, it could restore the queue/ seen state and resume.

I think it was already strongly consistent within regions?

like if you had tried to read a non-existent key, then wrote to it, it might continue to appear to not exist for a minute?

would it have been sufficient to alert on high memory usage? It might be reasonable to set an alert on say 70% rss. As long as the pod does not pass this threshold and die before a metric can be sampled.

that "no such file or directory" looks to be coming from building a dynamic executable on debian and trying to run it on alpine.

Also this, and maybe this is where social background comes into play. If my parents didn't value education, they wouldn't have moved us to a good school district or taken the time to look over my homework and assignments.

It's obvious the SAT does not measure native intelligence independent of education. I attribute my success in the SATs to the excellent teachers I had in high school. I didn't study at all and found that everything on the test had been covered by the classes I was already taking.

As for social background, yeah, the wealthier you are, the easier it is to move to a good district. Yet, within a single school district, what difference would social background make?

> Over the years, we have tried all tricks of the trade within Go. Using sync.Pool, maintaining our own freelists, avoiding allocations on heap wherever possible, using buffer arenas and so on

I'm interested to know why pools/freelists didn't work out for nodes

their readme there states

> GitHub is committed to continuing to offer free public repository services to developers with individual and organizational accounts in U.S.-sanctioned regions. This includes limited access to free services, such as public repositories for open source projects (and associated public Pages), public gists, and allotted free Action minutes, for personal communications only, and not for commercial purposes.

maybe someone can confirm how this works in practice?

I could be reading this wrong, but it looks like github allows open source contributions from sanctioned countries.

https://docs.github.com/en/free-pro-team@latest/github/site-...

tangentially related, but have you ever had to deal with statistics on a table indexed on (x, date), where there are a few million rows added each day, and x could be [0,1000], with a distribution like 80% of rows owing to only a few values of x.

ran into a situation like this where after enough days of data had accumulated, postgres would eventually fall behind on updating stats, such that a week could lapse without stats being updated, causing the query planner to think no rows existed within that time range. This would result in a nested loop instead of a more efficient hash join, causing a query to take 2 hours instead of 2 seconds.

increasing the number of rows sampled with set statistics didn't seem to help. wound up running a cronjob to inspect pg_stats, and manually running analyze when enough days had lapsed without most_common_vals being updated.

Interesting that you compare them to mixins, which the react team considered harmful and removed.

What problems do hooks solve better than composition or hoc?

I really haven’t kept up with react, when I last used it 6 months ago I started with the same boilerplate I made in 2017. Could it be things are being added to react that are unnecessary for most small and medium sized apps? I got that feeling when I saw react router 4. At least v6 is moving back towards a simplier api.

His asyncpg example doesn't make much sense to me. What if there was a config change with a bad password? I would like to know this immediately on startup, else my rolling deploy is going to bring down all the previously well configured instances, and by the time we lazily try to connect to postgres it's too late.

I'm not a big python user, but I do find it kind of surprising there isn't an awaitable and thread safe mutex in the stdlib.

Well put. The one time i expressed interest but reluctance at leaving my previous job, I was invited on a coffee and chat with a couple lead engineers to help decide. We got along well, talked about technical things, swapped stories about problems solved. Afterwards, i got the impression going through the technical interview would be more of a formality.

I think part of the problem is not everyone is good at giving interviews. But most software engineers can talk about their work once you get them going.

FWIW, I think white boarding interviews are pretty fun.