gt50201

what's staff+ comp nowadays?

How do you differentiate background and worker processes? I’m currently running VMs for background processes processing pub sub topics. Is that not something that could go on cloud run?

In a previous company we were scraping customers web pages to load customer and product information because the number of systems and teams involved to connect to their CDP would take months to get up and running. A few problems we had to solve was the web pages changing underneath us and some pages not being formatted the same. We ended up having to detect those changes server-side to alert us to update the scraper. How do you all deal with these use-cases?

Also, we tried to leverage: https://www.diffbot.com/ but the lack-of-accuracy/lack-of-complete-data + cost never justified it's usage.

Oh! That’s super interesting — what are some examples of this happening?

I think that while it is a common situation in small/scrappy companies, unless you want to get good at scrapping together tech solution, this may not be the best experience for you right now.

I was fortunate enough to start at one of the big four out of school and work on some cool projects. I eventually went to work at a few startups, where it was more about being a good plumber (similar to what you are doing) than being a good coder. Many of the junior engineers got burnt out by trying to build and create solutions (where do I make algorithms?!?) around these 3rd party systems but the truth is, that wasn't what the business needed. Since I had already had my fix for creating some complex software, I did okay. I probably stayed at certain startups too long but I did get better at knowing when to build vs buy and how to build+buy+glue.

What helped me develop that knowledge was actually spending years building so I knew the cost of design/implementation/support/extension.