Our use case requires real-time inserts as events happen.
dbt is as I understand it for batch processing transformations on a set schedule.
960 karma · joined March 27, 2007
dbt is as I understand it for batch processing transformations on a set schedule.
It works well. Their team is great. I feel a bit spoiled having had as much access to the engineering team during the private beta as we've experienced.
It's great for use cases where it makes sense to sync postgres tables across to clickhouse without denormalizing them. PeerDB can transform rows in a single table sent via CDC using a lua scripting language, but it can't (yet!) denormalize data into clickhouse that is stored in 3NF on Postgres across multiple tables.
On the clickhouse query side, we end up wanting denormalized data for query performance and to avoid JOINs. It's frequently not a great idea to query in clickhouse using the same table structure as you're using in your transactional db.
In our experience we sync a few tables with PeerDB but mostly end up using app-level custom code to sync denormalized data into Clickhouse for our core use-cases. Most of the PeerDB sync'd tables end up as Clickhouse Dictionaries which we then use in our queries.
PeerDB works well and I like it for what it is. Just don't expect to be satisfied with querying in Clickhouse against the same table structure as you've got in Postgres unless your data size is tiny.
Curious to know about how others are using it and the architectures you've developed.
* Poor visibility into detailed metrics, especially when problems happened in the render load balancer / routing mesh. We had a specific issue where a small number of requests were failing somewhere in render's infrastructure before reaching our application, and at the time there was no visibility to allow customers to know about requests that timed out or failed within render's infrastructure rather than our application. We collaborated with your team to surface and replicate the issue, but it was frustrating. I had a very good set of conversations with a product manager on your team about what we needed and why it was important in early 2024.
* At the time, the hosted postgres implementation was immature. I think this is an area you've already improved dramatically.
* Maybe you could add support for something like AWS PrivateLink so customers can run parts of their workloads on AWS securely over a private network. This would be a neat way to allow customers to stay on Render longer as their needs grow.
We load data from postgres tables that are used to build Clickhouse Dictionaries (a hash table for JOIN-ish operations).
The big tables do not arrive via real-time-ish sync from postgres but are bulk-appended using a separate infrastructure.
DuckDB is neat, and I understand why a company like BemiDB would build their product on top of it, but as a prospective customer embedded databases are a weird choice for serious workloads when there are other good open-source solutions like Clickhouse available.
Daisychain is building a modern platform for effective organizing. Our customers include progressive political campaigns, grassroots organizations, and ambitious non-profits.
We’re seeking a software engineer who is excited about using their talents to deepen the impact of the causes we work with (we're particularly focused on impacting the 2024 US elections) while having an opportunity to shape the technical direction of our product.
Send a note to hello@daisychain.app with questions or apply at: https://www.daisychain.app/jobs/software-engineer
https://en.wikipedia.org/wiki/Morgantown_Personal_Rapid_Tran...
How all this will play out and be interpreted by regulators is interesting and currently hard to see how everything will be reconciled between trade in digital services, the US national security state, and the fundemental rights guaranteed to EU citizens.
Google has been running on 100% renewable electricity since 2017. Microsoft has reached that milestone as well, but I can not find a date for when they achieved it.
AWS aspires to eventually reach that goal by 2025. One of the clouds is dirtier than the others.
Unfortunately, it's all lofty language and Amazon lags behind its peers on real action.
For the AWS cloud business Amazon lags far behind its peers at Microsoft and Google. Of the three main public clouds AWS is the only one still using coal power (coal is a big part of the power mix for the grid used by their largest point of presence in Northern Virgina). Microsoft and Google have run their data centers completely on renewables and have done so for years.
We're stuck on us-east-1 in Northern Virginia for legacy reasons and to make up for the dirty way that Amazon runs its cloud we buy feed-in RECs for the grid where our AWS instances run. AWS could be doing this themselves (there are RECs available! we're buying them!) to help jumpstart the transition to renewables in the energy markets where they operate but they're simply choosing not to spend the money.
Microsoft and Google deserve credit for their work in this area and they're doing a much better job. It's just too bad that AWS is a better technical product for our workload.
The trouble is that as a small business we can't afford to have two separate operations teams for the US instance vs. the EU instance. We're all based in North America too and it is not practical for us to hire a whole separate devops team for Europe.
Our US based engineers could in theory be compelled to hand over the data stored at rest in the EU. They could also in theory see PII like names or email addresses in the course of administering the application on their laptops in the US which counts as data export, so would still need Privacy Shield or now SCC to allow engineers to do their everyday work keeping the product up and working.
The lunch is not free. You're buying it for us.
They made a good faith effort to find non HRC campaign and non profit customers after the election but never found meaningful market fit. It was an odd assortment of tech that felt like what it was: an attempt to productize an enormous consulting engagement to build whatever the campaign wanted.
It's all an interesting campaign finance loophole though. You can fund money loosing tech companies to work with candidates and as long as they charge the campaign something, it's an interesting way to spend huge amounts of dark money on elections.
Are they only valid if we are not an existing customer of the partner, or is it for any spend with those vendors we put onto the card?
Do we work with the Partner to get that discount or is it automatically applied on the Stripe side while processing charges we make to the card?
It was also really frustrating to us doing work in the political and advocacy sector that the their EU GDPR Data Processing Agreement explicitly forbids using their product for any of the things our EU clients would need like measuring petition signatures or donations to causes -- since those might be information about "political opinions" which their terms prohibit. This particular GDPR interpretation is somewhat unusual, we've negotiated workable DPAs with almost everyone else we've wanted to do business with but Segment was unmovable.
They will even mail you a copy in exchange for a subscription. It's magical.
This is why Trump's refusal to divest himself from his businesses is an unacceptable conflict of interest even if he insists it is fine and there are no grounds to question it.
Anything political or critical of YC already gets disappeared from this site quite quickly. I had assumed that this policy (which always seemed misguided to me) was already in effect.
I think the historical moment we find ourselves in is a time to make ourselves more uncomfortable rather than retreat and pretend that the only things that matter are software and how we might make more money on the internet.
Politics is about coming together to find common solutions to problems and make sure that no one gets left behind. Isn't it the job of a responsible community (even a VC sponsored one dedicated to making money with tech as HN is) to lean in when things start to feel hard rather than tune out and ignore our responsibilities as citizens and fellow humans?
Sam Altman spends his summer advising vote.org in his role as a YC Partner.
Soon after demo day, Sam Altman decides to launch what amounts to a clone knock off of vote.org. Paul Graham tweets about VotePlz at launch calling it the most important thing to happen today, after never mentioning the YC funded vote.org on his twitter all summer as far as I can tell.
Shouldn't YC be as friendly to not-for-profit startup founders as they claim to be to for-profit founders?
I'm all for there being lots of voter-registration organizations. YC should fund lots of startups in this space.
It seems though that the role that YC Partners play as trusted strategic advisors is incompatible with those partners going on to start directly competitive organizations, whatever their tax status might be.
I'm so glad you're killing a product. My complaint has always been that hashicorp was doing so many different products it was unable to actually deliver any of them at high quality.
Packer is a reliable and trusty piece of tech for us, but terraform (which has so much potential) is also so rough around the edges we've wrapped it in a mess of our own tooling to make it somewhat usable.
I hope you can find a way to focus on one or two things and do them well.
Engineering support is great and responsive through IRC but, stripe engineering staff there just shrug if you ask a business question.
We're looking for a devops contractor to build a long term relationship with. We build organizing software for progressive not-for-profit organizations: http://www.controlshiftlabs.com/
Experience working with Terraform, Packer, Chef, Amazon Web Services.
Looking to improve our automation, security and reliability as we grow.
We're a globally distributed team so you can be located anywhere that allows you to overlap with normal business hours in the US east coast.
Send me a note with a cv and why you'd like to work with us. nathan@controlshiftlabs.com