HNHacker News
TopNewBestAskShowJobs

exAspArk

155 karma · joined September 14, 2016

submissionscomments
exAspArk··on Show HN: BemiDB – Postgres read replica optimized for analytics
Thank you!

The pg_analytics Postgres extension partially supports different file formats. We bet big on Iceberg open table format, which uses Parquet data files under the hood.

Our initial approach is to do periodic full table resyncing. The next step is to support incremental Iceberg operations like updates. This will involve creating a new "diff" Parquet file and using the Iceberg metadata to point to the new file version that changes some rows. Later this will enable time travel queries, schema evolution, etc.

exAspArk··on Show HN: BemiDB – Postgres read replica optimized for analytics
I agree that DuckDB may sometimes be buggy because it's being actively developed with a large surface area (a universal embeddable query engine that works with different storage layers if I were to simplify).

However, DuckDB (non-profit foundation) != MotherDuck (VC funded). These are two separate organizations with different goals. I see DuckDB as a tool, not as a SaaS or a VC-funded company. My hope is that it'll be adopted by other projects and not associated with just a single for-profit company.

exAspArk··on Show HN: BemiDB – Postgres read replica optimized for analytics
Yes, we want to use the approach like you described! We'll likely wait until enough changes are accumulated by using 2 configurable thresholds: time (like 30s) and size (like 100MB)
exAspArk··on Show HN: BemiDB – Postgres read replica optimized for analytics
We ran some benchmarks (TPC-H, designed for OLAP) with ~10M records https://github.com/BemiHQ/BemiDB#benchmark

The BemiDB storage layer produced ~300MB columnar Parquet files (with ZSTD compression) vs 1.6GB of data in Postgres.

exAspArk··on Show HN: BemiDB – Postgres read replica optimized for analytics
For now, BemiDB supports only full Postgres table data re-sync. We plan to enable real-time data syncing from Postgres into S3 by using logical replication (CDC), which is much more reliable than PG NOTIFY.

We use logical replication and this exact approach with our other project related to auditing and storing Postgres data changes https://github.com/BemiHQ/bemi. We're thinking about combining these approaches to leverage scalable and affordable separated storage layer on S3.

Lmk if that makes sense or if you had any more questions!

exAspArk··on Show HN: BemiDB – Postgres read replica optimized for analytics
Thank you!

Yes, absolutely!

1) You could use BemiDB to sync your Postgres data (e.g., partition time-series tables) to S3 in Iceberg format. Iceberg is essentially a "table" abstraction on top of columnar Parquet data files with a schema, history, etc.

2) If you don't need strong consistency and fine with delayed data (the main trade-off), you can use just BemiDB to query and visualize all data directly from S3. From a query perspective, it's like DuckDB that talks Postgres (wire protocol).

Feel free to give it a try! And although it's a new project, we plan to keep building and improving it based on user feedback.

exAspArk··on Ask HN: Who is hiring? (November 2019)
Hyre | Full-Stack Developer | Toronto, Canada | ONSITE, REMOTE Canada

Hyre is a marketplace that connects event organizers such as hotels and venues with event staff such as wait and bar staff. Essentially, an Uber-like model for the $80B+ event staffing industry.

We are an ambitious early-stage startup, looking for a motivated and experienced Full-Stack Developer in our office space in the heart of downtown Toronto, Canada. Join our growing team to work with talented people, iterate quickly, and help us expand to new markets.

Our tech stack:

– Ruby, Ruby on Rails, Sidekiq.

– Elixir, Phoenix, GraphQL.

– JavaScript, React, ReactNative.

– PostgreSQL, Redis.

– Docker, Kubernetes, GCP, AWS.

Apply: evgeny@hyrestaff.com or https://angel.co/company/hyrestaff/jobs/634659-senior-full-s...

exAspArk··on Ask HN: Who is hiring? (October 2019)
Hyre | Full-Stack Developer | Toronto, Canada | ONSITE, VISA

Hyre is a marketplace that connects event organizers such as hotels and venues with event staff such as wait and bar staff. Essentially, an Uber-like model for the $80B+ event staffing industry.

We are an ambitious early-stage startup, looking for a motivated and experienced Full-Stack Developer in our office space in the heart of downtown Toronto, Canada. Join our growing team to work with talented people, iterate quickly, and help us expand to new markets.

Our tech stack:

– Ruby, Ruby on Rails, Sidekiq.

– Elixir, Phoenix, GraphQL.

– JavaScript, React, ReactNative.

– PostgreSQL, Redis.

– Docker, Kubernetes, GCP, AWS.

Apply: evgeny@hyrestaff.com or https://angel.co/company/hyrestaff/jobs/634659-full-stack-de...

exAspArk··on Ask HN: Who is hiring? (August 2019)
Hyre | Full-Stack Developer | Toronto, Canada | REMOTE, ONSITE, VISA

Hyre is a marketplace that connects event organizers such as hotels and venues with event staff such as wait and bar staff. Essentially, an Uber-like model for the $80B+ event staffing industry.

We are an ambitious early-stage startup, looking for a motivated and experienced Full-Stack Developer in our office in the heart of downtown Toronto, Canada. Join us to work with our small talented team, iterate quickly, and help us expand to new markets.

Our tech stack:

– Ruby, Ruby on Rails, Sidekiq.

– Elixir, Phoenix, GraphQL.

– JavaScript, React, ReactNative.

– PostgreSQL, Redis.

– Docker, Kubernetes, GCP, AWS.

Apply: evgeny@hyrestaff.com or https://angel.co/company/hyrestaff/jobs/576101-full-stack-de...

exAspArk··on Ask HN: Who is hiring? (July 2019)
We started by building some new functionality with Elixir. For example, a new read-only GraphQL API. Later, we might begin performing mutations and writes, serving all HTTP requests with Phoenix, and handling asynchronous jobs.

Although Elixir solves some of our problems, we don't want to rewrite things for the sake of rewriting them. That's why will probably keep things which work fine in terms of maintainability, scalability, etc. for now to move fast and learn more about the system and the domain (to avoid rewriting wrong features or right features rebuilt wrong).

exAspArk··on Ask HN: Who is hiring? (July 2019)
Hyre | Full-Stack Developer | Toronto, Canada | REMOTE, ONSITE, VISA

Hyre is a marketplace that helps connect event organizers (hotel chains such as Marriott and Hilton, exhibition centers, etc.) with event staff (supervisors, bartenders, wait staff, etc.). Essentially, an Uber-like model for the $80B+ event staffing industry.

We are an ambitious early-stage startup, looking for a motivated and experienced Full Stack Developer in our office in the heart of downtown Toronto, Canada. Join us to work with our small talented team, iterate quickly, and help us expand to new markets. Visa sponsorship is available for the right candidate.

Our tech stack:

* Ruby, Ruby on Rails, Sidekiq.

* Elixir, Phoenix, GraphQL.

* JavaScript, React, ReactNative.

* PostgreSQL, Redis.

* Docker, Kubernetes, GCP, AWS.

Apply: evgeny@hyrestaff.com or https://angel.co/company/hyrestaff/jobs/576101-full-stack-de...

exAspArk··on WorkflowCI – IFTTT for developers
Wow, thank you so much for your feedback!
exAspArk··on WorkflowCI – IFTTT for developers
Currently, it supports GitHub, Slack, CircleCI and Google Cloud Build to trigger any custom code in Docker containers. Are there any other specific integrations you would like to have?
exAspArk··on WorkflowCI – IFTTT for developers
Noticed the exception, trying to fix it, thank you! In the meantime, I emailed you with a potential solution and upgraded your free plan :)
exAspArk··on WorkflowCI – IFTTT for developers
I removed most of the permissions, thank you!
← PreviousPage 2 of 2