Optimizing the Astronomer Registry for performance across millions of page views

Data pipeline of 3 pieces:

> A set of Airflow DAGs that look for new releases of Airflow provider packages. Once new releases are identified, the code is parsed and data is submitted to Airtable. These run on (you guessed it!) Astro.

> Three tables in Airtable: one for Airflow providers, one for modules (operators) within those providers, and one for example DAGs.

> A UI written in React & TypeScript, hosted on Netlify.

Big use of materialized views in Postgres.

Has anyone actually used Astronomer? Is it worth the incredible cost?