seddonm1

A completely valid concern.

You can see that all stages in the video implement the PipelineStagePlugin: https://arc.tripl.ai/plugins/#pipeline-stage-plugins. This means you can safely remove them from the code base and recompile without that stage at all. These are all dynamically loaded at runtime so it should be easy (and to implement your own custom logic).

Similarly the Dockerfile https://github.com/tripl-ai/docker/blob/master/arc/Dockerfil... just includes the relevant plugins (if not in the main Arc repository) so you can easily remove them or the Cloud SDKs/JDBC drivers to reduce your surface area.

We have endeavoured to write a large number of tests but there is always room to add more.

It was actually named after an electric arc as it was initially developed at a large power company - but yes, naming is heavily overloaded now.

Great! I am glad someone is reading them as it has taken a huge amount of effort.

We also have a link from the Jupyter Completer (autocomplete) to the docs.

Cool! I was not aware.

Thanks. dbt is very cool and evolved at the same time but focuses on the Transform step of ETL only. Unfortunately, as data engineers, we still spend a lot of time consolidating the many input sources to perform that transformation and also want to load it to places.

You can see the work we have done to build standardised methods for Extract (https://arc.tripl.ai/extract/) and Load (https://arc.tripl.ai/load/) in the documentation.

We provide multiple Docker images (https://github.com/orgs/tripl-ai/packages) that make the Spark deployment easy:

- arc-jupyter: allows you to develop on your local machine (and offline) or you can easily integrate it with a JupyterHub deployment on Kubernetes (https://zero-to-jupyterhub.readthedocs.io/en/stable/index.ht...). We have built JupyterHub on GCP Kubernetes (GKE) with full user-level auth via GCP IAM. If anyone is interested I can publish a secrets-removed version of our script.

- arc: is the execution only docker image (so is smaller than arc-jupyter). We have this orchestrated on Kubernetes too and now that Spark officially supports Kubernetes deployment it is actually really easy to create and destroy clusters on demand.

Cool!

Most of the time we run Spark as a single-node (i.e. --master local[*]) as now we can easily utilise large nodes like 128 core, 512GB and Spark does scale vertically well but also runs relatively well on a small node like the Docker example on the website running on a laptop. The ability to run SQL against separate storage is Spark's killer feature in my view.

Arc does support the full Scala API which you can implement as a plugin (https://arc.tripl.ai/plugins/) so for advanced teams they have full control.

The reason we went for SQL-first is that we are trying to find the balance that allows Business Analysts to develop their own logic without having to learn Scala or even Python - as they probably already know SQL.

Hopefully some of the ideas are relevant to what you are building.

Yes I think as a community we have largely got our levels of abstraction incorrect:

- Code for pipelines without frameworks leads to huge repetition of logic - or worse people reimplementing the same 'logic' differently. Also you end up with a massive upgrade problem when new versions of underling execution engines change.

- Databricks provides a low-level API which leads to a lot of duplication of common code across notebooks (and reusability is difficult).

- The GUI based tools are often too high level so have very high reusability but are difficult to customise - and hard to source control.

We have tried to build an abstraction somewhere in between which gives you the reusability of the GUI tools, plays nicely with source control and has the power to add custom logic via the plugin interface: https://arc.tripl.ai/plugins/ if required.

(formatting)

After being frustrated with building 'traditional' ETL (Extract-Transform-Load) pipelines - and around the same time as the famous 'Engineers Shouldn’t Write ETL' blog post - we started building a framework/toolkit to allow Technical Business Analysts to be able to build reliable data pipelines without much developer support: Arc. This has been implemented as a Jupyter Notebooks extension.

Arc is declarative and currently targets the Apache Spark execution engine but the abstracted API allows replacing execution engines without having to rewrite the logic or intent of the pipeline in future. It supports parameterized notebooks to build complex pipelines which can be executed in CICD environments for safe deployment.

We would be interested to hear your feedback.

We have used https://almond.sh/ to create a Spark SQL interpreter using Jupyter Notebooks - plus a whole lot more which you can see here: https://arc.tripl.ai/tutorial

After seeing many companies writing ETL using code we decided it was too hard to manage at scale so provided this abstraction layer - which is heavily centered around expressing business logic in SQL - to standardise development (JupyterLab) and allow rapid deployments.

Maybe.

In this case Argo uses the Kubernetes distributed state store (etcd or equivalent) and is a stateless service so can deal with failures quite well.

Agree that YAML is not really ideal for this case.

For other readers Argo does support 'Scripts' which means you can implement complex logic similar to Airflow if required but can be a bit clumsy: https://argoproj.github.io/argo/examples/#scripts-results

Additionally, I think we may see other layers on top of Argo in future that may abstract from the YAML similar to how Kubeflow uses Argo for the Kubeflow Pipelines capability: https://www.kubeflow.org/docs/components/pipelines/pipelines...

I am hoping Argo Workflows (https://argoproj.github.io/projects/argo) will make that happen. It decouples orchestration from data flow nicely and runs on Kubernetes so is highly-available.

These GUI driven/Visual ETL tools certainly have their place but are firmly at one end of the ease of use vs engineering discipline based ETL continuum.

As other posters commented Visual ETL often suffer from source control or limited extension ability but do provide the rapid development environment that users (generally more business oriented) seek. They also tend to trivialize the value of experience/discipline - for example I go to an accountant for my tax because they apply learned-experience relating to tax that I do not have (even though the math is easy) whereas in data engineering seemingly simple tasks such as correctly applying data typing to money or dealing with timezones seems to be glossed over in the pursuit of DIY - and wondering why your money columns don't reconcile or you lose data in failure scenarios.

At the other end of the continuum large teams writing bespoke ETL code for every job does not scale well for many reasons (https://reorchestrate.com/posts/code-doesnt-scale-for-etl/). I think the positive reaction to ideas like Data Mesh comes from the failures of these large, centralized teams which coincided with the Hadoop era.

Our solution has been to develop an open source (MIT) declarative framework (https://arc.tripl.ai/) that allows configuration driven ETL - mostly developed via a Jupyter Notebook environment (to allow rapid development and appeal to a larger audience) - whilst making most of the difficult tasks mentioned above easier. This has been in development for a few years now and continues to evolve. We value your feedback.

Hi, Can you elaborate on what was happening to your machine regarding 'which was spontaneous black screen and poweroffs'?

My late-2013 Macbook Pro has recently started similar behavior. Somewhat randomly the screen will go black, sit like that for a while, then fans speed up for maybe 10 seconds then it eventually turns off. This whole cycle is something like 20 seconds.

Is this similar to what you observed which was fixed in your case by the script?

If you are working within the Apache Spark ecosystem you can us DeltaLake https://delta.io/ to create 'merge' datasets which are transactional, versioned and allow time travel by both version number and timestamp.

We have been thinking about these problems for a few years now and have built Arc https://arc.tripl.ai (fully open source.) which is an abstraction layer on top of Apache Spark to help end-users rapidly build and deploy data pipelines without having to know about #dataops. Ultimately we decided that giving users a decent interface https://github.com/tripl-ai/arc-starter (based on Jupyter Notebooks) and encouraging a 'SQL first' approach means we can give users flexibility but also have a standardised way of deploying jobs with many of the devops attributes (like logging and reliability). You can run Arc as a standard docker run command or using Argo Workflows https://argoproj.github.io/ on Kubernetes as the orchestrator as it plays nicely with Arc and is easy to build resilient pipelines (retries etc.)

Hi everyone I would like to show the project we have been working on for a couple of years: Arc, an opinionated framework for defining predictable, repeatable and manageable data transformation pipelines;

- predictable in that data is used to define transformations - not code

- repeatable in that if a job is executed multiple times it will produce the same result

- manageable in that execution considerations and logging have been baked in from the start

- MIT licensed open-source and cloud agnostic

We have seen that it is hard to scale data engineering teams in a code-first environment. Arc solves a lot of the problems we have seen data engineering/science teams struggle with. It:

- makes data engineering accessible to audiences outside of data engineers - you don't need to be proficient at Scala/Spark to introduce data engineering into your team

- has a Jupyter Notebook based development environment to quickly build logic

- provides a clear path to production for machine learning (via MLTransform, TensorflowServingTransform or HTTPTransform for models as a service)

- has a plugin system allowing federated development for any features not in the base framework

Currently it uses the Apache Spark execution engine but due to its declarative nature can be executed against future engines.

Please let us know if you have any feedback/suggestions.