articsputnik

I wrote down my journey from macOS to Omarchy, in case anybody want to do the same switch but has many questions as I had. Please also, AMA here.

Thanks for sharing, just wanted to post it:)

I created the Data Engineering Toolkit as a resource I wish I had when I started as a data engineer. Based on my two decades in the field, it basically compiles the most essential (opinionated) tools and technologies.

The Data Engineering Toolkit contains 70+ Technologies & Tools, 10 Core Knowledge Areas (from Linux basics to Kubernetes mastery), and multiple programming languages + their ecosystems. It is open-source focused.

It's perfect for new data engineers, career switchers, or anyone building their Toolkit. I hope it is helpful. Let me know the one toolkit you'd add to replace an existing one.

I love my Makefiles. All my github repos contain a Makefile as I always forget the commands for every repo. Like this I have it stored nicely, but I can also add complex steps and run on each of my projects `make` and it will do what I'd expect without remembering any cmds.

I have one like these, too as my home server, HP Elitedesk 800 G5 DM, to be explicit.

In times of AI Writing, writing manually is more important than ever. As with many AI Generated texts, I’d rather see the prompts, it would have more soul and character than the output.

I couldn’t agree more with the sentiment of this article.

Writing yourself, _writing manually_ is much nicer, to hear your unfiltered thoughts, than condensing them through an LLM, and get average-sounding sentences with no soul. To me, LLM writing is soulless. I even started to turn to Grammarly and Copilot, as these were a mere distraction to the actual task at hand: writing. Instead of writing, I was constantly grammar fixing, and ultimately, nothing got done. I love the gym-analogy https://news.ycombinator.com/item?id=43888803 gave.

Thank you so much!

You still can on Bluesky https://bsky.app/profile/hamilton.bsky.social/post/3lk6yxzan...

Mehdi just wrote about this. Mainly starting DAGs parallelism using Ray (core) and their filesystem 3FS. See https://mehdio.substack.com/p/duckdb-goes-distributed-deepse....

I have written a lot. But the one I'm most proud of is how I am finding my pathless path.

It's my most personal and vulnerable piece. It's 1.5 years old, and I have added to the path with self-employment and many more decisions since. We can't plan these things; you only connect the dots afterward.

Some learning:

- I was blown away by how much I missed because I didn't speak English

- Mini-retirement shaped me to be who I became

- Stay curious, follow your instincts early

- Podcasts/books are Pandora's box of wisdom

- Stoicism or any practice helps a ton

- Different cultures are a blessing

- Everyone is fighting a battle you know nothing about. Be kind

- Try to be happy with less

- Travel doesn't always need to be to the other side of the globe

- Money is essential, but you can't buy happiness

- It's not sure that we'll find our soulmate

Thank you, Paul Millerd, for the book. I'd recommend you to read it and try to reflect on your own pathless path.

> Ultimately, it's your path, and it's never-ending. Stay curious, make memories, and follow your instincts.

This is so great! Used ReadWise so far. But like this workflow much more.

It's remarkable how little has changed in data engineering over the past 6.5 years. I wrote an article back then titled "Data Engineering, the Future of Data Warehousing" and revisiting it now is eye-opening.

The core of data engineering remains largely the same:

- Python is still the dominant language

- Key challenges persist: data integration, pipeline maintenance, quality assurance, process automation, and big data handling

- Fundamental skills (SQL, data modeling, ETL) are as crucial as ever

Some predictions that held true:

1. Python's rise in data engineering and science

2. Shift towards more programmatic skills in data roles

3. Growing importance of data engineering in data-driven companies

While tools have evolved, the underlying techniques and craft haven't changed dramatically. We've seen the rise of ELT and more flexible architectures, but the core purpose remains: making data accessible, reliable, and valuable for businesses. What's your take?

What a great read. Thanks for putting in the time and effort Maxime. I was wondering if a declarative data stack would help. Isn't a declarative approach the solution to much repetition? Much like in Kubernetes, you define what you want, and implementation code is abstracted away and not duplicated. The declarative approach lets you automate, version, and all the good things of SW's best practices.

Much of it also reminds me of the days of Data Warehouse Automation, when I could template all of the tables with 2-3 templates and generate the whole DWH. Actually, these tools are still out there and used daily. Time to revive?

It's all in my Obsidian Vault. And I update every day. I do it mainly for myself. I have been a writer for 8 years, but I have been taking notes already since 2003 (OneNote). But since I use Obsidian, it takes me zero effort to publish some notes publicly; that's why I'll have an easy time continuing. And I focus on small bites every day, rather than huge updates and try hard. This way it's just natural to me. Whenever I research something or learn something, I add it to my Vault, it only takes me 10 more seconds to put it in the work of research I did already. If I want, I spend some more minutes to add some thoughts and connect with other related terms. But that's where I learn most. So it's really rewarding to me.

btw: I made a YT video of how I update it, in case of interest: https://youtu.be/myHKHM2mIis?si=8bZtmO0cdpItDZV7

Before I wrote about it too: https://www.ssp.sh/blog/obsidian-note-taking-workflow/

I actually did the other thing too, that you can update and PR here: https://github.com/airbytehq/glossary/. The same approach, but not many did, it. And for me to keep going is much easier if it's just me, as I can spend time on producing valuable content and not merging, although the value of other opinions would be greatly beneficial too.

More than a mere collection of terms, this is a curated network of data engineering knowledge to facilitate exploration and discovery. Like a digital garden, it features 1000+ interconnected concepts as gateways to deeper insights.

Key features:

- Covers 100+ core data engineering topics

- Inspired by Second Brain and Digital Garden methodologies

- Includes links to books, podcasts, and thought leaders in the field

- Updated regularly

- Ideal for both beginners and experienced professionals

Happy to get feedback :)

Great job on Pathway. It's impressive to see a Python tool for ETL and RAG tasks with such strong features. The Python API and Rust runtime for quick updates look interesting. Focusing on security and performance, especially with self-hosted RAG pipelines, is fantastic. Excited to see how this OSS repo grows.

Wow, thanks for the great advice. I will indeed try to think at a higher level and more about the problem at hand. Love the ending with pain/finding the customer

Thanks for the comment —Author of the book here.

As the first paragraph explains, "This book is different from usual books. It does not come finished." The idea is to write a living book and share as I go. If you do not want early versions, skip for now and return later.

Suppose you like to explore the beginning of data engineering and challenges; you might find the intro chapters interesting. The greyed-out chapters are the ones I'm working on, and I'm planning to publish them later down the road.

Regarding the paywall, as mentioned here [1], part 2/3 of the chapters are closed, except for sponsored chapters such as The History of Data Orchestration [2]. I need to clarify what's free and what's not. Thanks for the hint.

[1] https://www.dedp.online/subscription.html [2] https://www.dedp.online/part-2/4-ce/bash-stored-procedure-et...

This is a beautiful blog, the same as the minimalistic app. Huge congrats. I'm seeing myself building this, as the joy it returns by using an app built by yourself. I have feelings similar to those expressed in the article using vim-motions or generally Neovim. It's so simple that even it works in a terminal. But once mastered (at least the basics), I feel like a surgeon when operating, as I edit text so fine-grained and exact.

This is a short brain dump about my "Vim-Verse". How (Neo)Vim and Markdown increased my data engineering and writing workflow.

Yep, it is. Super happy with it. I added some custom themes and links to my second brain, but other than that, all mdBook. I also have more notes about writing a book with Markdown and self-publish at https://www.ssp.sh/brain/create-a-book-on-markdown/.

Author of the book here — yes, I used Mdbook (https://github.com/rust-lang/mdBook), and the links strangely end with `.html. as you correctly updated. Initially, I wanted to change it but left it as it was.

Author of the book here — 100%. The idea is to release early and update on the go. You might wonder if it's worth delving into the book in its current, unfinished form. I'd say it depends.

There's quite a bit of content around general DE knowledge. However, the anticipated design patterns are still in the works. Suppose that's what you're most excited about. In that case, you'll find the beginnings of exploring the patterns of `caching` and `ad-hoc querying` in the first Convergent Evolution chapter, but otherwise, you need to wait for more.

> Vim is the only editor you get faster with time.

Vim is based solely on shortcuts. When I discovered that and played around a bit, I felt numb and a little stupid, having not learned the shortcuts (called Vim language) much earlier in my career.

> The Vim Language (and Motions)

Vim has a terrific language or grammar behind its shortcuts. Instead, remember a thousand shortcuts; you can learn a couple and combine them. This has nothing to do with the editor yet; they are universal and available in other editors as well.

> Vim is the only editor that optimizes editing text instead of writing from a blank page.

When you launch Vim, you are not typing what you click on your keyboard as you are not in the “insert” mode that you know. Instead, the normal mode you are in lets you do the commands.

> Why Neovim?

Today, there is even a newer version of Vim called Neovim. Super popular. Compared to Vim, Neovim uses Lua, an actual programming language, to configure and extend Neovim. This makes writing plugins and configuring Neovim easier compared to Vim native Vimscript.

> Learning Vim is hard

I also love learning new things and, even better, under the umbrella to make me more productive. But as many of you might have experienced, the hardest part is to get started. The initial learning curve is very steep .

> When you should NOT learn

Suppose you prefer to avoid fiddling and optimizing your workflow or if you do not write or code for a living. You can start with your current editor and activate Vim mode. It will save you a lot of frustration, trust me.

Found this new glossary for data engineering.

Some terms explained extensively:

  - Fan-Out: dagster.io/glossary/fan-out
  - Partition: dagster.io/glossary/partition
  - Wrangle: dagster.io/glossary/wrangle

There is also:

  - Data Glossary Airbyte: glossary.airbyte.com
  - Data Engineering Wiki by Reddit: dataengineering.wiki
  - Seconda Glossary: secoda.co/glossary
  - Glossary Databricks: databricks.com/glossary
  - Airtable Glossary: airtable.com/shrGh8BqZbkfkbrfk/tbluZ3ayLHC3CKsDb

And I'm collecting an updated list here (in case of interest): https://www.ssp.sh/brain/data-engineering-glossaries/

Most people miss "Vim Is More than Just an Editor." The Vim Language, its motions, and its modes will make you a better programmer and writer. If you invested in the vim language early on, you can use it on each editor, browser, and other apps. This has nothing to do with the editor yet – these are universal and available. For example, there's VSVim for VSCode, IdeaVim for the JetBrains products, Vintage Mode for Sublime, and so on. But there are also Browser extensions like Vimium or Firenvim, and Gmail even adopted some of Vim's shortcuts for navigation (j, k for moving, g for jumping).

Everyone who types on a computer eight hours a day should learn the Vim language. Yes it's hard initially, but that's the case with everything new and different. But getting better every day and having more fun coding or writing should be motivation enough. Also, imagine, if you work over a decade, how many times you had to learn a new editor (notepad++, atom, eclipse, ...) some of which do not exist anymore. If you once learned the vim language, you are getting stronger. There is a reason the editor has existed since the eclipse of programming.

The Vim grammar is exceptional, as spoken language has verbs, subjects, and objects, as does the Vim language. The grammar has different verbs, to begin with. Copying (or yanking) in Vim with y, deleting with d, pasting with p, changing with c, and so on.

You can also see in the comments here people fight over Vim and VSCode, where the natural beauty is the language behind Vim. If you do not want to configure and customize your own, use VSCode and add the Vim extension. But if you're going to get better at your job, you should try the Vim language.

Once mastered, you edit text at the speed of thought. Sounds cheeky, but it's true for me. Instead of thinking, "I want to edit that word," my fingers jump to it and change it with a few keystrokes.

I wrote more on https://www.ssp.sh/blog/why-using-neovim-data-engineer-and-w... in case you are interested.

The font and the layout are amazing. I was wondering what font it is.

On a related note, the dagster team did a similar search for fake stars: https://dagster.io/blog/fake-stars.

Nice. Let's remember the old days when everyone had their own blogs :). The more who share their knowledge on personal blogs, the better. Please get in touch with me if you have any questions, or the Quartz Discord (https://discord.gg/cRFFHYye7t) is also a lovely small helping community.