Hacker News Reader

Comment by akudha | Hacker News Reader

Could you please elaborate a bit more? I am your typical web-dev/frontend/backend developer, don't really know much about what is involved in data engineering.

Do you mean collecting, cleaning data? Or setting up databases (if yes, how is it different from me managing my employer's databases, except for size)?

In other words, what does a data engineer do all day?

View on HN

Several data engineers that I know in this space (AI for business applications) are doing the usual ETL + mapping pipeline related work. Nowadays a lot of them are also having to deal with unstructured data such as textual reviews of products, service quality feedback, policy documents. Data engineers are developing the pipelines for chunking, vectorization, ingestion into RAG pipeline and for LLM training fine tuning. So it's still collecting, cleaning data, but quite different at the next level of detail, in my opinion.

Thank you. How does one get into this field? Web dev isn’t that exciting anymore

The data engineers I know started out as data scientists. They discovered that their ability to make models that did anything useful was hampered by low-quality or absent data. Then they discovered that there was a lot more job satisfaction (not to mention much better pay) in making useful data available than there was in clicking the "train" button over and over in slightly different configurations.

> How does one get into this field?

Network and meet someone who will give you, a web developer (a profession that is heavily stigmatized in the industry), a chance in a very competitive field.

I didnt realize web devs are stigmatized in the industry, as a web developer

They are not. There is a stigma associated with the ecosystem I think.

People who can create beautiful and user-friendly interfaces are extremely valuable and respected.

But my experiences trying to set up a typescript project and the unbelievable number of frameworks and crazy things to navigate (mjs/cjs) made me yearn for the pains of the Python ecosystem again.

Web developers are, front end engineers are not.

Reply on news.ycombinator.com