Hacker News Reader

You'd have to be insane to trust GPT that much. I wouldn't want anything hallucinated in my data.

@data_ders We realized that more than 70% of business data is shared in CSV and Excel formats, and only a small percentage use API integrations for data exchange..So CSV is here to stay for sure..On the other side,data engine is a sub module inside YoBulk.We are trying to automate complete CSV importing workflow..mostly solving the CSV errors in a collaborative way with the data donor..Yobulk's USP is how we show the errors in human readable way.We have written wrapper on top of some open source data validation engines.Yes..I have used apache Arrow..we not competing with Apache Arrow..We are creating an alternative to flatfile.com

Man, been down this path for a long while. It gets tough! Flattening csvs with hierarchical headers (as in, headers that that apply a category to a second row of headers) are tough.

The ways csv can fail is just fucking nuts. Especially when they're half hand written, half automated, or where a failure is 20m rows in. Hard to have speed and strong checks simultaneously.

Yes you are right..In YoBulk we flatten the CSV to a JSON schema store it in a document DB and do all the validations.Chunking the CSV and analysing the stream buffers for validation is giving us speed also.

Have you been able to get something that might match a relational database? Auto-generation of a relational schema from a large dataset, or multiple datasets, is a deeply interesting idea.

Would you really pay for this? I made one for a client analyzing some X million line csvs by sharding the records then computing across 500 lambda instances to arrive at a schema

Sounds lovely, but not something I would be able to do for reasons that I won't go into here. I would love to hear how you do it, though I'd understand if you can't or won't share.

It was straight forward. Basically with the csv's I processed, there were a few sets of columns that would make natural tables, i.e. addresses that would create natural 'Street Name', 'City', 'Zip' Tables.

The first step was just sharding the csv's into 200 lines or so with a naming schema. Then storing those in S3.

Then the following was creating a manager Lambda that would send out the schema creation task to however many Lambda concurrent instances I could run, and hold the shards until instances freed up. So it allowed me to do the inefficient work in the individual Lambda instance.

The inefficient work was determining how many uniques were in the dataset with a threshold. As in if there were 50 or more uniques, then consider that column 'dead' to becoming a schema. However, if it were less, filter it up to the main manager thread.

Then the main manager thread would binary dispatch the results to combine the results into a main schema with another unique value threshold.

Overall it was a fun learning experience. The hardest part was removing the download time to get the full file into the lambda... I never got that right. The processing was dang fast, the latency to get the rows into the individual threads never got 'fast fast'

---

Good luck! Cheers!

We thought the same solution for YoBulk but intentionally did not use S3 and lambda because all cloud infra cost and maintenance cost associated with this solution.You can find the details in our blog https://www.yobulk.dev/blog/Scaling%20a%20CSV%20importer

Ha! Sweet - well done to you and the team.

Hope I get back on CSV projects again soon to try your stuff out. Cheers

Ohh Yes..You are spot on. It's there in our upcoming release.Stay tuned please.

I still like the jsonl standard quite a lot. JSON is pretty much universal, yet it‘s better structured than csv. The ‚l‘ine in jsonl makes it easier parsable and independent from the remaining structure. Keys are duplicated hard but that‘s where gzip comes in.

"half of a data professional's job" ... you mean like 90%

It’s useful to make a distinction between what is one’s job and where one spends his time - that is, the theory and practice of job titles. Data cleaning is 0% of a Data Scientist’s job despite occupying 90% of their time. My hope is that AI can help bridge that gap.

yes textninja.YoBulk's vision is to automate the first mile data onboarding and cleaning through AI so that Data scientists are free from doing any mundane task of data cleaning.

> CSV is ubiquitous but has no singularly well-adopted standard

RFC4180 exists regardless of adoption level. In a way, the simplicity of the spec causes the proliferation of grammars. No one thinks they can just yolo a PDF by hand in a text editor. Ok, maybe PDF is a bad example. But CSV (as specified in RFC4180) is so dead simple that people take shortcuts.

Yes you are absolutely right..We need a solution beyond standard as 80% businesses run on CSV..

99.99%?

And that's only the portion the businesses know about.

working on it, this is a hard problem actually

pd.yeet("/data/data_1.csv")

pd.yeet(lambda x: yeet(x))

pd.yeet_to_csv("/clean_data/cleaned_data_1.csv")

yeet is dispose of with haste, not “move, but zoomer” or “sloppily with haste”

Wot, no. It's throw hastily we with no care of what it splats into. Pretty sure it came from a video of someone throwing something (food or drink?) in a crowded highschool hallway, from within the hallway. It's chaotic, reckless energy in.. mostly.. harmless form.

You’re both saying the same thing from my pov. But thanks for the translations!

Now that I'm not a phone, here's the video: https://www.youtube.com/watch?v=2Bjy5YQ5xPc

1. CSV is ubiquitous but has no singularly well-adopted standard. 2. software and data engineers struggle with CSVs as a result of #1. 3. tool is created to reduce pain and friction. 4. profit? a new market? a new standard?

The "revolutionary new tool" to replace CSVs was XML in the late 1990s.

Except not at all. XML is harder to enter by hand than CSV and not mess it up. CSV optimizes for the easiest cases and performs well on them. XML optimizes for the most complicated cases and therefore performs poorly on the easiest cases. JSON is somewhere in the middle. The main problems with CSV have to do with 1) MS Excel and 2) some kind of delusion among programmers that formatting or parsing arbitrary data is easy and you don't need a library for it, so you get hand-rolled generators and parsers that emit broken files.

Otherwise, the problem with CSV has little to do with the CSV format as such and more to do with the fact that the data is stringly-typed. XML has the same problem. JSON interestingly does not. Everything has tradeoffs.

@nerdponx You are spot on..

nitpick, I wouldn't place JSON in the middle (the lack of proper integers and precision problems is one of the issues). but other than that, spot on.

That’s really interesting, I wonder if this simplifies down to “you want CSV with column typing and a typesafe CSV editor”, as you note, JSONs win is the lack of issues with typing, and CSV really isn’t complicated at all except for that property. JSON is just a row with keys that are column.

I definitely want that! Parquet is great for data interchange, but it's not easily hand-editable. I wonder if there's an open niche in the software world for an Excel-like data entry and manipulation tool, but with stronger/stricter typing of cells and columns, and with direct export to and import from SQLite and Parquet.

To solve the XML issues you described we got schemas and syntax highlighting.

I hate non-prettified JSON but its easy to prettify in any editor. So its a meh argument against JSON. But to solve the crap with the comma one needs a variant of JSON, and there's various of these...

One other neat feature of CSV is it can be imported in a very popular and powerful IDE, called... Excel.

@sgerenser Yes CSV is everywhere..YoBulk is smartly positioning itself for the data donor/provider or customer..It is the end customer or data provider who bite the bullet and do the time consuming data cleaning.The customer should know about the errors, duplicates,PII data,inconsitency in data.Customer has to be properly guided to clean the data in best possible manner.