Fiahil

a bit difficult and controversial in Data Science, but okay. I’ve taught data scientists to use docker, I might be able to teach some Rust !

For anything that is not related to DS/ML/Opt, yes, start with Rust. I had tremendous success with distributed Rust microservices using ZeroMQ for backbone communication.

Prototype in Python, but scale in Rust.

I do a lot of AI Engineering in Python, but production Data Engineering, and Feature Engineering workloads are done in parallel with Rust. Simple and Fast.

You can even make custom-made python modules with a Rust core when Numpy matrices are not enough to crack your graph-traversing problem efficiently ! (Thanks PyO3 and Maturin)

> We have a custom metadata database running in our cloud control plane which handles ordering.

This is the secret sauce, right there. Anybody can host a bunch of topics and artefacts on S3, and have them locally-replicated if they want. However, there is no world where you can have that without a synchronisation service that ensure cursors are uniques and properly ordered.

it’s upside down on landscape ipad safari, but it’s cool nonetheless !

I absolutely don't care about what's going on at Twitter, but, wow, the current owner like to spend very little time between announcing and shipping features!

I was thinking about little resin sculptures of birds. Not living creatures.

I'm rooting for the birds too, but I'd also like humans to stop putting metal spikes on stuff.

It doesn't take a lot of effort to research potential predator/competitors for seagulls and then hold trials on what works. There's a lot of options, and if the gang doesn't seem intimidated by your choice of predators, invent one.

I'm sure this would be a very interesting project for any student!

Spikes and other stuff like these never work. You know what does ? This : https://www.amazon.com/Hausse-Scarecrow-Sculpture-Deterrents...

It's also cheaper, looks better and protect a much larger area.

Amazing ! I have been looking for a solution like this for a very long time I look forward to trying it out !

If that's true, then OpenAI has probably taken extreme protective measure to ensure the secret is well protected. Even if OpenAI is big enough to make deals, they probably did not spend several years making deals with all of them.

It's, however, very interesting to see if they fund efforts to massively (re)start books digitalisation.

> Don't forget that software engineers in the states tend to work more hours.

Don't forget that software engineers in the states tend to be much less productive than their German counterparts, to continue on your example.

> except for emergencies, where you're paying them overtime

They're not paid overtime, because they're not paid by the hour.

Yes, that's true outside of big cities.

Commuting in and out of Rennes takes about 40 minutes (by car), double that during rush hour.

I think it’s a cultural thing. The study was made with North-American commuters, it’s not ever a thing in dense metro areas with good subways, like Paris or London.

Edit: The article does mentions that’s it’s different over here :)

In a recent French parody show, National Geographic was renamed _National Catastrophic_. It probably doesn't help them that most of the content we see oversea is a remake of "American white man vs Shark!!!!!"

I think Data Scientists would like a word with you. They have plenty of time since their parallel pipeline was OOMKilled.

Looks very nice and I adhere to the concepts quite a lot !

one thing you could add : tha ability to save messages sent throughout the Agency/system to a log. That would grant you instant replay ability.

Despite sharing a common prefix, GPT3.5 and GPT4 are completely different models. So, if you were hoping for a price drop on GPT4, the 3.5 drop might not be of any use to you

So I work for one of the consulting firms :)

I can confirm the giant FOMO happening, and LLMs projects are popping up everywhere.

Good news is, the "talk-to-your data" use case you're describing is not the only one that can deliver amazing new tools! There's the "explain-me this", "summarize that", and "next-best-action" use cases that shows great promise!

Every engineer I know have been using DBML for a while. Did they fail their market research ?

Edit: Damn, it really look like they were copying dbdiagram.io, down to the FK relationship syntax. Not cool.

Then, maybe the right choice isn't to start a fresh DataFrame library from arrow, but rather leverage Polars and build out the distributed part (in Rust, of course, not in Python).

> We're trying to start with a simpler API that maps well to a distributed query query that we can execute well and then add the features that people request for.

That would have been a good approach on a field that has not been standardised around a single library since its infancy. Polars is beating Pandas in every possible benchmark, yet will continue to struggle for adoption "until the end". Do you really think Daft can do better ? (If yes, go ahaid, and prove me wrong !)

As a comparison, it's like trying to introduce a new transport layer protocol (https://en.wikipedia.org/wiki/QUIC) against TCP. You can do that if and only if there are obvious benefits, no drawbacks and you are prepared to wait 15 years for 30% market share.

Thanks you for your responses !

I'm going to be very blunt here, because you need to hear this to go forward :

You HAVE TO be at least API-compatible with Polars or Pandas to exist. Being backend-compatible with arrow is not enough.

There is no technical reason why you would not pick one and go with it, apart from being a very difficult task.

As of today, I have 2 major pains : Pandas being a giant memory hog and Polars not being a drop-in replacement for Pandas. I am pushing Polars, as hard as I can in all projects I can touch, and it's a very long way from being the default DataFrame library. Data Scientists will continue to use Pandas for the foreseeable future, and that saddens me greatly because I will also have to work with OOMKilled pipelines for the foreseeable future.

There is no place for a 3rd alternative, so either you become a "distribution bridge for Polars", and that would be absolutely amazing. Or, you go your own way, I'll put a small star on github, a "Noice!" in the comment section and move on and never come back.

It's tough, but sadly real.

So Daft is a distributed Polars ?

If we set apart the distributed part, what's the "killer feature" of Daft for trying to compete with Polars (and Pandas) ? Are they API-compatible ? How's the memory consumption benchmark ? (TBH, this is the only interesting metric. Timing and Latency are not really important when your most important competitor is Spark)

Benevolent dictator for life.

"Benevolent dictator for life (BDFL) is a title given to a small number of open-source software development leaders, typically project founders who retain the final say in disputes or arguments within the community. The phrase originated in 1995 with reference to Guido van Rossum, creator of the Python programming language."

Between 200 and 400 years old, yes, exactly ! Most of those trees are older than the United States :)

C interop is not as much a "killer feature" as it used to be. When you deal with data science, web and other similar domains, reading and writing JSON ergonomically is much more important than being able to call a C function directly. It's just a nice-to-have.

Like siblings said : it's not a good benchmark, as forests were not exploited as much before.

In fact, almost all forests are exploited in some way or another. Some recently protected primal old growth will take a few centuries to recover.

https://en.m.wikipedia.org/wiki/Old-growth_forest

"The world has 1.11 billion ha of primary forest remaining.[when?] Combined, three countries (Brazil, Canada, and Russia) host more than half (61 percent) of the world's primary forest. The area of primary forest has decreased by 81 million ha since 1990, but the rate of loss more than halved in 2010–2020 compared with the previous decade."

I had a candidate, last week, that was using ChatGPT during a technical coding interview.

So, he managed to find perfect answers to the most difficult parts of the exercise in 30 seconds, and then struggled for 30 minutes, with my help, on getting the print right.

Of course, I used a slightly modified version of well known problems like "Fizzbuzz" and other stuff, mostly as a quick start to get into the real engineering challenge. It explains why ChatGPT was absolutely unable to finish the exercise. During the interview, it acts a a good filter. If it last 3 minutes, fine, no signal ; but if someone is struggling on it for half the interview time, it's a clear "No".

I haven't check if the candidate was invited to later rounds, but considering my negative review I think he was rejected.

I'm afraid these tools are not going to help you when things really matter :)

Absolutely not.

A single vendor/tech does not "solve" anything when the task at hand implies you need to entirely re-design data pipelines, ML modelling and benchmarking.

Yet, everyone misses reproducibility and data versioning :)

So, talking about monitoring, training, and recording model drift is only a single side of the domain.

Note of high importance : "Faiss" reads and sounds exactly like "Fesse" (butt), in French.

I see some French names in the author list. The joke was intended, well done ! :)