towelpluswater

Yep we’re here. I think lots of us who don’t often post

Modular Mojo is the most well funded and full of respectable players for making an alternative possible

I mean they’re building the labeled dataset right now by having creators label it for them.

I would suspect this helps make moderation models better at estimating confidence levels of ai generated content that isn’t labeled as such (ie for deception).

Surprised we aren’t seeing more of this in labeling datasets for this new world (outside of captchas)

Think it’s “vibes” based search (aka dense embedding similarity search)

I’ve never understood why people want a more verbose version of sql.

I think what people really want is business rules and data cleaning and schema discovery.

If you had to use English against multiple source systems to and tons of joins, the sentence would be paragraphs.

Where I think there’s value is in using something like a data catalog to label business rules against a data warehouse, tied to dashboard queries and other common ones.

But that’s a hard problem and a unique model to every customer. And always changing.

This is a fantastic write up and great parallel to the state of where we’re headed.

This is a really great idea and use case. It also makes a ton of sense as a pilot use case for this type of open source project given extensions are smaller in scope.

I mean even having it document a best draft of what the extension code is doing would be awesome.

Unless it’s made into an extension and then you have a recursive hell.

Bought a copy! Your posts and newsletter content has been such a huge inspiration for me throughout 2023 - good luck, this is a huge effort!

I think the bigger problem is that replication/ingestion (ie: what fivetran does) has come to represent 'ELT'. Likely by design.

And you don't need that pesky transformation part.

Except you really do, when you get beyond having a source system or two.

I'd love an invite if you still have any.

I really like the simplicity of this framework, and they hit on a lot of common problems found in other agent-based frameworks. Most intrigued by the RAG improvements.

Seems like Microsoft was frustrated with the pace of movement in this space and the shitty results of agents (which admittedly kept my interest turned away from agents for the last few months). I'm interested again because it makes practical sense, and from looking at the example notebooks, seems fairly easy to integrate into existing applications.

Maybe this is the 'low code' approach that might actually work, and bridge together engineering and non-engineering resources.

This example was what caught my eye: https://github.com/microsoft/FLAML/blob/main/notebook/autoge...

after using both of them for awhile, i see them as complementary as well. i'm more likely to use langchain for execution of tasks, and llamaindex for backend optimization and building out 'embeddings data pipelines'. if i need something from langchain in llamaindex, i can bring it in, and vice versa.

the more folks experiment with this stuff, i think they'll see where it all comes together, and where some of the crossover is. but given how quickly everything changes in this space, i'm glad there's a clear focus from each team on their core strengths rather than throwing the kitchen sink of new papers at it.

I'm using LlamaIndex way more these days, mostly because the big use cases from enterprise orgs is around using their own data in novel ways. it makes much more sense for backend tasks.

that said, while there's some clear crossover between the two, i find myself using langchain for things like huggingface embeddings for local models, and other helpers that work well with llamaindex.

somewhat akin to a data warehouse and all the techniques and abstractions that go into modeling it for non-technical end users, llamaindex makes a lot of that much easier to work with as a developer. structured and unstructured data can be indexed side by side, and the auto retriever functions they've recently built out work really well once you've got data indexed in a sensible way. our next step is to put a simple UI on top of it all with filters (like a dashboard) that pass metadata filters to the llamaindex autoretriever.

these patterns may not be exactly right today, but I don't see any others focusing on this area. just throwing all of your docs haphazardly into an index and calling it a day is no different than tossing all your data into a single database schema without any rhyme or reason, and hoping your dashboards can do 'magic' on top of it.

Congrats Jerry! Tackling a hard but very real problem that every organization is going to run into eventually.

If they can build the dev community around this successfully, which I think they can, this will be such a huge gain for the entire ecosystem.

Readwise Reader has completely changed the way I consume articles/PDFs/ebooks and annotate, especially with the Ghostreader (AI/LLM) feature. Easily my favorite app of the year - will be even better if/when an offline mode comes out.

Also Explainpaper and Elicit for research papers - hard to go back to the 'old' days after using all 3 of these.

We sold last year and got to say all these points are pretty on target.

the process is grueling and by the end you just want it over to return back to normal life. I agree fully that getting everything done before the LOI is the way to go.

Learned a ton in this process and hope to be able to use that experience to help others as they plan to go through it.

I'd agree. Staff aug roles for individual freelancers are always easier to find, but good luck getting a team of those unless you're a more established firm - plus, the rate and work isn't all that great. We avoid them as much as possible.

Sell project work that's strategic (ie: solving a business problem and coming with a solution), as opposed to 'We need a frontend developer for 12 months'. Better rates, opportunities to work with business instead of IT, etc.

Takes years, many years, of networking, and investing back into your network. Know someone who was let go and needs help finding a job? Help them find a job. This might turn into something in 3 months, maybe a year, maybe never, but it's a game of numbers.

Like any good long-term investment, you have to keep nurturing it. Continuously stay in touch with people you meet.

If you do it right, the leads come to you through natural conversations, and you don't even feel like you're selling. Without the relationships already in place, everything feels like selling, and it's exhausting and significantly harder to land strategic project work. There's no magic here, and no book or course that will tell you the secret (it's mostly snakeoil) - it just takes time and effort. It gets easier when some of these relationships lead to real opportunities, you show well, and can use that for referrals into other places as the people you meet move around.

Another route for early starters is to subcontract through established consulting firms. Our company is more of an FTE model, but when we need specific / niche skills that we don't always have in house, we consider subcontracting options. If that's something you're interested in (we're US based), I'd be happy to chat (email in profile) and see if there's anything coming up.

Erwin?

This is neat, nice work. Any plans to add Snowflake as a database?

Almost 40 and never unlearned this. Not sure I ever will, but it's nice to aspire to it.

If someone can manage to find a creative way to tap into the Kindle ecosystem and provide a similar service, I'd imagine they could do quite well.

Build relationships and network. It's a huge investment of your time, but it's the best way to start to build a pipeline of opportunities.

Just means we have different anecdotal experience, then. Very little of mine has been in the tech industry.

Nobody here suggested it, just something I see organizations doing quite often.

(edit: the rationale behind this tends to be that you can avoid the heavy lifting of ETL/transformation logic by just using a data lake - obviously not the case, as most of us know)

Completely agree with you. Data lakes were marketed well because, well... data warehousing is hard, and a lot of work. Data lakes don't make that hard work disappear, it just changes how and where it happens.

I've found data lakes complement DW's (in databases) well. Keep the raw data in the lake and query as needed for discovery, and load it into structured tables as the business needs arise.

Data lakes alone are doomed to be failures.