spmurrayzzz

Chroma certainly doesn't have the most advanced API in this area, but you can for sure store chunks or documents, its up to you. If your document size is too large to generate embeddings in a single forward pass, then yes you do need to chunk in that scenario.

Oftentimes though, even if the document does fit, you choose to chunk anyways or further transform the data with abstractive/extractive summarization techniques to improve your search dynamics. This is why I'm not sure the complexity noted in the article is relevant in anything beyond a "naive RAG" stack. How its stored or linked is an issue to some degree, but the greater more complex smell is in what happens before you even get to that point of inserting the data.

For more production-grade RAG, just blindly inserting embeddings wholesale for full documents is rarely going to get you great results (this varies a lot between document sizes and domains). So as a result, you're almost always going to be doing ahead-of-time chunking (or summarization/NER/etc) not because you have to due to document size, but because your search performance demands it. Frequently this involves more than one embeddings model for capturing different semantics or supporting different tasks, not to mention reranking after the initial sweep.

That's the complexity that I think is worth tackling in a paid product offering, but the current state of the module described in the article isn't really competitive with the rest of the field in that respect IMHO.

This is also the case with weaviate (as you assumed). If you update the value of any previously vectorized property, weaviate generates new vectors automatically for you.

Yea I think I understand your motivation re: not donating your time. I guess my assumption was more-so that the likelihood of 3k SLOC UI project becoming a business seemed incredibly remote. Perhaps that is misguided.

Not the author, but given that this is a relatively small UI wrapper of a zoekt[1] backend, it seems like the risk here is isolated to the upstream Sourcegraph-maintained search dependency. By relatively small, I mean that the total SLOC for UI code in the entire project is around ~3.5k (compared to the backend which is currently 25x the size). Seems difficult to ascribe any enterprise motivations given that and additionally the UI seems very useful as-is even if you had to fork it and build a new community from there.

[1] https://github.com/sourcegraph/zoekt

I've definitely noticed the same in my career. Its easy to get caught up in the day-to-day and forget some of the reasons you got started doing all this in the first place (in my case, because its fun and I'm passionate about software/hardware).

On a whim, I decided to invest time in writing down one idea per week of anything fun I could hack on. It doesn't really matter whether or not I go through with it, I keep the stakes low: just write an idea down. That way it forces me to think about things I could build for myself or others/friends/family without much cognitive investment.

The end result has not only had a nonzero impact on my motivation to start new projects, it has impacted my ability to actually follow through. And I've noticed the practice has made the ideation loop happen more frequently than once per week over time.

I appreciate the time you took to write this all out (both the article and your response here). In particular, this line from the article resonated with my own experience over the last couple of decades:

> This particular setup—viewing DE as a very technical, general-purpose distributed system SWE discipline and making the people who know best what real-world scenarios they want to model—makes our setup work in practice.

The common analyst-to-DE path has some benefits for sure with respect to business-centric data modeling, but without the deep technical infrastructure investments and related support, the stack becomes a beast to deal with at scale (or just ends up being a massive cost on the balance sheet from outside vendor sourcing). You really need both verticals in order to be optimal IMO.

Of course if internally an org doesn't already have the platform/infrastructure to dogfood in the first place, this admittedly makes the proposition a bit more of a gamble.

Ah, interesting. That sort of split voting is indeed very bad luck, potentially a config-specific issue, or just a cluster that's seeing a catastrophic partition failure between every node.

In canonical Raft assuming no partition failures, this could only happen if every node's election timeout triggered at roughly the same time and they all become candidates simultaneously. For this state to persist (assuming short election timeouts and short heartbeat intervals), you have to get _really_ unlucky.

In terms of probabilistic likelihood though, this is about as likely as the live-lock issue in Paxos in which multiple proposals with differing proposal ids are made at the same time. You'd seem a similar delay in consensus in that scenario as well. Obviously MultiPaxos handles this with a separate leadership algorithm which makes that outcome much less likely, but the same types of strategies common in those systems to mitigate contention issues can be used in Raft as well (randomized backoffs for example).

> Reminds me of this one time we had a Raft cluster arguing over who was the leader for 20 minutes in production

That's certainly an interesting failure mode. Do you recall the details around root cause? I could imagine ephemeral network partitions (flapping interfaces? peering loss?) causing something like this for sure.

In my own experience, I've been running services that use Raft under the hood for the last ~10 years in production and haven't seen this happen myself. Though I do absolutely remember having misconfigured election timeouts causing very painful latency issues in failover scenarios.

I tend to agree that many explanations of raft dont get into the useful details and handwave some of the hard problems. But the original paper does do a good job of this and is pretty accessible to read IMO.

> I read "once the leader has been elected", um, hangon, according to whom? Has node 1 finally agreed on the leader, just while node 3 has given up and started another election?

The simple response I think to "according to whom" is "the majority of voting nodes". When the leader assumes its role, it sends heartbeats which are then accepted by the other nodes in the cluster. Even if (in your example) node 3 starts a new election, it will only succeed if it can get a majority of votes. If node 2 has already acknowledged a leader, it won't vote for node 3 in the same term.

There's some implicit concessions inherent there around eventual consistency, but I don't think thats novel to Raft compared to other distributed consensus protocols.

I should have quoted you more specifically, my apologies. I was responding to the comment that there was some training of the "model on your chosen book".

There is no fine-tuning done specific to the corpus you own. I noted this in a sibling comment, but both the pretraining and fine-tuning objective uses a generic dataset (redpajama) which "aims to maximize the generation probability of the next token given the KV cache of the previous memory tokens" (quote from section 2.2 of the paper).

This is why I noted you could really use any long-context model that also has good retrieval performance. They're training their own draft model in lieu of using an existing model, but you could get similar/better outcomes using something like claude sonnet 3.5.

> they have some long-context "Memory Model" (whose details I've yet to fully understand)

Section 2.2 of the paper[1] goes into this in more detail. They pretrain the draft model using the redpajama dataset, followed by a supervised fine-tuning step. The training objective "aims to maximize the generation probability of the next token given the KV cache of the previous memory tokens".

This suggests that any model with long context and good retrieval performance could do the same job (and maybe better in the case of the SOTA frontier models).

[1] https://arxiv.org/pdf/2409.05591

> AFAICT from a quick glance at the code (https://github.com/qhjqhj00/MemoRAG/blob/main/memorag/memora...), it is indeed "fine tuning" (jargon!) a model on your chosen book, presumably in the most basic/direct sense: asking it reproduce sections of text at random from the book given their surrounding context, and rewarding/penalizing the neural network based on how well it did.

Maybe your use of quotes is intentional here, but for posterity's sake there is no actual fine-tuning happening using user input in the code you linked, insofar as the weights of the model aren't being touched at all, nor are they modifying anything else that could impact the original weights (like a LoRA adapter). You touch on this, I think (?), in some of your subsequent language but it read as a little confusing to me at first glance. Or maybe I've been too deep in the ML weeds for too many years at this point.

The paper details the actual process, but the TL;DR is that the memory module they use, basically a draft model, does go through a pretraining phase using the redpajama dataset, and then an SFT phase with a different objective. This all happens before and irrespective of the inference-time task (i.e. asking questions about a given text). Also, as has been pointed out in other comments, the draft model could really be any model that supports long context and has decent retrieval performance. So the actual training phases here may be non-essential depending on your infra/cost constraints.

FWIW Their most expensive flex price I've ever seen for 80GB A100 was $0.00130 back in January of this year, which is still cheaper albeit by a smaller magnitude, if that's helpful at all for your own competitive market analysis.

(Congrats on the launch as well, by the way).

> Yes RunPod does have cheaper pricing than us however they don't allow you to specify your exact resources but rather charge you the full resource (see example of A100 above) so depending on your resource requirements our pricing could be competitive since we charge you only for the resources you use.

I may be misunderstanding your explanation a bit here, but Runpod's serverless "flex" tier looks like the same model (it only charges you for non-idle resources). And at that tier they are still 2x cheaper for A100, at your price point with them you could rent an H100.

The premier open weight models don't even comparatively perform well on the public benchmarks compared to frontier models. And that's assuming at least some degree of benchmark contamination for the open weight models.

While I don't think they're completely useless (though its close), calling them fantastic replacements feels like an egregious overstatement of their value.

EDIT: Also wanted to note that I think this becomes as much an expectations-setting exercise as it is evaluation on raw programming performance. Some people are incredibly impressed by the ability to assist in building simple web apps, others not so much. Experience will vary across that continuum.

> Producing copyrighted things isn’t illegal

That isn't true, strictly speaking. The right to reproduce work is covered by copyright law, irrespective of whether the reproduction is commercial or not.

I agree with respect to admiring it from afar. I've gone through large chunks of the source many times and always have an appreciation for what it does and how it accomplishes it. It has a great, supportive community around it as well (if not a tiny bit proselytizing at times, which doesn't bother me really).

With all that said, while I have no "hate" for the stack, I still have no plans to migrate our container infrastructure to it now or in the foreseeable future. I say that precisely because I've seen the source, not in spite of it. The net ROI on subsuming that level of complexity for most application ecosystems just doesn't strike me as obvious.

Relevant to the original inquiry — I really admire that you bring up the etcd and consul comparison right up front in the readme. For my own comprehension at least, it makes obvious the type of workloads for which you're optimizing and I appreciate that context as a past user of both of those stacks.

I can't speak for all use cases, but I've done a great deal of work in the space of using deep learning approaches for anomaly detection in network device telemetry. In particular with high resolution univariate time series of latency measurements, we saw success using convolutional autoencoders and GANs. These methods lean on reconstruction loss rather than forecasting, but still effective.

There is some prior art for this that we leaned on [1][2].

RE: transformers — I did some early experimentation with Temporal Fusion Transformers [3] which worked pretty well for forecasting compared to other deep learning methods, but rarely did I see it outperform standard baselines (like ARIMA) in our datasets.

[1] https://www.mdpi.com/2076-3417/12/23/12472

[2] https://arxiv.org/abs/2009.07769

[3] https://arxiv.org/abs/1912.09363

I would argue this post (and the majority of resultant comments) have demonstrated that programmers staying relevant isn't as difficult as it seems. They were curious about how to pivot, had a forum in which they could ask, and in minutes started getting a wealth of practical, actionable advice from folks who have done the same or similar. The theme so far is that those programming skills aren't obsolete just because you want to change the vertical you work in and learning materials to help you achieve that goal are abundant.

There is so much in the ML world that's being built in the open which you can use as a foundation for your learning. Some folks here already mentioned Jeremy Howard's fast.ai course which is absolutely a great place to start for anyone that a) has motivation to learn DL, and b) already knows how to code.

Beyond that, there are hundreds of open source projects you can fork to start building intuitions around what the inner loop of AI dev project cycles look like. You'll be surprised at how much of your web dev skills remain relevant in these projects, particularly in UI-related tasks. Data scientists and folks of similar ilk default to notebooks, gradio, streamlit et al. to ship interfaces for their experiments. You though have the ability to do that on your own if you choose to (sometimes a notebook is enough), which can be a valuable differentiator for you as a candidate if you also have all the other skills needed to be productive in this space.

My own background is in distributed systems with some full stack and embedded work mixed in over the years. I started tinkering with ML projects back in 2012 when I first discovered AlexNet and resources were far more limited. I was still able to get productive relatively quickly even though most of what I built wasn't really applicable to my work in a practical sense on day one. Where my background became relevant was when I needed something approximating an MLOps pipeline for data processing, training, and eval. Most of the code you're writing for that isn't really specific to ML, its nearly identical to CI/CD systems but with the obvious infrastructure caveats native to ML workloads.

Nowadays though, especially if you're intrepid/resourceful, there is so much more learning material by comparison and much of what you work on can likely augment your day-to-day web dev tasks as well if you're creative enough.

marimo is very good, been using it for a few months now and have switched over to it for most of my notebook-related tasks (it ships with copilot support)

https://github.com/marimo-team/marimo

Are you saying that the torch performance with MPS support enabled didn't meet your performance expectations? Or are you using an intel macbook and/or one with a tiny amount of ram/vram?

Quite a few router OEMs do this these days. Its often a bespoke fork of openwrt so they can add their own features (OTA upgrades, web ui customization, etc), but still openwrt under the hood.

Speaking for myself (and I guess anyone else dealing with pcie riser hell in on-prem deep learning setups), its nice to see the massive orgs dealing with pretty much the same exact pain points as not-so-massive orgs.

> cohere's Command R+ is also available and while less capable, for instruction following, I believe its superior to Anthropic's models

My experience recently is that its actually noticeably better for instruction following than Claude, but can be finicky if you're not careful about adhering to the prompt template. But between the RAG and multi-step tool use capabilities, even if it was slightly worse on the instruction-following side of things I'd still say, as you do, thats its much better than Claude on average.

Agree on titan as well. I recently was forced into a meeting with our AWS TAM, and they kept shoehorning Q into every conversation. I held my tongue knowing that titan was the model powering it under the hood.

The grandstanding definitely got in the way of otherwise legitimate feedback/concerns about the feature, as sadly tends to be the case in open source software when the mob descends.

Can you expand on this a bit? My recent experiences with MLX have been really positive, so I'm curious what footguns you're alluding to here.

(I don't do most of my work locally, but for smaller models its pretty convenient to work on my mbp).

> In fact you deliberately asked for something impossible and hold up undefined behavior as undefined like it's impugning something.

Correct, I did. This is a direct indictment on a given model's ability to plan/reason in this particular context. There are plenty of situations where models will respond with "Sorry, that's not possible". Ask GPT-4 "Tell me how to grow biological wings on a human" and it will respond with something along the lines of "this isn't currently possible, but here's a theoretical exploration of the idea"

GPT-4 gets very close on its own to the node.js question via a similar response breakdown above, provided the prompt is clear and detailed enough. But I test the open weight models in the same way to see if they have the capacity to exhibit similar reasoning or chain of thought process on their own. They usually don't without excessive prompt engineering or few-shot.

I said that I don't expect models to get this right not because I don't _want_ them to, it's because I think its an important milestone when they do. Autoregressive token prediction is unlikely to produce the real outcome im testing for here, but if it ever does thats an interesting finding.

I love to ask it to "make me a Node.js library that pings an ipv4 address, but you must use ZERO dependencies, you must only the native Node.js API modules"

The majority of models (both proprietary and open-weight) don't understand:

- by inference, ping means we're talking about ICMP

- ICMP requires raw sockets

- Node.js has no native raw socket API

You can do some CoT trickery to help it reason about the problem and maybe finally get it settled on a variety of solutions (usually some flavor of building a native add-on using C/C++/Rust/Go), or just guide it there step by step yourself, but the back and forth to get there requires a ton of pre-knowledge of the problem space which sorta defeats the purpose. If you just feed it the errors you get verbatim trying to run the code it generates, you end up in painful feedback loops.

(Note: I never expect the models to get this right, it's just a good microcosmic but concrete example of where knowledge & reasoning meets actual programming acumen, so its cool to see how models evolve to get better, if at all, at the task).