remilouf

Agree, working on a Colab with a "better" model as we speak.

We're unfortunately only human and didn't catch every single paper on the topic while writing the draft. Thanks for bringing it to our attention.

It can be used with any open source model (if you can get the logits), and to some extent with OpenAI's API. Here is an example with `transformers`: https://github.com/normal-computing/outlines#efficient-json-...

We plan on adding more model integrations, but it is completely decoupled from the method implementation.

Figure 2 in our paper (https://arxiv.org/abs/2307.09702) shows the difference between guidance and outlines to generate a sequence that is valid to a regex. Jsonformer uses the same technique as guidance. Extrapolate this to several fields.

Note that we still need to manage the KV cache in outlines. It’s a small interface change that will be made this week hopefully, but we’ve been focusing on constrained generation so far.

Perhaps I didn’t explain clearly enough in the original post?

Figure 2 in our paper (https://arxiv.org/abs/2307.09702) shows the difference for a single regex.

Indeed, this remains an empirical question.

Thanks for bringing clownfish and relm to my attention! afaik other libraries loop over the entire vocabulary at every step of the generation. We on the other hand build an index at initialization by looping once over the vocabulary. Then generation is just as fast as standard generation.

Thank you! Hope this helps and opens many applications :)

Indeed. And we're able to update the mask with a dictionary lookup instead of looping over the entire vocabulary (slow!).

LQML (and guidance https://github.com/guidance-ai/guidance) are much more inefficient. They loop over the entire vocabulary at each step, we only do it once at initialization.

Thanks! We have extended the approach to grammar-based sampling. We describe the approach in the paper linked above. The following PR is relevant: https://github.com/normal-computing/outlines/pull/178

We can extend our approach to grammar-based sampling, as explained in the paper linked above. Relevant PR: https://github.com/normal-computing/outlines/pull/178

Our method is much more efficient. llama.cpp loops over the entire vocabulary (~50k tokens) at each step to generate the mask. We generate an index at initialization, and building the masks at each step only requires a dictionary lookup (trade speed for memory). Sampling is just as fast as standard sampling.

Two differences:

(1) This feature only requires regex-guided generation. We have a PR for BNF sampling that is about to be merged. (2) ggml loops over the entire vocabulary (~50k tokens) at each step, which introduces a noticeable overhead, and makes it unusable for complex grammars. Our method works by building an index at initialization, and build the masks at each step with a dictionary lookup. Once the index is built, generation is just as fast as standard generation. Doesn't depend on the complexity of the grammar, the size of the LLM or its vocabulary size.

That's what we noticed as well, and we were not satisfied with the `guardrails` approach of just rejecting invalid outputs. The method makes the interface robust.

Thanks for bringing this library to my attention! From my understanding, TypeChat proceeds by (1) generating (2) attempting validation (3) if it fails, call the LLM again to fix the output (4) etc.

Our method on the other guarantees that the output will follow the specs of the JSON schema. No need to call the LLM several times.

I’ve had the same frustration recently and coded a working prototype in Elixir. I’d be happy to share.

For me culture fit is working culture fit. I’m always looking for someone who can get on board with the way we work as a team. I don’t care if we can become friends or not, if you are willing to work long hours or not (I’d rather not) as long as we can work efficiently together and rely on each other to get the job done.

In early stages, for instance, I would look for people who have a shipping fast mentality. In later stage for people who value technical excellence.

Everything else is at best childish and naive, and too often discriminating against certain persons who would actually be a valuable asset. A grown up can work efficiently in a team without going to Thursday beers every week.

Back in my days as an academic I would follow the same pattern every day: start working early from 7 till 10, take a long walk to the lab have a couple informal meetings with colleague, have lunch. Work there for a couple hours (my « available hours », run some errands or see a friend for coffee and work another couple hours. I have never been as productive as I was back then.

I don’t think this schedule would work for everyone but the general idea was: intense work for a short periods of time, take some time to talk with colleagues and go through meetings and then work again, with long breaks in between. That way I could manage 8 hours of productivity without burning out. And yes, sometimes I would get in flow and work for 12 hours straight without eating. But those days were more the exception than the rule. The point is I could have roughly 8 hours most days working this way.

In companies I’ve worked with I’ve always felt babysitted, as though I was unable to discipline myself when not watched all the time by managers. The truth is we don’t all work in the same way, and we are all reasonably interested in our job—-and if we’re not, sitting all day in the office is not going to change that. So why don’t we make room for everyone’s pattern while keeping some team time every day?

This. Having a tiling windows manager also helped me focusing on the task at hand somehow.

Having used Erlang for a side project recently: pattern matching and supervision trees.

I think it was intended to show you the landscape, give you enough tools and background knowledge so you can go and explore the literature by yourself. Years after it’s publication it still does a really good job at it.

What we call gut feeling is often a subconscious interpretation of the person’s body language, physical appearance and tone of voice. It’s always good to try to explicit why you had that feeling before basing your decision on it.

I often think that shy people are assholes at first. Now that I’m aware of that I try to test the shyness hypothesis first when I have that feeling.

So exploit it, it’s something humans developed for good reasons. Just be aware of the biais and use it as a good starting point for a behavioral interview.

I really recommend Murphy’s “Machine Learning: a probabilistic perspective”. Murphy’s lays the groundwork for understanding how the algorithms work, why and how they could be adapted to the problem you’re dealing with. It takes you from complete beginner (with a reasonable math level) to one step above `import sklearn as sk`.

The other books I read make the field look like a bunch of heuristics that just happen to work.

I will consider that. How about Redshift?

Thanks! Do you have any tips on convincing people that SQL is a good paradigm?

I was also hired because I also have an affinity (affinity, not expertise) with data engineering and am familiar with development good practices. The idea is not to spend 100% of my time doing this, more like 30%.

Business value comes first, and with a better infrastructure we could deliver a lot more value with the same head count.

I think it may come across as trolling :)

> Don't ruin it by becoming that cliched new hire that sees all their problems and knows how to make it all better.

I don't, which is why I'm asking around. I'm also scheduling chats with people to understand the background and the history to see if it's worth changing anything. I will make a move if that make sense. In the meantime, I'm just gathering information to not make a stupid decision.

> And you're a perm now, in a big corp, doing data science. Relax, you got it made, right?

> So enjoy playing that game, because you are happy to be in a big corp, so the sort of benefits that can bring you is what you want, right?

I don't think this was necessary. I've only worked for startups before, and I was hired in part to see if we can do a better job with the resources we have. Buy in from management is not an issue. I am not asking for life advice.

You changed my perspective a little bit by asking the right questions.

> Moves from an architecture that is clustered for scale (ie. spark) to one that only scales vertically

I did a quick estimate of the volume, and we won't reach 1Tb before > 5 years. We're not in a line of business where the number of clients can increase dramatically so it's fairly predictable. I don't want to design for imaginary scaling issues.

> Potentially introduces yet more sources of truth for some data.

It is more intended to replace the current mess.

> SQL is terrible language to write transformations in (its a query language, not an ETL pipeline)

Actually this is the point that concerns me the most. The need to transform the data in non-trivial ways. But surely people didn't wait for Spark to do this?

> Unless you can very clearly demonstrate that what you're making is meaningfully better

This is a very good point, and I think I should come up with a quick POC to demonstrate and get buy-in.

> Could you perhaps find better way to orchestrate your spark tasks, eg. with airflow or ADF or AWS Glue or whatever?

I feel that it would just be solving the mess by adding more mess.