localhost

For your use case, what if you could use Python in Excel with pandas, numpy and the Anaconda ecosystem readily available? How would that change things?

Disclosure: I was a founding member of the Python in Excel team and am looking for new problems that Python in Excel could solve.

Do you have examples of where this has been done? Based on my understanding you can do things like cache the embeddings to avoid the tokenization/embedding cost, but you will still need to do a forward pass through the model with the new user prompt and the cached context. That is where the naive O(N^2) complexity comes from and that is the cost that cannot be avoided (because the whole point is to present the next user prompt to the model along with the cached context).

The tokens need to be fed into the model along with the prompt and this takes time. Naive attention is O(N^2). They probably use at least flash attention, and likely something more exotic to their hardware.

You'll notice in their video [1] that they never show the prompts running interactively. This is for a roughly 800K context. They claim that "the model took around 60s to respond to each of these prompts".

This is not really usable as an interactive experience. I don't want to wait 1 minute for an answer each time I have a question.

[1] https://www.youtube.com/watch?v=SSnsmqIj1MI

RE: RAG - they haven't released pricing, but if input tokens are priced at GPT-4 levels - $0.01/1K then sending 10M tokens will cost you $100.

That's what AI is for, and it does a great job of this today.

Thanks! Really looking forward to interruptions.

I think about the cue as kind of being like "Hey Siri/Alexa/Cortana" but in reverse.

Totally agree with your take. But a pre-programmed phrase would work today and hopefully wouldn't be too difficult to implement. I would imagine that higher latency would be more tolerable as well. But in the fullness of time, your approach is better.

When I'm listening to someone else talk, I'm already formulating responses or at least an outline of responses in my head. If the LLM could do a progressive summarization of the conversation in real-time as part of its context this would be super cool as well. It could also interrupt you if the LLM self-reflects on the summary and realizes that now would be a good time to interrupt.

There are two things that I think are needed and that I'm not sure if anyone provides yet to make this scenario work well:

1. Interruption - I need to be able to say "hang on" and have the LLM pause. 2. Wait for a specific cue before responding. I like "What do you think?"

That + low latency are crucial. It needs to feel like talking to another person.

I think it would be a wonderful education tool for future hackers on that codebase!

Would it scale to something like the Linux kernel? How much would it cost to process something of that size?

For sure this is a super-impressive (solo?) effort and I'm certain a ton of work went into this.

Some feedback:

Who is the customer for this? Can you describe what their day looks like and how Nino helps them get their work done better/faster/cheaper? What are the top 5 problems that they face that Nino is clearly better than the competition? Be specific and show the workflows.

Where do they work? Who do they collaborate with? What are they collaborating on? Do the same exercise - SxS comparison with how they do things in existing tools and show how Nino is clearly better than their existing solution.

Finally, don't ever underestimate how difficult it is to get people to change from whatever they are doing today. Today is not 1990 - people have been using solutions to the general information worker problem for decades now. Why will they switch?

Use ollama [1]

$ ollama run mixtral

[1] https://github.com/jmorganca/ollama

My gone but not forgotten E46 M3 had drive by wire throttle in 2000.

He has nearly zero equity in OpenAI by design.

If you look at the quote tweets on Sam's latest tweet[1] that contain just a single heart and no words, those are all OpenAI employees voting for Sam's return. It's quite a sight to see.

[1] https://twitter.com/sama/status/1726099792600903681

The latest Stratechery interview had Pat Gelsinger on it and he was talking about how Intel is happy to package up TSMC dies in their foundries. Relevant quote:

> So it really brings together many of the things that Intel is doing as an IDM, now bringing it together in a heterogeneous environment where we’re taking TSMC dies. We’re going to be using other foundries in the industry, we’re standardizing that with UCIe. So I really see ourself as the front end of this multi-chip chiplet world doing so in the Intel way, standardizing it for the industry’s participation with UCIE, and then just winning a better technology.

I haven't used this before, but are there things that you wish you could have used Python to analyze the data in the workbook? I currently use Mint, so need to migrate to something and maybe I could find a way to extend any missing functionality via Python (disclosure: I'm a founding member of the Python in Excel team).

Edit: I see that it was shut down too.

The Steve Jobs quote in response to Walt Mossberg's question:

The core functions of the device form factor formerly known as the cellphone, whatever we want to call it — the pocket device — what would you say the core functions are five years out?

at the centerpiece of Ben's argument is lovely.

I don’t know.

The reason I don’t know is because I wouldn’t have thought that there would have been maps on it five years ago. But something comes along, gets really popular, people love it, get used to it, you want it on there. People are inventing things constantly and I think the art of it is balancing what’s on there and what’s not — it’s the editing function.

Contrast it with Bill Gates' answer to the same question:

How quickly all these things that have been somewhat specialized — the navigation device, the digital wallet, the phone, the camera, the video camera — how quickly those all come together, that’s hard to chart out, but eventually you’ll be able to make something that has the capability to do every one of those things. And yet given the small size, you still won’t want to edit your homework or edit a movie on a screen of that size, and so you’ll have something else that lets you do the reading and editing and those things. Now if we could ever get a screen that would just roll out like a scroll, then you might be able to have the device that did everything.

It makes me reflect on how important it is to remain humble about the possibilities of AI and what developers can create using it (and the tools that they will use to create it). "And we are just getting started".

character.ai

I think about agents as being an entity that you can collaborate with. Among other things, it needs 1/ memory, 2/ ability to create and execute long-term plans. One of the best papers that I have read on this topic is the Generative Agents paper by Park [1]

Today's LLMs are kind of like Dory from Finding Nemo - you have to recreate the context every time you do a slightly different task, or when the context window for the LLM is no longer sufficient to remember previous turns in a conversation.

An agent can sit on the other side of a piece of collaborative software that was designed for human collaboration. This is why chatbots are the current "killer app" for AI, we already understand how to collaborate with other people via chat.

Now imagine what we can do with more sophisticated pieces of software like Figma or Excel. Disclosure: I work on the Python in Excel feature and we just announced Copilot for Excel a couple of days ago. [2] Other modalities that excite me are tools like Vision Pro which will be an interesting test bed for multi-modal generative agents.

[1] https://arxiv.org/abs/2304.03442

[2] https://twitter.com/john_lam/status/1704886164753363261

Many years ago I was at a small group meeting with Terry Myerson who ran the Windows phone org and had just assumed responsibility of the entire Windows org. I had a Nokia 1520 (the giant phone) and he asked me what I thought of it. I said I loved it, and then complained about how it kept on dropping bluetooth with my car's head unit, a new Subaru STi.

The next day there was an engineer sitting in the back of my car with a bunch of test devices capturing traces of the BT comms with the head unit. Apparently Subaru didn't sell enough units to warrant its own certification process for BT so this was the first time engineers had looked at it. IIRC it did get better a few updates later; it was maddeningly unusable out of the box.

We totally understand that there are folks who want local in addition to cloud. But local is a very hard security problem. We needed to ship something to see if we have product market fit, so we prioritized that first.

Disclosure: I work on the design of the integration.

Dependency hell may be an issue if you install arbitrary libraries into the container at runtime. Note that this feature isn't currently available in the preview.

On versioning: we freeze the container image that your Workbook was authored against. You need to manually accept (and validate things continue to work) updates to that container image as we roll forward.

> games have been skipping with Python scripting for decades

That only works if there is a forever fixed version of Python embedded in your game. The value of Python in this context is its ecosystem and folks will need to install additional packages and libraries into the execution environment. Now you're managing a local distribution of Python.

There are few key reasons for running in a hypervisor isolated container on Azure with no access to the Internet:

- We can guarantee a consistent experience for all users. Imagine having to maintain your own local distribution of Python and guaranteeing that it works with Excel as versions diverge over time? Yikes.

- We make it possible to share your Excel workbook with other users and have the calculation just work. That wouldn't work with random local installs of Python and users would be super frustrated by this.

- Security. Imagine opening an Excel workbook that can execute Python code running locally as you.

Also, I totally agree with your second point. Trying to write an internal app in Python that integrates with all of your existing IT infrastructure is an exercise in frustration at best. Excel is already part of the IT infrastructure virtually everywhere and is a programmable reactive canvas.

Disclosure: I work on the design team for the feature.

A satellite in orbit uses the energy imparted by the gravitational force of the Earth to stay in orbit (and very little to no drag depending on the altitude).

Many of the research grants are funded from public tax dollars

Porsche for one - don't know of others.

It's under experimental features or something like that. I just enabled Bard access for myself yesterday.

I'd love to have something that could create daily summaries of private Twitter lists that I use to track developments in AI. The audio part would be great to consume during my re-emerging commute to the office. I would imagine this would be a pretty cool thing to have for any information worker as well. The difference is that this would be for an audience of 1.