typest

Hi! I’m a product engineer on the Claude.ai team. Claude.ai does support branching conversations. If you hover on a message, there should be an edit button, and once you edit the message, you can again hover on it, which will show you left/right arrows that will switch between the branches. Please let me know if you have any troubles with this!

I don’t want to die today, and tomorrow I’m not going to want to die, and the day after I’m not going to want to die, and so on. Therefore, I want to live forever — proof by induction on the natural numbers.

If a model hasn't been explicitly told (via some system prompt or something) about its weights, it won't know them. It would be akin to asking you how many neurons you had. How would you know?

It seems to me that RAG is really search, and search is generally a hard problem without an easy one size fits all solution. E.g., as people push retrieval further and further in the context of LLM generation, they're going to go further down the rabbit hole of how to build a good search system.

Is everyone currently reinventing search from first principles?

I just got engaged. When we looked at rings, the jeweler asked my fiancé if she wanted natural or synthetic, and she responded “I don’t want a blood diamond!!” Of course, mined diamonds aren’t blood diamonds, but her impression was still they were a little ickier.

The jeweler told me that one reason to get a natural diamond was that the prices of lab grown diamonds had been falling, whereas natural hasn’t as much, so the ring would hold more value. I told her that was exactly why we wanted to go with a lab grown diamond! This isn’t an investment — we aren’t planning to sell the ring.

Ultimately, for a price that didn’t break the bank, we got an absolutely gorgeous ring with diamonds larger and higher quality than we would have been able to afford with natural. Diamond rings may have started as something to resell in divorce, but for us (for my fiancé really), it was more about getting something that was beautiful, and if it didn’t cost as much, great! I suspect most Americans will feel similarly.

Yes, PMF = "product market fit".

A city on its own generally has the power to change zoning and land use laws. There are limitations (some states give cities more or less power here) but generally cities do have the ability to make changes that would reduce noise and improve quality of life for their residents.

On the other hand, cities do not have the ability to redesign cars. A city can’t just choose to invent a quieter car!

I would love it if we can engineer cars and trucks to be quieter; I think though that a more feasible path for any city is to work on zoning and land use changes.

> All eligible entries must include either the word "wallabywinter" or the word "yallabywinter" (the “eligible keywords”) in one or more places as close as possible to the code.

If I'm training codegen models, why wouldn't I just exclude code that contains these keywords? Shouldn't you have secret keywords, that people have to register to you, but you don't make public until after the fact, in order to avoid this?

It's not that they're just charging per token -- the actual models are operating on a token level. The model sees things in terms of tokens, and in openai's case, these tokens are subword (pieces of words), not words themselves, not characters.

So the real question is, what is the benefit of modeling your tokens as subwords, rather than as characters or words?

I think there is a lot of nuance here, and I don't understand it all. But, some benefits:

* Words, at least in English, are composed of different pieces, like roots, prefixes, and stems. Modeling at the subword level more naturally aligns your model with this aspect of language. If I tokenize "warmest", I get "warm" and "est". So, the meaning of the token "est" can be learned by the model -- whereas if you modeled by words, the model would have to individually relearn this aspect of information for every word ending in "est".

* Modeling at the subword level makes your sequences a lot shorter than modeling at the character level, which should help with things like efficiency.

* Modeling at the subword level makes your vocabulary a lot bigger than just modeling at the character level, which I suspect helps the model, as it can assign the subwords themselves meaning. E.g., it can learn the meaning of the token "warm" on its own, rather than having to learn this meaning only through learning the relationship of the tokens "w" "a" "r" and "m".

Hope this helps! Would love for anyone else to chime in/add on/correct me.

This is not really true. The Chinchilla paper showed that a 4% difference in loss between Chinchilla and Gopher led Chinchilla to blow Gopher out of the water at most tasks, including 30x performance in physics.

Empirically, LLMs have shown to have emergent abilities appear at different loss levels. So, a 10% difference could really matter.

^^ this. For instance, LLMs are really good at turning natural language into SQL. And if you know SQL, you can read it and make sure it looks good. But, it's much faster and easier than writing SQL by hand.

Do you know of any advice or guides on how to use the chatgpt api? It's unclear to me when things should be a system message, and if not, if they should be user or assistant messages.

You say here that the simple message should be really simple. So, further instructions should be in a user message? Any idea what constitutes a "simple" system message?

The other day, I was trying to build a web scraper, but I kept getting an error that the data wasn't utf-8 encoded. I gave chatgpt the error along with the first 10 bytes (e.g. b'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03') and it informed me that this was a gzip-encoded binary file, and then it wrote me working code to decompress the file.

This is extremely valuable and saved me hours/was the difference between me stopping or not.

There will surely be people that overhype this, but there is real value as well.

One tip to speed up how you're finding the closest matches -- openai's embedding vectors are normalized and all fall upon a sphere. So, the cosine similarity is equal to the dot product. Might speed things up a bit to use only the dot product.

I was also in STL when this happened. Had to drive 4 hours away to Evansville, IN in order to get a rental car. Just showed up here and they told me I was lucky to reserve it last night, they’re now turning away walk ups at the counter.

This makes sense. And the game sounds really interesting! What is it called? I may try it out.

I mean, I don't really disagree with you. For sure, most all services have in the history of society been mostly, if not entirely, consumed by the rich. For most of society, most everyone has been poor. If you could afford fancy stonework on your house, if you could afford someone to make your bed, of course you were rich.

The past century has seen an explosion of wealth in the world, with many people coming out of extreme poverty, emergence of large middle class in the US, etc. So, the world has gotten a lot richer. But, what does it mean to be richer?

Some things seem relatively obvious -- people have bigger houses. They get to have things like fridges and vacuums. These are tangible goods, for which the cost has fallen in line with increasing specialization and automation.

I'm saying that I think it's interesting that while a richer world can purchase more goods, it can't necessarily purchase more services, since the cost of labor rises in a richer world. Basically, productivity has to increase for people to be able to purchase the services with the rising cost of labor.

I'm not really concerned with the things I mentioned going extinct -- it's not a huge deal if we don't have stonework on buildings in NYC, and indeed if we can build housing faster than before by way of automation, that's great. But many people have wondered "why can't we build beautiful buildings in NYC like they used to?" and I'm basically saying, I think rising cost of labor is the reason, and it's interesting that this goes hand in hand with becoming a richer society.

Let's say there is a masseuse. They use their labor to give massages. They are also self employed, so they own their own business. If the cost of labor rises, the masseuse makes more money from giving massages, but this necessarily means that the price of their massages is going up (the cost of labor and the price of the massage are the same thing). Okay, let's take this one step further.

Let's assume that there is a massage parlor, with masseuses who do not own the business, and that the owner of the business has some profit. Cost of labor goes up, and now the business owner no longer has any profit, and it is earned by the masseuses. Yay! However, what happens if/when cost of labor goes up further, and there is no more profit? Prices will rise.

> You keep assuming that business owners can and will just raise prices and the markets will bear it, but this is not always the case.

I think this is my point actually. Markets won't always bear it. Sometimes they will raise their prices and the markets won't bear it, and then the business will no longer exist, and society won't be able to pay for it. E.g., the business of a hotel that makes everyones beds every day, or the business that builds a beautiful NYC building adorned with hand carved stone across the entire facade.

I think this makes sense with what I'm saying.

Laborers get more of the profits of companies and in turn get wealthier. However, the costs of services in turn go up (due to rising cost of labor), so those laborers (society) cannot actually afford more services (provided by people). Workers can more easily purchase goods which benefit from increasing automation and technology.

So, these workers which are capturing more of the profit can afford to buy fancier kitchen appliances and cook themselves better meals. But, can they all then afford to go out to eat more -- when going out to eat is primarily an endeavor dependent on the labor of others (and thus it becomes more expensive)?

There is a take that consistently comes up in discussions like this (and indeed is prominent several times on this post), which says "the problem isn't a shortage of workers, it's that employers don't want to pay workers enough." This seems somewhat limited to me, and there is an alternate explanation I would like to explore.

It seems to me rising cost of labor, while good for the people being paid that rising cost, can actually be bad for society overall, because you some businesses and industries will just disappear when the cost of labor goes high enough. And sure, you can respond that those businesses shouldn't exist if they can't pay workers enough, and yeah, fine. But the result is that we just then don't have those businesses (making society worse off?).

For this hotel example -- maybe cost of labor continues to go up, and it's much harder to automate making a bed than vacuuming a floor, and in 10 years, we all generally stay at hotels where robots vacuum our floors every day, but no one ever makes your bed for you -- that becomes a relic of the past.

I live in NYC, so a more prominent example I think about frequently is how we can no longer afford beautiful stone masonry on all our buildings. When labor was cheaper 100 years ago, it was easier to pay someone to carve stone -- now, not so much.

I want workers to be paid a lot so they can live good lives, but it's worth thinking about what happens to society as cost of labor continues to go up, and why the cost of labor is going up in the first place (Baumol's Cost Disease? Land rent?).

Other commenters arguing against you seem to be missing your point somewhat. I think right-sizing infrastructure to the community (and tax base) that supports it is a good and important idea.

It took me awhile to realize the necessity of building infrastructure this way. Strongtowns talks a lot about this -- for instance, see this piece about how many cities in the US cannot afford their own water infrastructure [1].

It makes sense that larger, more dense, more economically productive areas need and can afford different infrastructure than more rural areas. If you mandate or incentivize rural areas to use the same infrastructure as the more urban ones, you can often end up with problems.

[1] https://www.strongtowns.org/journal/2020/12/1/we-cant-afford...

This isn’t true. Science is ultimately the scientific method — make a hypothesis, test a change, observe the results, repeat. It’s an algorithm for learning and broadly gaining information about reality. It can equally be applied to things having to do with humans and things not having to do with humans.

That’s amazing that it can rhyme, without having the ability to hear.

How much of this is stable diffusion 2, and how much is something else? For instance, the text based masks, the syntax like AND and OR, the face up scaling — are these all part of stable diffusion 2 (and can be used via other stable diffusion apis)?

Any cookbooks or recipes you recommend? How does one figure out which different pieces of the meal go together well?

> “The idea that you might make synthetic fuel that has a higher energy density than petroleum while being cheaper is very alluring” Of course it sounds good - but reality isn’t that simple to twist to our desires. Impossible meat was more expensive (in spite of them running at a loss), didn’t taste as good, wasn’t any healthier, and by many accounts wasn’t any better environmentally. All downsides and no upside.

To me, it seems that the question comes down to energy efficiency. How many calories in are required to produce one calorie of beef? Our World In Data claims the efficiency is 1.9% [1], whereas Yale Center for Business and the Environment claims it's 2.5% [2]. So, it seems that if somehow you could figure out how to reconfigure plants into meat in a way that isn't as inefficient, you could come out ahead on cost. Now, maybe this is way too hard to do currently (I don't know since I don't know much of the science), but it at least seems like there could be an opportunity.

Another place where cost could cost factor in would be if governments were to enact a carbon tax (or, alternatively, tax meat, or subsidize plant based meats). If the actual cost to the environment could be reflected in the cost consumers pay, that's another opportunity for plant based meats to come out cheaper.

[1] https://ourworldindata.org/grapher/energy-efficiency-of-meat... [2] https://cbey.yale.edu/our-stories/disrupting-meat

You can do both at the same time. If Impossible could actually succeed in making a meat alternative that tastes as good as meat, but is cheaper (e.g. because they don't have to waste a bunch of energy growing an entire animal), then they could be poised both make a ton of money (by being a major player in the global food market) AND help end factory farming/animal consumption (by providing a superior product).

Furthermore, this is such a research heavy endeavor, you need upfront capital to fund it, which means it would be hard to structure as a non profit. Structuring as a for profit is the right move, and it doesn't mean what they're working towards isn't a good cause, or that they are somehow ethically compromised.

I am really not that knowledgeable about this space, but I would imagine it's because A) there are a lot of subsidies for the meat industry, and B) there are a lot of economies of scale present in the meat industry that do not yet exist in alternative meat industries.

Of course, it makes sense that when introducing a new product like this to the market, it is potentially going to be more expensive than existing products. It seems like, in theory, one should be able to make it much cheaper just because producing meat by way of animals is just very wasteful from an energy perspective. However, I have no clue what is required to synthesize such compounds without the use of animals, so I don't know how far away we are from being able to do this.

As someone who would want to be a vegan but struggles not to eat meat, I've been very interested in Impossible since it started. The idea that you might make synthetic meat that tastes as good as real meat while being cheaper is very alluring -- seemingly, you could eliminate most meat consumption globally.

At the same time, over the past few years, I've explored more of the world's cuisines and generally become a better cook. What I've learned (which surely is obvious to many but was not obvious to me) is that the typical US diet is unnecessarily meat centric, and that being a vegetarian is much easier, healthier, and tastier when one basically stops eating typical American meals. Being a vegetarian is much easier when eating Indian food for instance, and it's laughably easy when eating Ethiopian.

I'm still rooting heavily for Impossible, because factory farming is a global atrocity. And, it would be really cool if one day, I could eat an Impossible burger that was cheaper and healthier than a regular burger. But for now, and for the foreseeable future, I think that my attempts to be vegetarian will revolve around learning to better cook food from non-American cuisines, foods that are better suited to not having meat.

Heroku’s loss is Replit’s gain. They’re making everything so easy, I expect them to basically be able to pick up this slack in helping make deployments simple.