throwaway4aday

Agree. It's bizarre how everyone jumps to the conclusion that increased demand for electricity is a bad thing and can only be solved by limiting its use. The amount of electricity available is determined by how much we choose to produce and we're far from the theoretical limit on production. The obvious answer should always be to build more generating capacity whether that's nuclear, solar, hydro or anything else. It seems to be a uniquely Western perspective as well since everyone else in the world is busy increasing their generating capacity as a matter of course. We should be doing the same and laying the legislative groundwork to make it easy to build more power plants instead of trying to shoot ourselves in the foot over and over again.

Depending on what you learned first, dots and parentheses are a lot simpler to understand than math expressions.

A lot of what you need is already written, you just need to find the right libraries and write the bindings. From my encounters with Python ML it seems like "just use Pytorch" is a bit like "simply walk into Mordor".

If you're using Node you can write whatever you want in C++ and then add a binding to call it from within your Node app. Don't need WebGL.

The benefit of having it in JS is not speed but portability and access to the JS ecosystem of tools. Having the code run in the browser without needing a complex setup is a huge benefit for sharing demos. Node.js provides a way to use native code as well and it's quite commonly used https://github.com/nodejs/node-gyp so there's no reason you couldn't use those same or similar libraries in a JS implementation.

If each of those operators were implemented as functions then you'd have different names for different implementations in order to avoid confusion over what type of division or multiplication they were performing. It's more verbose but that's a good thing since it prevents you from making incorrect assumptions about what's going to happen when you do a * b.

Wow! Thank you for doing this. It looks like a great starting point for anyone approaching deep learning from the JS ecosystem. It is very plainly written and looks like it will be a joy to learn from. Thank you for adding JSDoc comments with type hints!

Are you open to pull requests? If I have the time I'd love to contribute. I'm sure others would as well.

You should write up a short article on this, even something really simple like one of the examples in the README but with some commentary and examples of output and then post it to a few places like https://dev.to/ or maybe https://hashnode.com/ or even Medium (even though I'm not a big fan). There aren't many newer implementations of PyTorch in JS and I've been looking for one to learn from for some time so I'm sure there are a lot of other JS/TS developers out there that would be interested. Getting to the front page of HN certainly helps but having an article somewhere will help everyone after this week find it through a Google search.

Again, thanks so much for doing this work! It's really helpful to have everything spelled out in JS for those of us who haven't used Python much (I'm sure Python devs can relate when they think about JS projects).

What's wrong with creating a function that does those things? It would be less surprising to people new to the library, would be self-documenting by having a name and an easily inspected declaration with named arguments, and it would be idiomatic JS. You could also have variants that are purely functional and return a new value or ones that mutate in place that you could use depending on your needs.

Interesting that this involves a response similar to an immune response to a pathogen. I've read a couple articles about alternate theories of Alzheimer's linking it to an increased immune response in the brain.

I think you're talking to the wrong point. These memories aren't being encoded in germ cells, they are after the fact changes to DNA in mature neurons which have completely differentiated. I would think it's very possible at that stage of development for them to add or remove segments of DNA in order to encode new information not related to the development of the cell as long as it didn't interfere too much with parts that are actively used for the ongoing upkeep of cell activity. It would need to alter how the cell functions a little bit for the changes to modify the neuron's ability to process signals though.

As many others have noted, once you've got everything stable (and hopefully profitable) you should seriously explore a way to use this as input into any text field in any program. Microsoft is actively experimenting with something similar in Copilot Voice although theirs is very integrated with the editor and specialized for code. It would be great to have these types of voice interfaces in all software. Maybe you could look at providing a way to integrate with your system through an API so others could do the heavy lifting of creating a native experience for each app?

Absolutely amazing product by the way! The 1000 free tokens is enough, the fact that people are complaining about running out too soon is good, it shows that they like the product and want to use it more. They do have a point about adding a rough word count, maybe just a subheading that says "on average, X spoken words".

I wouldn't say always. The way the "confidently wrong" answers make sense to me are in the context of asking people trick questions that seem to have an obvious answer but actually require more deliberate thought to get right. What happens in that case is you are engaging the part of the brain that has simply memorized a fact or developed a quick but not always accurate heuristic which it then applies to the question that seems to meet the criteria and it results in a confidently wrong answer given by a real meat and bones human. LLMs right now are essentially a big collection of facts and heuristics with the ability to find the often right association between input and the applicable output. Everyone is working on building the other thing that does the careful step by step reasoning and error checking, the so called System 2, and if brains can tell us anything about the implementation of such a system it is that it's made of the same stuff that System 1 is but arranged and controlled in such a way that it produces more reliable answers.

*edit: I should specify that when I say LLMs are a collection of facts and heuristics I mean they are a collection of those things encoded as language which itself has been encoded as vectors of floats which in turn have modified the weights of the network to produce yet another encoding. I don't mean that the facts and heuristics are stored in a lookup table or as procedures.

From the phrasing of the second quote it seems like GPT-5 might be synonymous with their conception of AGI or at least agentic AI. Maybe it was just bad phrasing and he was using GPT-5 as a stand in for the ultimate goal but to me at least it seems like they're holding back on it because they're still trying to get the agentic and reasoning parts right. I think I agree with his musings about Lex's reaction that maybe they aren't releasing iterations fast enough for them not to look like big leaps. I would think they'd have something by now that could at least be labeled GPT-4.5 and be deployed as an incremental improvement. Maybe now with Claude 3 Opus on the scene they'll need to ship something soon even if it isn't a huge improvement.

Nice work! I built something similar years ago and I did compile the probabilities based on a corpus of text (public domain books) in an attempt to produce writing in the style of various authors. The results were actually quite similar to the output of nanoGPT[0]. It was very unoptimized and everything was kept in memory. I also knew nothing about embeddings at the time and only a little about NLP techniques that would certainly have helped. Using a graph database would have probably been better than the datastructure I came up with at the time. You should look into stuff like Datalog, Tries[1], and N-Triples[2] for more inspiration.

Your idea of splitting the probabilities based on whether you're starting the sentence or finishing it is interesting but you might be able to benefit from an approach that creates a "window" of text you can use for lookup, using an LCS[3] algorithm could do that. There's probably a lot of optimization you could do based on the probabilities of different sequences, I think this was the fundamental thing I was exploring in my project.

Seeing this has inspired me further to consider working on that project again at some point.

[0] https://github.com/karpathy/nanoGPT

[1] https://en.wikipedia.org/wiki/Trie

[2] https://en.wikipedia.org/wiki/N-Triples

[3] https://en.wikipedia.org/wiki/Longest_common_subsequence

Features like function calling are moving in that direction. Microsoft also seems to have plans to deeply integrate LLMs into its OS and if they do a good job it could become a primary way to interact with its features and programs. Considering the progress made on image generation models I could image a special purpose model that is specifically trained on operating APIs and producing good results. The big hurdle would be building the APIs that don't exist for the tools that people like to use. I'm sure there are interesting ways you could think of generating labeled data for actions in various programs.

We need to produce and use more energy, not less, not the same amount, more. At the most basic level, our survival depends on it to produce the things we need: food, water, heat, shelter, but beyond that the quality of life and the ability to thrive for every human being on the planet depends on using more energy. We should continue to strive for efficiency as this is equivalent to an increase in production but we should not starve ourselves of it.

Natural resources exist to be utilized. Once again, they provide the necessities and also the comforts that all deserve. If we limit our energy use our ability to extract natural resources will suffer. The resources we can access grow in proportion to the amount of energy we make available. No where is this relationship more direct than in the production of fresh water via desalinization and that alone should be sufficient incentive to utilize more energy. It takes resources and energy to develop more resources and produce more energy, you can't stop it or reverse it, you need to keep moving forward.

Fiat currency by definition is infinite being created by decree.

The rest of the points are increasingly wobbly so I'll leave you with this exchange from the comments on his page:

---

Christopher Toth You say that GPT-3 training consumed 700,000 liters of water, as if that is a large amount. With five seconds of research, I found that the global average water footprint for beef is around 15,415 liters of water per kilogram of beef produced, so an average cow costs >4.6 million liters of water. For a single cow. I am disappointed in your inability to contextualize the numbers you use.

Gary Marcus dude the context is that it will be way more for gpt-4, gpt-5 etc, but maybe you were unable to read that far.

Christopher Toth Okay, so can you speculate as to how much more water? Three cows worth? Ten cows worth? A hundred cows worth of water to train GPT-5?

Turns out we kill 900,000 cows every day, so around four trillion liters of water are used for beef production for a single day.

Do you expect GPT-5 to use more than this?

Otherwise why ever would you mention it other than because it looks like a large number to the uninformed?

---

How many cows indeed.

Check out the "In the Wild" section on that page, here's the first link https://github.com/nalgeon/sqlime

What about the official project? https://sqlite.org/wasm/doc/trunk/index.md

Kinda neat but it seems a lot of "cyberdecks" are now just converging on "laptop" or "palm pilot". The essence behind a cyberdeck is its retro-futuristic design which produces an anachronistic feeling like some out of place object from another timeline dropped by a multiverse traveler. The Lisperati1000[0] nailed it with its surprising screen dimensions, form factor, color and keycap choices, and using it for Lisp programming. For a commercial solution it's hard to beat the Cardputer[1] with its chunky off white case, riotous multi-color labelling, quirky features, inscrutable purpose and the fact it comes in a blister pack like it's something you'd pick up in a gas station convenience store in an alternate 1988.

[0] https://www.hackster.io/news/the-lisperati1000-is-a-cyberdec...

[1] https://shop.m5stack.com/products/m5stack-cardputer-kit-w-m5...

True, I was thinking this over and I can see where surprise plays a part in curiosity but I don't think it's the main driver. I think your example shows this because when I click on the link out of curiosity it's because I don't have a good prediction for what will be on the other end of it. If I were a simple neural net then the outputs I would produce when seeing that link would be mostly similar across all the categories I could predict since I've never seen this url and there is little in it that allows me to predict what type of content it leads to. I am surprised to see the link since I couldn't predict it, so that's a point in favor of this approach, and I am also surprised by the content I see after clicking on it although only mildly because I didn't have a good prediction to start with. You're right that after I click on it I have very little curiosity to find out more about it but I think that is the key difference. The rest of that site remains an unexplored place, I've only seen one page of it which it seems would be a very small fraction of all of it. So why am I not curious about the rest of it and more crucially, what would make me curious to explore it? I could imagine someone with a particular love of mathematics being able to exercise their curiosity on that page but is that because they would be surprised by what they found after clicking on each link or entering a new sequence? What would drive them to explore it? I think that speaks to the problem, if you optimize for surprise then the ideal reward is paying attention to an infinite number of TV channels (or a page with a bunch of links you've never seen before leading to different pages you've never seen) but I wouldn't call that curiosity.

I'm trying to imagine the simplest case, say a button you could press and every time you pressed it something entirely random would happen, always guaranteeing surprise. It would have a great deal of novelty at first but after a while it would cease to hold your attention even though your prediction of what would happen would never be accurate. I'd bet that after a while you might even never bother to push it again. The only way you would be convinced to push it consistently would be a) if you were assigned a reward for pushing it e.g. money in which case it is a slot machine or b) if by pushing it you could somehow reduce your uncertainty about what would happen which would as a by product reduce your surprise.

Thinking about it this way, surprise is certainly a key element at first. It grabs your attention initially but it doesn't hold it. What keeps you focused on exploring the thing that surprised you initially is the learning process which involves reducing prediction error i.e. reducing surprise. So there is a tension between the two.

The combination probably makes for a good exploration strategy. Initial surprise, look for a learnable pattern and follow it until another surprise, maybe backtrack and try other familiar patterns until those are exhausted and then investigate each sequence that led to a surprise by recursing through these steps.

This would also explain the example where my curiosity was prompted by the unknown link but I was not motivated to explore further. The website wasn't interesting to me because it was too unfamiliar and I wasn't able to find any familiar routes to explore through it due to my lack of interest in that area of mathematics but our hypothetical mathematician with a fondness for integers would see lots of familiar patterns they could explore attached to which are likely some enticingly unknown and surprising links.

Thanks for the prompt to think about this more!

Using prediction error as the definition of curiosity rings hollow for me. Curiosity in my mind is more about mapping out an unexplored thing and not about being surprised.

Mincing words. The term is necessary when you have to discuss whether someone has the requisite knowledge to do-the-thing. If someone has a heart attack in a theatre you ask if there are any doctors in the audience. If the whole flight crew dies from food poisoning you ask if any of the passengers are pilots. They're not trying to hurt your feelings, they're just asking if you can build the feature, debug the system, or handle the deploy.

They're not compelled to but they don't want to invest in stocks due to bad experiences in the past. Now there is a bad experience with investing in real estate.

You have to store wealth somewhere otherwise you'll constantly be losing wealth. Ideally, one could simply bank their savings without it losing value while also investing in a variety of other assets rather than putting everything into stocks or real estate but that's not the world we live in at the moment.

If you believe people should be paid for their work then you should also believe that the money they are paid for their work should not be bled away from them.

Not a great comparison, custom GPTs use plugins or function calling similar to how they use code interpreter. You can't really compare them because plugins or functions are a tool that GPTs use in addition to other features. The advantage a custom GPT has is it is easy to set up but it comes with big disadvantages such as having to use their RAG system which is very opaque and only being able to use one system message. Building with the assistant API can be far superior but requires a lot more effort and skill in building your own APIs.

The US has a long history of public housing as well and it suffered and suffers from very real problems.

Nationalizing inner city land would likely have many unintended consequences. I could see it resulting in migration of people and businesses away from the nationalized area only to regroup at a new center that wasn't restricted by direct governmental control. Reasons for that that spring to mind are that planning permissions would certainly be a nightmare, difficult to obtain and extremely slow. If there was a hard quota of apartments vs commercial space it could create a bad situation for businesses and drive them away. The types of residential units may not meet the demand which could drive segments of the population to seek housing elsewhere even if the costs were higher. Poor design decisions for the units constructed could produce areas conducive to crime or just make it unattractive or inconvenient.

There's a long history of national building projects in many countries that have created problems like these.

Ghost cities are a good example of how simply building housing is not a perfect solution. If the housing is somewhere where people don't want to be then it is worthless and also useless since no one can make use of it if they can't make a living there.

How is the East better positioned when this article is about a massive failure due to property speculation? It just took a different and exaggerated form in China due to the rightly perceived shakiness of Chinese stocks and the limitations on land and property ownership. If this development cements in the minds of the Chinese populace that buying an extra apartment as an investment is no longer a safe place to store their money then who knows what's going to happen next. If you can't keep your money in stocks and you can't keep it in real estate then how do you invest for your future? This is an even more pressing question due to their rapidly aging populace with the prospect of little to no support from their children and non-existent grand-children.

The real-estate market in the West may be bonkers but at least someone in their prime earning years can invest in stocks if they can't purchase a home. That has its own risks but with a little knowledge you can mitigate them and with patience you can wait them out and even benefit from them.

This is a big missing piece of the puzzle and it could all be fixed by simply repealing legislation.

> In 1917, California passed a new hotel act that prevented the building of new hotels with small cubicle rooms.[12] In addition to banning or restricting SRO hotels, land use reformers also passed zoning rules that indirectly reduced SROs: banning mixed residential and commercial use in neighbourhoods, an approach which meant that any remaining SRO hotel's residents would find it hard to eat at a local cafe or walk to a nearby corner grocery to buy food.

For low density urban areas they aren't. It's not only major cities that have homeless populations you know.