HNHacker News
TopNewBestAskShowJobs

musebox35

198 karma · joined June 14, 2023

submissionscomments
musebox35··on Andrej Karpathy – It will take a decade to work through the issues with agents
waymo already has driverless taxi service in a major us city and is expanding. Tesla is in the process. again this is if they cover the last 5%. Scalability arguments wont matter when they can not launch such a service. And no, cmos cameras are close but are not better than the human eye in low light unless you have an ir camera and can flood everywhere with active ir lights. they are certainly inferior in dynamic range. I have been doing vision for more than two decades and I would not be comfortable in a camera only robotaxi at high speed. Certainly not at night or under adverse weather conditions. But this is all speculation of course. Considering fully autonomous driving at scale has been a major unrealised promise for the past 10 years, I stand by my assessment until I see a major advancement in camera technology or affordable active sensors.
musebox35··on Andrej Karpathy – It will take a decade to work through the issues with agents
And that last 5% is the toughest nut to crack. There is a reason waymo is way ahead even if they can not scale. Cameras are passive devices with relatively poor dynamic range and low light behavior. They are nowhere near a match/replacement for the human eye. Just try to picture a 5 year old at dusk or indoors and what you see will not be what you get.
musebox35··on Andrej Karpathy – It will take a decade to work through the issues with agents
Honestly, if you have any actual interest in LLMs or other generative ai variants, just go after a concrete goal post that you yourself set with measurable metrics to gauge your progress. Then the predicted timeline from podcasts and blog posts will become irrelevant. Experts and non-experts have both been terrible at predicting timelines since the dawn of ai. Self driving cars and llms are no exception. When you are making predictions based solely on intuition and experience it is mostly an extrapolation. It is not useless. It always helps to ask questions and try to frame the future within the bounds of our current understanding. But at the same time it is important to remember that this is just speculation, not empirical science. That is also why there is such varied opinions on the topic of ai timelines. Relax and enjoy witnessing a major leap in our understanding of natural language, vision, and high dimensional probabilistic vector spaces ;-)
musebox35··on America is getting an AI gold rush instead of a factory boom
AFAIK, certain abilities such as understanding arithmetic manifest at discrete scale points even though there is a continuous build up of potential. There is also the more remote possibility of a discrete scale that AI takes over its own training or at least starts to contribute substantially. A lot of real world leverage and arbitrage depends on such discrete surprises that may not be visible during the continuous incremental evolution. I think this principle holds computationally as much as it does biologically.
musebox35··on America is getting an AI gold rush instead of a factory boom
While I agree with the general sentiment on throwaway compute infra, the generated know-how with large scale experiments is not thrown away. I think a lot hinges on the scaling laws and whether you will hit the jackpot at a certain scale before everyone else. This is hard to guesstimate so someone has to do it in the spirit of empiricism. This might sound a lot like gambling or exploring depending on your sentiment. So, I think it is more justified to criticize the scale and the risks than the spirit of these investments.
musebox35··on Examples are the best documentation
I think conceptually diataxis is brilliant. However, it is not trivial to implement. Every project needs a varying ratio of each component and stacking all forms in a single website in the same format is very ineffective. The ratio also evolves with community adoption and expertise level. I really wish it was simply more actionable. Documentation is a hard problem, maybe we will figure out a better way one day. Until then docs will only be as good as the amount of time and expertise spent on them, which is usually not as high as it should be due to resource constraints.
musebox35··on The death of industrial design and the era of dull electronics
Brilliant, I have always felt that one of the major problems with machine learning, consequently LLMs, is the boring average based loss functions that under-represent the unique and the rare. It seems our collective civilization is using a similar function and heading in the same direction of optimizing for the average.
musebox35··on Reader Response to "AI Overinvestment"
That is a fine point. However I am not sure if replacing the gpus themselves will be the bottleneck investment for datacenter costs. After all you have so much more infrastructure in a datacenter (cooling and networking). Plus custom chips like tpus might catch up at lower cost eventually. I think the bigger question is whether demand for compute will evaporate or not.
musebox35··on Bayesian Data Analysis, Third edition (2013) [pdf]
I found the book from David Mackay on Information Theory, Inference, and Learning Algorithms to be well written and easy to follow. Plus it is freely available from his website: https://www.inference.org.uk/itprnn/book.pdf

It goes through fundamentals of Bayesian ideas in the context of applications in communication and machine learning problems. I find his explanations uncluttered.

musebox35··on Context is the bottleneck for coding agents now
Context is also a bottleneck in many human to human interactions as well so this is not surprising. Especially juniors often start by talking about their problems without providing adequate context about what they’re trying to accomplish or why they’re doing it.

Mind you, I was exactly like that when I started my career and it took quite a while and being on both sides of the conversation to improve. One difference is that it is not so easy to put oneself in the shoes of an LLM. Maybe I will improve with time. So far assuming the LLM is knowledgeable but not very smart has been the most effective strategy for my LLM interactions.

musebox35··on Getting AI to work in complex codebases
"The prompt could be perfect, but there's no way to guarantee that the LLM will turn it into a reasonable implementation."

I think it is worse than that. The prompt, written in natural language, is by its very nature vague and incomplete, which is great if you are aiming for creative artistry. I am also really happy that we are able to search for dates using phrases like "get me something close to a weekend, but not on Tuesdays" on a booking website instead of picking dates from a dropdown box.

However, if natural language was the right tool for software requirements, software engineering would have been a solved problem long ago. We got rightfully excited with LLMs, but now we are trying to solve every problem with it. IMO, for requirements specification, the situation is similar to earlier efforts using formal systems and full verification, but at the exact opposite end. Similar to formal software verification, I expect this phase to end up as a partially failed experiment that will teach us new ways to think about software development. It will create real value in some domains and it will be totally abandoned in others. Interesting times...

musebox35··on Nvidia is full of shit
With the rise of LLM training, Nvidia’s main revenue stream switched to datacenter gpus (>10x gaming revenue). I wonder whether this have affected the quality of these consumer cards, including both their design and product processes:

https://stockanalysis.com/stocks/nvda/metrics/revenue-by-seg...

musebox35··on QuACK: A Quirky Assortment of Cute Kernels
CuTe DSL examples from the MIT Dao-AILab for writing high performance cuda kernels using Python (see https://github.com/NVIDIA/cutlass for more background info).
musebox35··on TPU Deep Dive
From the acknowledgment at the end, I guess the author has access to TPUs through https://sites.research.google/trc/about/

This is not the only way though. TPUs are available to companies operating on GCP as an alternative to GPUs with a different price/performance point. That is another way to get hands-on experience with TPUs.

musebox35··on TPU Deep Dive
I think https://jax-ml.github.io/scaling-book/ is one of the best references to go through. It details how single device and distributed computations map to TPU hardware features. The emphasis is on mapping the transformer computations, both forwards and backwards, so requires some familiarity with how transformer networks are structured.
musebox35··on Highly efficient matrix transpose in Mojo
I totally agree that the resulting kernel will be rarely useful. I just wanted to highlight that it is a commonly used educational exercise to showcase how to optimize for memory throughput. If the post showed how to fuse a transpose + rmsnorm epilogue to a gemm then the kernel would be more functional but the blog post would be much harder to follow for newcomers.

Jay Shah’s later articles contain examples that involve epilogue fusion. IMHO, understanding how to write an efficient transpose helps with following the more involved ones.

musebox35··on Highly efficient matrix transpose in Mojo
Matrix transpose is a canonical example of a memory bound operation and often used to showcase optimization in a particular programming language or library. See for example the cutlass matrix transpose tutorial from Jay Shah of flash attention 3 paper: https://research.colfax-intl.com/tutorial-matrix-transpose-i...
musebox35··on Introduction to CUDA programming for Python developers
I suggest having a look at https://m.youtube.com/@GPUMODE

They have excellent resources to get you started with Cuda/Triton on top of torch. It also has a good community around it so you get to listen to some amazing people :)

musebox35··on Beating cuBLAS in Single-Precision General Matrix Multiplication
Considering recent developments in GPU hardware (Tensor Cores for GEMM), another hardware accelerated algorithm is ray tracing for photo-realistic rendering. As far as I understand the Ray Tracing Cores provide an efficient hardware implementation of ray-triangle intersection, pulling data from a Bounded Volume Hierarchy (https://en.wikipedia.org/wiki/Bounding_volume_hierarchy).
musebox35··on Please don't mention AI again
Fun to read. The style of the article reminded me of https://scholar.harvard.edu/files/mickens/files/thenightwatc...
musebox35··on Successful room temperature ambient-pressure magnetic levitation of LK-99
It seems to me that technological developments and empirical scientific breakthroughs come in cycles, technology making it cheaper to experiment, science reducing cost of new technical developments. I would be happy to hear about pointers to any source discussing such a tech/science cycle.

I feel our generation (I am in my mid-forties) lived through enormous technological advancements but not as many scientific breakthroughs as the previous generation. So maybe it is not surprising that we are suddenly more likely to have breakthroughs in basic science.

I hope there is a phase transition to science mode now, so we that have a chance to solve the hard pressing issues.

musebox35··on A non-mathematical introduction to Kalman filters for programmers
For a robotics oriented, part theory part hands-on learning material on Kalman Filtering, I would suggest The Robot Mapping course from Cyrill Stachniss:

http://ais.informatik.uni-freiburg.de/teaching/ws22/mapping/ http://www.ipb.uni-bonn.de/people/cyrill-stachniss/

There are YouTube videos from 2013 here: https://www.youtube.com/playlist?list=PLgnQpQtFTOGQrZ4O5QzbI...

The course itself is mostly based on the Probabilistic Robotics book from Sebastian Thrun et al: https://mitpress.mit.edu/9780262201629/probabilistic-robotic...

musebox35··on Excellence is a habit, but so is failure
This seems like a specific application of the "inverted thinking/inversion" mentality. I once came across a post on this but can not find it now. I think it was about this article: https://fs.blog/inversion/

Edit - Or this one: https://jamesclear.com/inversion

← PreviousPage 3 of 3