time_to_smile

What about blog spam written by human content writers?

The trouble is we've already had a web flooded with "ai content" long before GPT was public. Plenty of young writers have been trained to churn out thoughtless streams of writing based on prompts that appear to be written by an intelligent mind but are often filled with meaningless non-sense.

My industry specific example is Towards Data Science, content created by fleshy AIs that often looks very insightful at first glance, but when viewed by an expert ends up being mostly incorrect gibberish.

The problem lies with carbon's essential role as part of the energy cycle that powers the majority of this planet.

H2O + (Solar) Energy + CO2 => Useful hydrocarbons (everything form sugar to gasoline)

O2 + Hydrocarbons => Useful Energy + CO2

When you burn a log you're really using a solar battery that took potentially decades to charge and roughly 25x the energy you feel from the fire (photosynthesis is only 4% efficient).

The same process that feeds us powers the global economy, but at the cost of emitting CO2 in direct proportion to the energy we're using and benefiting from.

This is why "miracle" solutions are so unlikely. Because they require a major disruption of this process in a way that it's not clear is fundamentally possible. Anyway to massively remove CO2 from the atmosphere is fundamentally going to require energy. And because of the nature of inefficiency, will always be a poor use of any energy you used to create the problem.

This is obviously where things like nuclear fusion do provide the possibility of breaking this process because they create a lot of energy outside of solar powered carbon cycle.

Strong agree on the remote training being entirely possible. My first dev job was remote (long before the pandemic) and onboarding was not a problem at all.

In fact because you need people to get set up remotely, I find the documentation tends to be better at all remote companies. In-Office companies sort of assume that you can just tap someone on the shoulder if you get stuck so there's more often, in my experience, gaps in the documentation.

I particularly find this a strange claim since open source projects have been successfully onboarding new people remotely prior to there even being efficient ways to screen share/video chat etc.

Not to mention the added hypocrisy that at nearly every company I've worked at, big and small, C-levels are almost never physically in the main office building. Sometimes they're traveling the globe to work on making deals, but sometimes they just want to be at home with their family, or take a semi-vacation.

If you can run a company on the go or at home, certainly I am capable of shipping quality code at home.

> I don't understand how intelligent people can continue to believe this stuff is the future of finance.

My experience is that all of the really intelligent people interested in crypto did leave after the ~2012 wave of excitement. At that time when you saw people give talks on crypto they were almost entirely technical with very little focus (or interest) on becoming rich. That was when the people involved tended to be technical idealists. I didn't buy that crypto was the future then, but I wanted to be wrong.

Fast forward to ~2017 during the next crypto boom and the conversation was around the non-technical people at technical companies getting excited. People did believe crypto was going to become the currency of the world, so there was still some idealism, but it was mainly about getting in to get rich. By this wave a good chunk of the idealists I knew were entirely disillusioned.

Then the 3rd wave which just happened to correlated with a massive injection of money in the market by the Fed. At this point it was just literally get-rich-quick dreamers with more dollars in their hands than sense. Nobody I know who has gotten in during this period even has a coherent vision of what the future looks like, they just have too much money and think crypto is the way to get insanely rich one day. It's also when people completely unrelated to tech started getting involves. People who don't even know how to use a wallet, and rely 100% on 3rd parties to manage all of it.

People are holding on to crypto for the same silly reasons that coworkers of mine keep all their vested stock in companies that have dropped 50%+ in value over the last year. It's because they earnest believe that the era of low interest, free money is the norm. They believe this current macro is just a blip, and if they just hodl a bit longer it everything will go back to "normal".

For those curious what the big deal is here: PyTrees make it wildly easier to take derivatives with respect to parameters involving a complex structure. This makes it much easier to organize code for non-trivial models.

As an example: if you want to implement logistic regression in JAX, you need to optimize the weights. This is easy enough since this can be modeled as a single value, a matrix of weights. If you want to model a 2 layer MLP, now you have to use 2 matrices of weights (at least). You could treat this as two parameters to your function (which makes the derivative more complicated to manage) or you could concatenate the weights and split them up, etc. Annoying, but managable.

When you get to something like a diffusion model you now need to manage parameters for a variety of different, quite complex, models. It really helps if you can keep track of all these parameters in whatever data structure you like, but also trivially just call "grad" with regard to these and get your models derivative with respect to its parameters.

Pytrees make this incredibly simple, and is a major quality of life improvement in automatic differentiation.

A good chunk of this comes directly from Richard Hamming's incredible The Art of doing Science and Engineering (a video of the specific lecture on n-dimensional spaces can be found here[0]) and yet I see no mention of this talk anywhere in the article, which is unfortunate.

I highly recommend checking out Hamming's lectures if you find this enjoyable.

0. https://www.youtube.com/watch?v=uU_Q2a0S0zI

Problem is that everything needs X powered by clean energy, which means we need both a lot more clean energy and, if we want to slow/stop climate change, to massively reduce our use of fossil fuels... which requires a lot more clean energy.

We also, so far, globally have not shown any evidence of replacing fossil fuels with green energy, only supplementing them.

I'm still not entirely convinced that pipes aren't an anti-pattern. Absolutely an improvement over nested function calls:

a(b(c(d))) vs d |> c |> b |> a

but I'm not convinced pipes are better than more verbose code that explains each step:

step1 = c(d)

step2 = b(step1)

result = a(step2)

I've written a lot of tidy R and do understand the specific use cases where it really doesn't make sense to use the more verbose format, but generally find when I'm building complex mathematical models the verbose method is much easier to understand.

It's open source in the sense of OSINT [0]. Clearly confusing on a site like Hacker News, but this has been standard usage of the term for that community for a long time now.

0. https://en.wikipedia.org/wiki/Open-source_intelligence

I'm surprised how long the "solution in search of a problem" trend has dominated tech product design, despite obvious and repeated failures of this approach to produce results.

AI/ML products fundamentally don't make sense compared to products that happen to use some AI/ML to aid in solving a problem.

It's sort of like loving to use redis (which I do) and thinking you want to found a company based on using redis in the product, or start a redis product team, dedicated to shipping products that use redis.

It's one thing if you want to host redis as your business, which is solving a problem involving redis, but if your aim is to use redis to solve a problem then you're going to be in trouble.

Imagine a PM on the "use redis" team rejecting a great idea for customers because it could be more efficiently solved using a traditional database, or forcing the use of redis when a cheaper, easier solution already works just as well if not better. This is actually the case on AI/ML teams.

GPT startups that will thrive are the ones that aren't GPT startups, but instead solving some other, real, problem that happens to only be solvable in a post-GPT word.

You can implement SVM, gradient boosted decisions trees, and almost all classical models using the techniques of differentiable programming and it will have 0 impact on the amount of data required.

Massive Neural Nets do require a lot of data and are often not the best solution, but differentiable programming in general does not have higher data requirements than manually computing your derivatives or using OLS. You can still approach classical ML from the perspective of differentiable programming (and likely end up with a better sense of our how your models work in the end).

I'm a huge fan of classical AI, and adore PAIP, but this isn't really true if your goal is anything other than a deep understanding of AI in the most general sense.

While it would be great if everyone interested in the topic was well versed in the fundamentals, the truth is if you want to do anything from building something cool over the weekend to getting an actual job doing AI work, you're much better off starting not only with ML, but specifically with current SotA neural networks.

If you really want to get started in AI I highly recommend building even a trivial implementation of Stable Diffusion on your own. Not just because it's cool, but because at its heart it is an excellent demonstration of how current differentiable programming works. Diffusion models involve chaining together 3 separate models into an entire system that learns to solve a complex task. Once you understand this deeply, you can now solve a very broad range of tricky problems and are really approaching what we think of when we think of AI.

Differentiable programming is really the current pathway to any sort of AI solution to a problem.

I say this as the token "have you tried logistic regression?" guy in my org.

Out of curiosity, what's the use case here?

It seems like if the goal is to "play around with vector databases", why not just install it on your local machine? Part of using these tools is learning how they work and configuring them yourself.

If the goal is "start developing products using vector data bases" then it seems like you would surely want something a bit more under your control than using replit.

> Gone is the naive belief that rare/valuable skills secure higher salaries over a long period of time.

While I agree that this is a naive belief to have (at this point in my career I think there's almost a slightly negative correlation between skill and TC) the general talent pool for software engineers has, at least in my experience, dropped tremendously while TC has exploded.

The most important skills for getting high paying jobs in the last few years has been grinding leet code, then grinding systems design etc, etc. Software engineers no longer have "rare/valuable" skills, they have highly commodified, easily replicable skills (at least at the interview level).

Software engineers today simply aren't that skilled (at least on average) despite what they want to believe. It reminds me a lot of dotcom bubble where anyone with a pulse that could turn on a PC could get a high paying job.

Personally I think this is a strength of Haskell that many people don't really recognize and appreciate. It's a language the provides a lot of flexibility. If you want it to be a pure research language, it's happy to do that. If you need to make some sacrifices in regards to purity so you can get stuff into production, you can do that too.

As someone with a lot of time spent experimenting with programming languages, I can't think of any that can be so excellent from a research/experimentation perspective that also share the real production usage that Haskell sees. Take for example Racket, an amazing and also extremely flexible research language. It's probably easier to get started with than Haskell, but has never seen the real world usage that I've seen from Haskell.

I worked for a travel startup for a bit and after that experience I only book airfare and hotels directly.

Specifically with airfare, a 3rd party is not allowed to sell for less than the airline directly, so it's always better options since it is much easier to reschedule/cancel/refund directly with the airline. Plus, if you travel a lot, it is better to find a favorite airline and stick to them. Any bonus "features" offered by a 3rd party I can assure you are either not in your interest or actually a scam.

I don't know if the pricing rules applies to hotels, but I'd rather pay extra then get to the hotel and be screwed over last minute because some 3rd party is trying something "clever" behind the scenes and it turns out it ruins your travel plans.

Thanks for the correction!

I have a hard time believing you have serious experience working in a tech company anywhere near leadership.

I've worked at a pretty wide range of tech companies throughout my 15+ year career. When I was young I would ask myself "what is leadership thinking!?", my realization later in my career was simply: "oh, they're not thinking"

14 year old and younger tech companies have only existed during a tech boom period, when money flowed easy from both investors and customers. There has been zero market pressure to put thought into building products.

In the case of dropbox in particular, any long term customers (such as myself) can confirm that there has clearly not been a coherent product strategy for at least the last few years.

oh I fully agree. I'm just pointing out that even the defining example of "moonshot" took some serious dedication of resources and time.

What's really wild is that the entire Apollo program cost ~$25 Billion in 1973 (of course est. $163 billion today) while Meta already spent $36 Billion on the Metaverse [0].

0. https://www.businessinsider.com/meta-lost-30-billion-on-meta...

Metaverse has already been deprioritized, and Meta is shifting to generative AI [0].

I was always a huge skeptic of the Metaverse conceptually, but it was a moonshot project (and I'll be the first to admit, skeptics are often proven wrong with moonshots). The problem is you can never have a successful moonshot if it can't survive a few bad quarters.

I mean the actual Apollo program took 8 years to achieve its goal of getting humans on the moon.

0. https://qz.com/meta-layoffs-2023-jobs-metaverse-ai-185019657...

Don't love it, it's not correct.

> what the reward / punishment system really equates to

Nothing, and least as far as neural network training goes. This is an extremely poor analogy regarding how neural networks learn.

If you've ever done any kind of physical training and have had a trainer sightly adjust the position of your limbs until what ever activity you're doing feels better, that's a much closer analogy. You're gently searching the space of possible correct positions, guided by an algorithm (your trainer) that knows how to move you towards a more correct solution.

There's nothing analogous to a "reward" or "punishment" when neural networks are learning.

> I'm sure it leaves enough out to make most experts angry

It's not that it leaves out details, it's that the articles metaphors are not actually correct in regards to the way deep learning works.

This post mostly confuses both reinforcement learning and ensemble models with deep learning. If you only enough "enough to be dangerous" then this post will steer your intuition in the wrong direction.

> This is how neural networks work: they see many examples and get rewarded or punished based on whether their guesses are correct.

This description more closely describes reinforcement learning, rather than gradient based optimization.

In fact, the entire metaphor of a confused individual being slapped or rewarded without understanding what's going on doesn't really make sense when considering gradient optimization because the gradient wrt the to loss function tells the network exactly how to change it's behavior to improve it's performance.

This last point is incredibly important to understand correctly since it contains one of the biggest assumptions about network behavior: that the optimal solution, or at least good enough for our concerns solution, can be found by slowing taking small steps in the right direction.

Neural networks are great at refining their beliefs but have a difficult time radically changing them. A better analogy might be trying to very slowly convince your uncle that climate change is real, and not a liberal conspiracy.

edit: it also does a poor job of explaining layers, which reads much more similar to how ensemble methods work (lots of little classifiers voting) than how deep networks work.

Leetcode is also just an awful signal. It would be fine if it was only problems with false negatives, but the false positive rate is through the roof.

I'm consistently shocked by coworkers I've had, who I knew had to pass challenging leetcode interviews, who seem to not only know nothing about building real world software, but don't really have a firm grasp on algorithms.

Yes they can process a leetcode question requiring dynamic programming in record time, but when mapping a real world problem to a dp (or any other similar solution) are completely at a loss.

"Tech Bosses" are dictators and I'm always surprised that most people have absolutely zero intellectual challenge suspending the values of democracy for 8-12 hours a day.

Virtually all of the rights we learn about in grade school are suspended in the workplace. And while employees do have the ability to quit, they a.) still have to find some other dictator to work for and b.) are much more heavily impacted by a loss of income than the employer is the individual loss of skill.

So the dictators we work for day in and day out are aligned with other dictators. You aren't entitled to free speech in your office, so why would you expect to the people running your office to care about your free speech after hours?

edit: I'm still surprised that people are incapable of question the ideology that shapes your worldview. The very concept that "well you're working for someone else so the of course suspension of liberties is okay" is doctrine that you have been lead to believe since birth precisely because it benefits those people in power.

Whenever I see this comment on HN I always am a bit surprised: Do you not have any friends from/in China?

This conspiracy theory always gets a good chuckle out of me and my Chinese friends whenever it comes up. I have a good chunk of my family living in China and I can assure you, Chinese kids watch just as much if not more junk over there than they do in the US.

This is similar to trying to argue that America kids only play wholesome games on the Xbox or Youtube because they have parent controls.

But to my first point: surely working in tech right now you have a few Chinese coworkers. I recommend you grab lunch with them sometime (and let them choose the place to eat!)

I fully agree and came here to post something very similar. Advanced (or even non-trivial) prolog requires such a deep understanding of what prolog is doing under the hood that, at that point, you should be able to implement whatever you're trying to solve just as easily in your favorite non-prolog programming language.

What makes this problem really bad is that everything about your problem that isn't directly related to logic programming is much harder to do in pure prolog. Just reading in a list of facts from a CSV file is non-trivial in prolog.

In the end Prolog makes trivial but hard problems easy, while non-trivial hard problems remain hard to solve, and easy problems, even some trivial ones, also remain hard.

I still love Prolog as well because it really does change the way you think about programming, but it is very far from even Haskell in allowing you to extend these new ways of thinking to solving real world problems.

Because humans (and your language is confusing here between individual humans as models and human society as the model) are constantly receiving new information. Even before humans learned "some berries are bad" they initially thought either "all known eaten berries are bad" or "all known eaten berries are good", then somebody at a good/bad berry and updated their believes (i.e. their model of the world).

Humans learn from other humans because humans don't individually share the same information and model of the world. A science teach can speed up how you learn science by taking the compressed information and explaining it quickly (essentially what is happening in the post), but out scientific model is expanded when we have experiences that call into question the strength of our current model.

However both individually and as a society we are constantly taking in new information (sometimes more sometimes less) and using that to update our model.

> You prompt the LLM to generate a bunch of arithmetic questions, prompting it to show its working. Then you take that output, remove the intermediate steps, and train on the results.

Removing or editing the output of the model is providing new information to the model, what's improving the performance of the model in this scenario is that you are explicitly adding new information and fine tuning it on these new cases.

> information theory is not especially relevant here

It's extremely relevant because people seem to be arguing with about mathematical facts as though they were somehow opinions.

You cannot improve the performance of a model without adding new information to that model.