Jacob Ziv has died
twitter.com
twitter.com
Kudos, to the both of you.
I'm sad we're starting to rapidly lose so many of our titans.
For all kinds of reasons, the inventor (who is sometimes by now rich and/or powerful), may have a specific vision or path they want to proceed down. When they die, other people start investigating other paths, often with more success.
I wonder if tech will see the same?
New creations in tech have to wait for patents to expire and the incumbents to crumble to the point where they don’t have the cash to buy out the competition any more.
The patent harassment might even be a factor in the curiosity mentioned elsewhere in this thread, that a lot of practical progress in file compression has been published in a Russian web forum.
Unisys[0] claimed the patent (for all kind of uses) and had the patent for many a year, PNG was created not to use lzw since it was patented. The algorithm was patented for the encoding part but not decoding.
[0]: https://eng.libretexts.org/Bookshelves/Electrical_Engineerin...
The most surefire way to kill a young hacker’s spirit is to force them to hack on things they don’t care about. This goes with anything, not just computing. To whit, a childhood friend of mine was a gifted pianist who really loved jazz. His parents forced him to study with a classical teacher. Guess who doesn’t play piano anymore?
as a kid who played chess, the kids who were 'forced' to play chess -- the ones who came from a long chess playing family and had different scenarios drilled into their head from the day they could talk -- were much more talented at chess than I.
Their talent persuaded them to continue pursuing it far longer than I ever did, with many of them still involved.
The most involvement I have now is a friendly lichess game every few months.
It was their talent that allowed them to find more enjoyment within the pursuit than I ever did, and that talent was essentially forced by their immediate family.
So, essentially, I am just saying that forced persuasion thing that parents' sometimes do is a double-edged sword; it can create animosity and bitter hatred but it can also create the drive and motivation for boundless talent and personal satisfaction.
There’s such thing as pushing too hard, but nobody got buff without being pushed by those around them
This is completely contrary to my experience. The successes I’ve had were my own motivations and drive.
I sincerely hope that you were just ignorant of the help you received along the way. A child with absolutely no support or encouragement is a very depressing thought.
And I did not have bad parents. Many actually hinder their childrens development, instead of being ambivalent to it.
Isn't the general term for that a "problem with authority"? ;)
- Dad, I'm so done with this Snake game, give me some other task?
- Ok, how about making that LZW thing we talked earlier?
- Cool.
I picked up skills after my parents because I was curious and they were both knowledgeable in their respective fields(my dad, a medical equipment engineer, could repair a TV, which for a five year old amounted to a superpower).
I was never forced to do anything and yet had the exact same headstart you're talking about, but in e.g. school math.
Of course that did a lot of damage to my work ethic, but the point is that kids are naturally curious and, at least initially, want to do what their parents do.
https://en.m.wikipedia.org/wiki/Judit_Polgár
Judit Polgár (born 23 July 1976) is a Hungarian chess grandmaster, widely regarded as the strongest female chess player of all time.
Polgár and her two older sisters, Grandmaster Susan and International Master Sofia, were part of an educational experiment carried out by their father, László Polgár, in an attempt to prove that children could make exceptional achievements if trained in a specialist subject from a very early age. "Geniuses are made, not born," was László's thesis. He and his wife Klára educated their three daughters at home, with chess as the specialist subject.
I’ve seen him play at SF Jazz a few times and it seems like his loves his job.
How do you go about teaching your kids? I don't have kids yet, but if they do have in interest in programming, I'd love to give them a strong head start like you have.
Regarding teaching, I understood early that while I can program, I have very little understanding on how to teach it. So first, she underwent two Python courses (basic and advanced) on stepik.org [0], which had automated tests and have her an understanding of the language.
Then, I planned to start throwing tasks of increasing complexity at her until she solves them, giving her a more or less complete coverage of all things programming. So far, she has written such things as
- Tetris, in the console, with keyboard input
- Snake game, with competing snakes which use wave algorithm to find path
- LZW algorithm to learn bits and compression and other things
Next my plan for her follows roughly this path:
- some GUI programming, using GTK and Windows Forms. Calculator or maybe Checkers or something.
- some web programming, backend and frontend. Most likely Django / jQuery
- maybe some neural networks since are all so hot now
- some hardcore assembler stuff, like maybe some simple game or visual demo
- some mobile development, Android or iOS
Once she'll be done with all that, I think she'll have quite a clear picture of most general areas of development and will be free to pursue the area she liked the most.
[0]: https://stepik.org
Demoscene/sizecoding might be interesting, especially if she also likes artistic stuff.
Yes. Bravo to you and your daughter for your perseverance in spite of the looming "disruption" coming for all human knowledge workers.
Regarding Racket, it looks interesting. While I never did anything like it myself back in the day, it looks like it'll be an interesting task for my daughter further down the road.
1) sequence in which you gave her tutorials e.g. first tetris, then snake game, etc.
2) sources that you used to guide her through these tasks? e.g. youtube videos or other links.
Thanks.
Then more advanced Python course [1]
Then we went for Tetris. I did provide her help in getting the initial loop running where you press the key and something happens, then she implemented the game logic by herself. I did ask her to explain the architecture to me in broad strokes prior to development, and gave her some feedback, so the architecture was adjusted a bit.
First version was built in a very direct way, so once it was done I told her about Object Oriented Programming and she rewrote Tetris using objects in a more elegant way.
Then she moved over to Snake game, and mostly did it by herself, reusing parts of the code that were used in Tetris (main game loop thing, mostly), I mostly provided feedback and beta testing. We had some interesting moments debugging the wave algorithm which computer player snakes used to find a target (a simple wave algorithm), and improving performance. Then, we moved on to this LZW thing and, unfortunately, its creators immediately started dying.
The reasoning why I chose these particular tasks is because they are all relatively limited in scope, so a studying task is not enormous in size so you never finish it and actually have a chance to ship a finished product to show it to friends/teachers.
I followed this teaching path: scratch > python > sqlite > javascript (web). I am missing algorithms and OOP, and will look for gradual learning resources in these area.
One recent area I tried was observability. With prometheus and grafana, they got into collecting more data points and creating visualizations for them. Bangle.js is another one.
Maybe Qt instead?
Suggestion that because the Qt docs tend to be much, much better + it's realistically cross platform, thus opens up more possibilities.
For some reason, it seems the former USSR is highly prominent in compression algorithms. RAR, PAQ, encode.su, etc. and of course Andrey Markov.
I’d take a wild guess that it has to do with making more with having less, such as access to larger storage volumes or broadband.
Specific to compression, when you’re “tech poor”, the time you spend waiting for data to get processed is essentially free.
zstd is developed at Facebook, not known for a lack of computing resources.
And, which is quite odd, a lot of proprietary Windows software.
At that time many young programmers regarded compression, compiler writing and anti-virus writing as very important. Now people just learn React. :)
Take cars: cars have a reasonable upper limit in physics so low that the difference between 'mild' (100 kph on the highway) and insane (300 kph+, no matter where) is a mere factor of three. And it took a good 100 years before that barrier was broken. It won't progress much beyond the standard of 100 or so kph under normal circumstances. There are just too many things that work against you when you go much faster than that including energy consumption, braking distance, the willingness of your vehicle to stay on the ground, attention requirements (ok, AI... someday...) and so on. So even if it is possible in exceptional cases it likely won't be the norm, even on closed over and evacuated freeways.
Aircraft did a little better, but even there the supersonic passenger jet came and went. Though it may be back one day.
In comparison computers have gotten many orders of magnitude faster in a much shorter time. But there is a physical limit, and we are very likely to run into that soon. And once that happens the only way to progress is to make more efficient use of the resources you've already got.
Good points. And I will even argue that while we didn't hit a physical limit in hardware yet, it's still important to have a reasonable performance because not having performance means resources are wasted, money are wasted and it might lead to user frustration.
But it seems to not be the majority's opinion or at least not most of the business owners opinion.
I've lost days to solve a memory bug in one of the services and no one cared.
The original architect and programmers were gone, the engineering managers were pushing Clean, SOLID and design patterns hopping that will help and the domain was hard because we had to deal with real money while respecting tens of different laws and locales.
We tried at our best to do the impossible, we were understaffed by a factor of 5X.
And the services were crashing like crazy but it looked like a no downtime because we provisioned another POD in Kubernetes.
Provisioning was like: how many pods do we need? 3? Let's make it nine. How much memory do we need? Half a gig? Let's make it 4 gig to be sure.
So it was a mess and it worked and that mess continues to work somehow.
Not sure if mess driving engineering is a sound bussineses idea, though.
So as long as your supervision mechanism is bulletproof (supervision trees) then you can get away with this for very long.
We’re reaching the diminishing returns portion of the logistic curve for computing with regards to both things like hardware and compiling, compression, etc. It’s just hard to compare industries which are done growing with ones that still feel like they might be exponential forever.
Some industries never evolved too much. I don't see a big functional difference between cars a 100 years ago and cars now. And there were electric cars in 1900s.
A car is constraint by lots of things just like GP comment affirmed. Putting more gadgets in a car won't change what a car is or how it works at a fundamental level.
From the very first microprocessors to today's there is a big leap. If we apply that leap to cars they should be using nuclear fusion and take us to moon and back in seconds. And it's not like trillions of dollars weren't invested in car R&D in more than a century.
Software growth seems exponential only because of hardware and the Moore's Law. Which will come to an end soon.
Thanks to Moore's Law, we programmers, afforded to be careless, not think about performance or use unnecessary abstractions just out of esthetical or ideologic considerations. It allowed businesses to say that time to market trumps everything and customers should simply buy more powerful boxes or more boxes. Now this will come to an end soon and we will have to think about efficiency, speed and performance.
Or maybe chips will move from silicon to some alternative processes where growth is still possible.
And the diminishing returns for computing are for the most part driven by the lack of investment in new ways of computing because we got so much for free. But once the free ride is over you can expect the research into efficiency to be picked up again because that will be the only field where real progress can be made. There have been a few half-assed (apologies to those involved, it was no doubt a ton of work) attempts at making computing fabrics and clockless machinery. And that's because by the time you have something working the scaling advantages have already overtaken your work and absolutely nobody is going to give up a working architecture for something unproven if it doesn't given an immediate return. Some pretty good ideas died like that.
But I suspect their day will come.
Yes, for digital computing. But computing is on the cusp of becoming more non-determistic, which will allow analog computing to absolutely demolish digital computing in performance.
They still do regard it as important, just in a different sense. Shannon formally proved that compression is congruent to statistical inference — an efficient compression algorithm seeks to minimize the statistical entropy of the scheme used to encode a message. Compression is thus loosely congruent to a branch of statistical inference you may have heard a thing or two about called "artificial intelligence" [0], which many young programmers are very much engaged with.
Take the Hutter Prize [1], which awards money for the ability to compress the entire English Wikipedia, or the Large Text Compression Benchmark [2], the latter of which's current top performer is a transformer-based neural network [3]. As the LTCP's rationale says [0],
>Given a probability distribution P over strings s, the Shannon capacity theorem states that the optimal (shortest average) code for s has length log 1/P(s) bits. If the distribution P is known, then such codes are in fact easy to find and compute. The problem is that for natural language, we do not know how to explicitly compute the model P(s) for a given string s.
>However humans must implicitly know P(s) because our speech and writing, by definition, follows this distribution. Now suppose that a compute can compute P(s) for all s. Then for any question q (or dialog ending in a question) and any answer a, the machine can compute P(a|q) = P(q,a)/P(q), and use this distribution to answer any question q posed by the judge. By definition, this distribution would be identical to the distribution of answers given by the human, and the two would be indistinguishable.
[0] http://www.mattmahoney.net/dc/rationale.html
Even for realizing what you've said one needs a bit of theory, basic computer science and some math. And most people today have a "do" philosophy and are result oriented. They are not learn oriented, they hate CS and math and they hate low level programming.
I don't think compression is highly relevant today, but at the time I started learning it was a good tool to sharpen one's mind. Learn algorithms, learn about memory management, learn about hardware, learn low level constructs, learn a bit of math.
Whenever a young person wanting to undertake a programmer career is asking me for advice I'll ask what their motivation is. If they just want money I advice them to learn the tools in fashion today, maybe Javascript and React, maybe Python, maybe C#. But if they respond they want to do it for both enjoyment and money, I put them on the hard path. I ask they enroll an BS CS program, or if that is not possible, give them long lists of books, tutorials, courses which cover CS basics, math used in CS basics, hardware, low level programming and the programming hot topics du jour. In that last case I also design their learning paths to accommodate their goals and I am always taking time to answer them questions in the future if I can, if not I will at least indicate where should they look or which is the right person to ask.
Well, maybe no one sensible? I still think it's quite cool that you can take any general purpose compression algorithm, and abuse it to do e.g. image classification. (Just append the image you want to classify to each of the classes' sets in turn, and see which compresses best!)
And actually I do remember a paper that tried to use ideas from PAQ to do online learning. Gated Linear Networks, out of DeepMind, in 2019:
https://arxiv.org/abs/1910.01526
All compression is related to AI, but especially sample efficient online learning is basically what data compression is all about.
I'd think that calculus or functional theory or category theory can find more bijections towards statistics or even congruences than mere arithmetics or algebra ever will. Ok, you can explain or derive any mathematics construct using only algebra, and there were efforts to do so, but does it makes sense?
https://web.archive.org/web/20181123231027/http://backup.its...
In Hebrew (or rather Aramaeic) we say:
חבל על דאבדין ולא משתכחין
A pity due to those who absent and not present.
Jacob *Z*iv (Hebrew: 27 November 1931 – 25 March 2023)
was an Israeli electrical engineer who, along with Abraham
*L*empel, developed the *LZ* family of lossless data compression
algorithms.With better compression, heaven will soon have more space available.
I started a project to implement a LZ77 decoder in Game Boy assembly to see if it could compress the sprites in Pokémon Red and Blue versions better than the algorithms actually used in the game. Results are inconclusive so far, but it's been an enlightening experience.
I think, though, that most instruction texts do the reader a disservice by starting with LZW (and thus having to deal with the kwkwk case from the get go). They should start with LZ78, which is much more simple and elegant; then show how -- although unimportant asymptotically -- we can save some bits by using Welch's trick (W in LZW), and show how that gives rise to the special case which needs to be handled.
Somewhat unexpectedly (for me anyway), if you write the decompressor as a memory stream decompressor rather than as a dictionary decompressor -- that is, instead of putting <w>,k pairs into your decoder dictionary while decoding, you note the output <offset> of where that <w> first appeared - there is no special case, and the implementation goes back to being LZ78 elegant and uniform.
This is worthy of an HN black bar.