167 karma · joined July 12, 2014
Big fan of Systems, Unix Philosophy, Vim, Emacs, Lisp, and Functional Programming.
Follower of the Bhagavad Gita and Vishishta Advaita Vedanta.
www.chandergovind.org mail@chandergovind.org
The point of Fine-tuning small models is for specific downstream tasks, which SOTA models can do, but at higher costs. It is purely an economic play, not an attempt at pushing boundaries of SOTA.
I want more blogs/discussion from the community about the existing tools.
In 3/6 months, how many skills have you written? How many times have you used each skill? Did you have edit skills later due to unseen corner cases, or did they generalize? Are skills being used predominantly at the individual level, or are entire teams/orgs able to use a skill as is? What are the usecases where skills are not good at? What are the shortcomings?
(You being the metaphorical HN reader here of course.)
HN has always been a place of greater technical depth than other internet sites and I would like to see more of this sort of thing on the front page along with day one calls.
Would strongly encourage you to open-source/write blog posts on some concrete examples from your experience to bridge this gap.
I would encourage you to write up a blog post of your experience and share a version of the skill you have built. And then follow up with a blog post after 3 months with analysis like how well the skill generalized for your daily use, whether you had to make some changes, what didn't work etc. This is the sort of content we need more of here.
3 months ago, Anthropic and Simon claimed that Skills were the next big thing and going to completely change the game. So far, from my exploration, I don't see any good examples out there, nor is a there a big growing/active community of users.
Today, we are talking about Cowork. My prediction is that 3 months from now, there will be yet another new Anthropic positioning, followed up with a detailed blog from Simon, followed by HN discussing possibilities. Rinse and Repeat.
This is something I have experienced first hand participating in the Vim/Emacs/Ricing communities. The newbie spends hours installing and tuning workflows with the mental justification of long-term savings, only to throw it all away in a few weeks when they see a new, shinier thing. I have been there and done that. For many, many years.
The mature user configures and installs 1 or 2 shiny new things, possibly spending several hours even. Then he goes back to work. 6 months later, he reviews his workflow and decides what has worked well, what hasn't and looks for the new shiny things in the market. Because, you need to use your tools in anger, in the ups and downs, to truly evaluate them in various real scenarios. Scenarios that won't show up until serious use.
My point is that Anthropic is incentivized in continuously moving goalposts. Simon is incentivized in writing new blogs every other day. But none of that is healthy for you and me.
I love org for all its bells and whistles and use them in various ways. But most of the time I need a small subset of org in a form-factor that allows ease of use.
I had been using dwm (4000 lines of C) for many years and wished to write my own for a long time, but what made me take the leap was really steveWM [2] and TinyWM [3] which are both super small.
[1] - https://github.com/ChanderG/cellwm [2] - https://github.com/stodd1031/steveWM [3] - https://github.com/mackstann/tinywm
When I got into BQN recently, I had the same thought and tried my hand at recreating LLM building blocks and saw the compactness APL-likes afford first-hand: https://www.chandergovind.org/blog/5126/aipasi/Array-program...
1. Software: this is all Pytorch/HF, so completely open-source. This is total parity between what corporates have and what the public has.
2. Model weights: Meta and a few other orgs release open models - as opposed to OpenAI's closed models. So, ok, we have something to work with.
3. Data: to actually do anything useful you need tons of data. This is beyond the reach of the ordinary man, setting aside the legality issues.
4. Hardware: GPUs, which are extremely expensive. Not just that, even if you have the top dollars, you have to go stand in a queue and wait for O(months), since mega-corporates have gotten there before you.
For Inference, you need 1,2 and 4. For training (or fine-tuning), you need all of these. With newer and larger models like the latest Llama, 4 is truly beyond the reach of ordinary entities.
This is NOTHING like open-source, where a random guy can edit/recompile/deploy software on a commodity computer. Wrt LLMs, Data/Hardware are in the equation, the playing field is complete stacked. This thread has a bunch of people discussing nuances of 1 and 2, but this bike-shedding only hides the basic point: Control of LLMs are for mega-corps, not for individuals.
Stories/memes/narratives are the most easy/potent form of mental ingestion - so much so that I think humans cannot ingest facts or ideas at all, only stories. And this puts a collective responsibility on all of us to be very careful about the stories we create.
Even a simple para oriented fuzzy search might be a good place to start.
What I got from it is that, the process is less about storing/retrieving information, but more about building a system that can surprise you - the way ideas emerge when you brainstorm with others. These others being people with the same level of knowledge as you - so, not the kind of information flow that happens between an expert and a beginner, but between peers who have access to the same kind of information, but simply look at things differently.
For some reason, every Zettelkasten system out there focuses on the mechanics, and in my opinion, not enabling the original intent of the approach. Not that I have a solution to the problem.
DuckDuckGo gets this article as the first hit - top 4-5 links are all relevant.
So basically, your develop the solution bottom up. You can pretty much write an elisp function anywhere in emacs (in the very file you are editing, in the already open scratch buffer, in an elisp popup shell) and test the function incrementally. Also, the extensive in-built help system lets you very quickly lookup functions that you need.
Once you have something that solves your current problem, you can save it, if you want. Over time your custom collection of functions improves and then you decide if something there needs to be generalized or made into a package.
Obviously this needs some basic elisp knowledge and then just practice over time.
I was happy with the git cli (and occasional tig) when I started using magit full time and initially never understood all the buzz. It did not seem that powerful over what I was using.
After 3-6 months of use, for some reason I had to go back to the cli and then I realized the ease and speed I had got accustomed to.
Magit is magical; use it for some time and try going back to your previous setup.