Command line functions around OpenAI
kadekillary.work
kadekillary.work
Hyperproductivity in software is all about deciding what problems to tackle. Richard Hipp isn't worried about chipotle restaurant orders in golang, he's worried about how to store data reliably. That isn't a coding puzzle that ChatGTP is likely to help with. Either ChatGTP can do the whole thing itself (not yet the case) or it is a minor productivity boost because the hard part is articulating the problem.
Writing code quickly really isn't a challenge that high performing software engineers need to tackle. ChatGTP is a cool tool, we're all going to be using things like it in a few years, it'll change everything. But it won't make any old engineer a rival to the big names in software engineering.
In relative terms, the unaided 10x engineer is now only a 2x engineer as his peers are now vastly more productive.
That's huge.
Knowing when and how to use dependency injection, making code testable, what to instrument with logs and stats, how and when to build extensible interfaces, avoiding common pitfalls like race conditions, bad time handling, when to add support for retries, speculative execution, etc. are part of parcel of what we do.
If ChatGPT can help raise the quality of work while also increasing its quantity, that'll be a huge leap in productivity.
It's not all there yet, but I've been using it to write some simple programs that I can hand out to ops / business people to help automate or validate some of their tasks to ensure the smooth rollout of a feature I've developed.
The resulting ChatGPT code--I chose Go because I can hand over a big fat self contained binary--avoids certain subtle pitfalls.
For example, the Go HTTP client requires that you call Close() on a response body, but only if there's no error.
The code it spat out does indeed do that correctly. And it's well documented.
It's far from perfect; I've seen a subtle mistake or two in my playing around with it.
I'm now no longer so much an author but an editor for basic stuff. I now have my own dedicated engineer with a quality level that oscillates wildly between fresh intern and grizzled veteran.
It'll only get better over time.
As an example, if you took away the top half of the engineers on my current team and gave the rest state of the art Copilot (or equivalent best in breed) from 2030, you would not end up with equivalent productivity. What would happen is they would paint themselves into a corner of technical debt faster until they couldn’t write any more code (assisted or unassisted) without creating more bugs than they were solving with each change.
That doesn’t mean the improved tooling isn’t important, but it’s more akin to being a touch typist, or using a modern debugger. It improves your workflow but it doesn’t make you a better software engineer.
And all this with documentation explaining what it did, and optimising the code for human readability, so that even with huge reworks you can still get the gist in a time measured in hours rather than it taking, what, weeks to do manually?
... but believing it is a theorem would be similar to believing that horoscopes can predict the future as well.
Maybe some day we'll have a model that can be trained to reason as humans do and can do mathematics on its own... we've been talking about that possibility for decades in the automated theorem proving space. However it seems that this is a tough nut to crack.
Training LLMs already takes quite a lot of compute resources and energy. Maybe we will have to wait until we have fusion energy and can afford to cool entire data centers dedicated to training these reasoning models as new theorems are postulated and proofs added.
... or we could simply do it ourselves. The energy inputs for humans compared to output is pretty good and affordable.
However having an LLM that also has facilities to interact with an automated theorem proving system would be a handy tool indeed. There are plenty of times in formal proofs where we want to elide the proof of a theorem we want to use because it's obvious and proving it would be tedious and not make the proof you're writing any more elegant; a future reasoning model that could understand the proof goals and use tactics to solve would be a nice tool indeed.
However I think we're still a long way from that. No reason to get hyped about it.
Still a good post and reply.
I have been skeptical about all this, but decided to play with using ChatGPT for coding a task I had. It wasn't perfect, either in workflow or in final results but what was clear was that not adopting it will very quickly make you lag behind your peers.
I think the additive bonus is especially high for senior devs who are fast, in that those will be faster in grokking the generated code and better at guiding it to the right solution.
I guarantee you that in five years anybody who cares at all about their productivity will be using it constantly in their workflow, probably much sooner.
A builder bot could then ask a series of questions and get permissions such as “can we use X package?”, and then update the specification before performing the build.
I know I could have used google but the way google presented me is not easy to find and digest, you have ads, you have article that is not exactly what you are looking for and the code examples are not exact. Gpt eliminates all that time and effort. It almost eliminates laziness
* Understanding the value proposition as a human, often as through a conversation; often with hidden criteria that you'll have to uncover
* Making sure the code respects the value proposition. As reading other people's code takes more time than writing it, ChatGPT won't help that much here.
----
One development to this story I'm anticipating for though is connecting ChatGPT to formal proof systems, and making sure it has sufficient coverage with 'internal thoughts' that are logic-verified (by opposition to killing the problem big-data-style). Sort of like the internal monologue that we have sometimes.
I say short and medium term we're sort of ok; but I don't know about 10 years from now.
Your two points may not be as sound as you think. The second one can be defeated now since ChatGPT is quite good at generating descriptions of what code is doing. The first point is more subtle, you can iterate on code with ChatGPT but right now you need to know how to run the code to get the results before providing feedback. Once tools are developed for ChatGPT to do that itself then it could just show the results and a non technical person could ask for follow up changes.
Being efficient in the perspective of writing new automation so you don’t have to do the same manual things again can easily make someone a 10x (or greater) engineer. This typically means writing your own tools and not hoping some third party package or framework will do it for you.
Conversely the same is true. If a 10x engineer is in an environment where they are prohibited from writing original tools they will just be average or much worse when their motivation is destroyed.
Ouch
The problem with this is that neither is good for your mental health. Unless you're joining that project with a specific goal of improving its development process, don't bother. Run away. Fast.
Especially if there are already other devs that are militantly against your take on dev process improvements. The level of emotional distress is hard to fully appreciate without experiencing a situation like this. Suggesting the simplest and most obvious things becomes a brutal fight that you're guaranteed to lose. Living without them makes you miserable in its own right. You're trapped between a rock and a hard place with no way out.
If out of various kinds of psychotherapy you pick the one based on an experience of living in a Nazi concentration camp as the most applicable - it's time to run. Really. Don't look back.
Also, some people just keep growing their creative outlook. They are not afraid to try solving bigger & bigger problems.
I worked with an engineer (individual contributor) at a company with a lot of related software products.
Time and again, he would hibernate then come out with a new well developed tool or product that either impacted the entire product line or started a new product line.
Once the value of what he had done was obvious, and his clever design had stabilized in a form understandable to others, he would hand it off to a team of great developers to polish, release, update etc.
10x impact would be an understatement.
If we measure impact, it is easier to see where 1x, 5x, 10x, or 100x can be achievable and verifiable.
Also x/2, x/10 or (and I have seen it) -x. One team leader, -5x at least, until they were removed.
But as a rule, it would not have been challenging enough to keep them engaged for long. And it would have been a waste of their talents for themselves and the organization.
It wouldn't just have wasted their time directly, but wasted the opportunity for less (but still A-grade) developers from diving into a new area and maximizing its value. There was still lots of creative work to be done. These were not trivial new areas.
The problem is many, many people are not in a position to advance that way. I've worked in startups where you're almost always given an ambiguous problem and rewarded for meaningful outcomes.
Many people are working as part of team/projects where the general problem and approach are already determined for them.
This stood out, so I looked it up, and it seems Richard Hipp is 61 years old. I guess it is more about the "any" than the "old". Or maybe old engineers are engineers doing it the "old way" (pre-chatgpt)?
Fabrice Bellard is utilizing transformer models for lossless compression, for example: https://bellard.org/nncp/
What a ridiculous dichotomy, or course there's plenty of room in between. The hard part might be articulating the problem, but the other part takes time. Make that part not take time and you can do more of the hard part.
This isn’t very conducive to corporate work though. It’s about timelines and features promised to customers.
I find I’m 10x more productive in the long term on projects I’m a sole developer on or when working with a small knit team. And I don’t measure that in lines of code or number of features.
I wonder if ChatGPT can help there.
Obviously not, it will just make expected windows shorter and shorter.
"This would take a C developer a month, but you use Java so 2 weeks should suffice"
"This would take a Java developer two weeks, but you use Python so a week should suffice"
"This would take a Python developer a week, but you use ChatGPT so I'll expect it sometime tomorrow."
The faster I've gotten at moving code around, the more subconsciously willing I've been to try something with a modest probability of working out, and more importantly the easier it's gotten to throw away a draft that isn't working out and try again.
LLMs haven't made it into my coding workflow yet, but I can see how they could be useful and the trend seems to be that they'll be in most high-octane workflows at some point.
Maybe that makes everyone hit that point easier, maybe it brings that point in for you. Not sure.
Smartness isn't wisdom.
AIs are, at most, producers of synthetic smartness.
of course a fast typer does necessarily make a good programmer
Typing will get you places faster, but some of the most productive tools we have don’t rely on it at all.
Slinging code is skills & tactics at best. It can only have a small multiplier.
All of it will reduce mental load for senior devs and enable them to tackle more problems or more difficult problems.
Not really though, at least not yet. I've seen some impressive results for trivial cases: the type of thing you'd see blogged on new web devs blog 10x over, but I've yet to see anything that wasn't trivial and worked come out of GPT.
In a winner-take-all economy, usually the winner is the person with the ability to do something the slowest.
Not always, but usually.
So, for example, someone who can productively spend 100 hours working on a blog post is probably going to get win out against someone who can only productively spend 5 hours working a blog post on the same topic.
And given a society where 99% of the rewards go to 1st place, 1% of the rewards go to 2nd place, and 0% of the rewards go to everyone else, having this ability is usually more valuable than the ability to pump out a little more work in some fixed amount of time.
Based on my experience, you’d be surprised how hard it is sometimes. I have a team of 10 engineers that failed to ship a basic CRUD UI after several months (and it wasn’t even close to done). Eventually it got taken over by one individual eng who finished it in two weeks.
Over my career, the higher up the pay grade I went, the less actual code I was writing and the more it was figuring out what to do, what's priority, how the existing code all fits together, evaluating risk, and how to make the right surgical incisions to accomplish the goals.
Glorious and quite rare is the moment when a problem is clearly written up by a PM and the codebase is greenfield or clear enough that just pumping out code is happening and taking up my day.
Most senior software engineers are less programmer and more analyst.
[1] https://en.wikipedia.org/wiki/Generative_pre-trained_transfo...
As you say, I find the act of writing code is hardly the bottleneck to my productivity. One of the best case scenarios, LLMs become the ultimate high level language. In that case, Software Engineer roles would switch from coding to more of a mix of old school Software Architect, PM and entrepreneur.
By the way, I find interesting to run the argument of "coding is not the bottleneck" in reverse. What if you design an organization so that coding is the bottleneck? How transformative would LLMs be?
When people claim ChatGPT can replace software engineers because it can sort of code is always puzzling to me. 20 years into this career, and writing code is only a minor part of my day-to-day activities.
A part I do enjoy doing, but still minor part nonetheless.
> What if you design an organization so that coding is the bottleneck? How transformative would LLMs be?
The funny thing is that although ChatGPT has been an amazing coding assistant tool, it can't meaningfully write code beyond boilerplate to me. I use it more as search engine on steroids, rather than actually relying on the code it writes.
A few times it even got into a hallucination loop going through different flavors of uncompilable code when I asked "how to do X using library Y", and refining the question was getting nowhere.
ChatGPT is an amazing tool that provides me value beyond the 20 bucks per month I pay. But still has far too many limitations. It's sort of funny to see people LARPing that it will kill knowledge jobs in the near term.
Though I don't expect software dev job to go away. It'll just change in nature. Less typey typey, more managing the LLM.
There's also not orders of magnitude more existing data we can feed it anymore.
Not as a language model. Other leaps in technology would need to happen before it can code.
Design Patterns are said to compensate for features missing in a language (e.g. Visitor for multiple dispatch). ChatGPT is sounding similar - solving accidental complexity that "shouldn't" be there in the first place.
Stuff like
match network_foo_int {
TYPE_A => Foo::A(read_a()),
TYPE_B => Foo::B(read_b()),
...
The problem isn't generating code. It's articulating a functional specification with enough precision and clarity that you can solve for the unknown, which is the program that satisfies the specification.
I agree with the conclusion that tools like ChatGPT won't be replacing programmers any time soon. And it won't be helping us to write programs until it can properly reason about the information it is trained on and prompted with. The problem with natural language is that it is often too imprecise. When you get to concurrency or optimizing data layout an LLM isn't going to understand anything about the problem space and the training data probably won't contain enough examples to produce something remotely plausible let alone good or elegant.
Writing, thinking, and asking the right questions will still be important for quite some time. We're not in the age of having a magical genie in our computers that can do all the work for us... yet.
Although using it as a procedural generation tool for fake data is nice... still not as nice as writing a property test with generators and shrink... but as a little time saver for quick prototyping, neat.
- Knowing or having control of the tools you use
- Knowing what problems are most important to revenue and which are the greatest risk
- Knowing the language well enough to plan ahead without much labor
When a company I worked at mandated an IDE it took me a while to be productive beyond typing speed. When I discovered my company had support for generators, I was no longer spending forever on boilerplate REST code. When I know my test harnesses well it's easier for me to prove functionality in a smaller feedback loop.
- Explanation/research ("how does this work?")
- Code analysis ("tell me if you think you see any bugs, refactoring suggestions, etc in this sprawling legacy codebase")
Things that feed into the developer's thought process instead of crudely trying to execute on what it wants
You are correct that I made an error in my previous response.
I apologize for the confusion I may have caused in my previous response
I appreciate you bringing this to my attention.
I apologize, thank you for your attention to detail!
Incredible technology, though. Feels like whoever decides not to use it will be at huge disadvantage real soon
I mentioned that an approach would be a lot of work. His response:
“It’s just typing. Typey typey type (makes fingers on keyboard gesture).” Absolute lightbulb moment for me.
To become a senior engineer is to realize that the code writing part is inconsequential as long as there are no technical unknowns in the work.
It’s freed me to try out huge sweeping changes just for fun and generally detach from my code. Sometimes I delete large, intricate changes so quickly and ruthlessly that my colleagues have to ask me to reopen them.
We aren’t paid to write working code. We are paid to grow harmonious systems. Working code is table stakes.
An org of a few hundred people at a big tech company many only have a couple of Individual Contributors making this much money.
I know it's unbelievable for those not in one of these companies, but this data is accurate: https://www.levels.fyi
but they are extremely rare
I make over $1m as a SWE. “Senior Staff” or “Level 7” role at a big tech company everyone knows.
I mostly write code! I am way faster than most others here, and I have encouraged and mentored the people on my team to be similarly efficient.
I also help decide technical direction but it’s far from the “architecture astronaut” stereotype that writes no code.
It’s an infra team so no significant influence on product direction, other than considering our frameworks and tools products. I do consider them in that way, but it’s still a big difference from being involved in the direction of the external product.
What were the main changes you introduced/points you mentored them on to get their productivity to also be similarly efficient?
I see people on other teams that seem to be stuck for days sometimes, on a really basic error that they haven’t really tried to debug at all. Seriously, just read the error message or dig into the code!
My team also has a strong culture of just pushing code changes. This is including tons of deletions and simplifications, it’s not just writing a million lines of code.
It’s also a very senior team, so I can’t take credit for everyone’s growth from junior engineers. Another part of it is creating an environment that productive senior coders want to join, so that they can avoid process-heavy teams and get stuff done.
There is an extreme level of trust on the team because everyone is senior and frankly extremely good at their job, which allows us to be even more productive.
Timing gets a 10/10 for my $750k year at staff level though.
With all due respect, you may have learned the wrong lesson — or missed some important context — and the above is a false dichotomy. Working code isn't just table stakes; it is virtually impossible to "grow harmonious systems" if you don't have solid working code, because without solid working code you cannot grow the system reliably.
The other thing to note is that just because someone makes a lot of money doesn't necessarily mean that they are good at their job, or that they are a good team player, or that the way they operate and the context they operate in applies to you. I know consultants who pull in a million or more every year, and you know how they do it? They write absolute shit software that, while fulfilling the contract requirements, ends up hamstringing the client down the line. Sometimes prioritizing short-term gains might be acceptable or even necessary (e.g. the auditor asks you to compile data from a bunch of sources and you need to do it fast), but for projects with longer life spans one should definitely not blindly "typey typey type".
I’d go further and say that in big tech, well factored well tested code is table stakes and the ability to produce it quickly is assumed. Junior engineers have to prove that to get their first promotion.
If that’s all they show, it will be their last promotion, and that’s perfectly fine too.
I grew up poor though and teaching myself coding moved be to earn more then I thought I'd ever earn without a college degree.
I knew about for loops almost a decade before I decided web dev would be my career, not just my hobby.
it's insane that you could get hired and not know how they work.
linked lists, sure. I still don't fully understand those lol, I was going for an associate's and that was in my c++ class but I've mostly only touched PHP, python, ruby, JavaScript... playing around with go and rust some as well, but I've never had to really think about linked lists, loops on the other hand is a daily thing.
hell, not knowing loops is like not knowing how to increment, like seriously, his to add 1 to an int.
Smh....
Even not-so-great-sounding contracts pay 50-65/hr, might be a confidence boost. Level up your skills on udemy, if necessary (wait for courses to be 80% off or whatever, happens regularly).
Could also go corporate at a non-tech for low stress and a decent salary.
I'm super obsessed lately w/ AI, on youtube, github, HN, reddit, etc... and working on actually building a clone of myself to do the mundane parts I don't like to do... hehe... this is kind of a green field, and the opportunities are huge, might be the last dev jobs when AGI hits -- training custom AGI's...
I've been watching the AI hype (again) and it's completely going over my head, I really really don't get it - I doubt {x}GPT would be able to analyze hundreds of thousands of project code across a big system (multiple languages, multiple services, interconnectedness of everything) and tell me what to do, even in the case I would want to paste in everything proprietary to a 3rd party service which I really don't.
Maybe it just works for a subset of programming, but then again I don't really see how it differs to reading docs for generating boilerplate or whatever it's useful for in greenfield projects. I was also not really impressed by it generating improvements to code when you ask it the right questions since that's the hard part of the job, not typing out the code.
Maybe I'm just a dinosaur and dumb.
Currently, the Language Models (GPT or GithubCopilot) are mostly good as ... copilots. You bounce off ideas off them or use them to kickstart something you want to do. It's great for junior devs (me!) but it still cannot do big context or cognition (what you describe), that part will be done by humans for quite a while.
Telling any AI system what to do will be the biggest problem. A lot of toy like algorithm problems can be easily stated. Stuff like inverting binary trees, sorting, and your myriad different Leetcode stuff is the easiest thing to do here.
Real world software has if/else statements peppered all over large code bases/domain logic. Lots of IO exceptions, lots of domain specific logic, all of this is nearly impossible to dictate in words. And even more you will eventually need some concrete way of describing things, which in all honesty will look a lot similar to code itself if you have to get it right.
Modifying code will likely be even more harder to describe in talk than in text.
That said, the real work is insight away from the keyboard.
I think your "what problems to tackle" is overlapping management or business strategy - though yes, it absolutely is the greatest multiplier.
(yes, I've read them all because I can't believe almost everyone missed the point)
The point of the article is actually neat: a small shell (Fish) script to send query to ChatGPT or DALL-E and get response back in the terminal. As a terminal user myself, this is really nice! (I rewrote it for myself in bash which I use, but used the curl command virtually as is)
The post itself is hilariously over the top (1000x c'mon), quoting 50 Cent and including statements like:
> Unfortunately, due to inflation — real and imagined, 10X just won’t cut it anymore. We need bigger gains, bigger wins, more code, more PRs, more lines, less linting, etc...
Either people really need to be pointed out the sarcasm these days, or everyone's just philosophising about the title (I'm willing to bet it's the latter).
It's so easy to bait with a #x in a title, and especially when defining in terms of engineer (also an elusive term at times).
prompt="$*"
Since I can use this with multiline input I didn't create a separate data-gpt one.
(Note I also used gpt-3.5-turbo model and not gpt-4 because the former is faster and cheaper and for quick one-offs from cmdline it's usually enough for me).
"Please use the original title, unless it is misleading or linkbait; don't editorialize."
If anyone suggests a better title (i.e. accurate and neutral, preferably using representative language from the article), we can change it.
is in the first paragraph. I think that's too long for HN though, so maybe:
> how to catapult your productivity via command line functions around OpenAI.
Headline: Here’s a neutral headline for the article: “Improving Programmer Productivity with Command Line Wrapper Functions around OpenAI API” .
Summary: The article at https://kadekillary.work/posts/1000x-eng/ discusses how to improve programmer productivity using command line wrapper functions around the OpenAI API. The author provides examples of functions for generating text and images, and for making code edits. The article is written by Kade Killary and was published on their website on March 29th, 2023
That says almost everything there is to say about the article in 1 comment; and even you've sent most of your time discussing the title and meta-issues.
People aren't missing the point, it just happens that programmer productivity is more interesting (hence the link bait title, I suspect). The article stands on its own without comment.
It's a pity that our interviews are all biased towards the "how"
I had an intern a few years ago -- recent college grad in CompSci. I tried my best to lightly mentor him. One day I was talking about the diff between a compiled language and scripting, mentioned REPL. To demonstrate REPL, I opened up both the windows CMD prompt and the Chrome Developer tools. I mentioned that with a REPL like the Chrome Tools, it's trivial to do FizzBuzz in JavaScript. I explained the problem to him and asked him to take a stab at writing it. This wasn't an interview question, just a discussion and a mentoring opportunity.
He couldn't. What he said next blew me away, coming from a CompSci graduate "Oh, loops, yeah, I never quite understood those. Like, for loops and while loops - I never really got that". I asked if he meant recursion, cause that can be tricky. No - he really could not write a for loop in any language. I wasn't going to shame him and I walked him through it, but I was disappoint. ಠ_ಠ
[Edit: Spelling]
Edit 2...Before I start claiming that CompSci programs are letting students down, I have to consider that the claim that he had a BS in CompSci may have not been accurate. I did not check or verify his transcript. The more I reflect on it, I think he may have had a degree is Web Design and we got pressured to add him to our Software Eng. team because the hiring manager (and his actual mentor outside of work) passed him off as a "Web Developer". Now that I think about it... that seems more likely....Edit 3: It was driving me crazy so I dug up the resume in my inbox ... it was def CompSci, listing C++, C#, Java, and SQL as technologies and data structures and algorithms as courses taken... I'm not sure what to think...
How did this person pass a technical interview?
I’m confused. Are you using “pass” as in “pass on the candidate”(fail) instead of “pass” as it is used in the context of an exam?
The lie is that anyone can become a 10x [insert thing here] just by trying harder. Self-improvement is a real thing, but everyone can’t be 10x or 6-sigma or world class.
Strive, reach, sure… but ultimately, just make sure you live your life. You only get one, and the clock’s ticking.
Putting an actual individual factor on it and especially on engineer productivity, like we were at a line factory producing everything in a repeatable and measurable way is more questionable. 1x, 10x, 1000x? People here are discussing 10x becoming 2x, it sounds very mathematical. Now if there is an actual scale with an industry baseline and an individual measurement, I'm more than interested to learn about it and I think employers should fairly compensate engineer based on the scale factor you would put on your CV.
I often think of the Gene Hackman line in Heist: "I try to imagine someone a little smarter than myself, and I try to imagine what he would do."
It's not a level playing field, there are people who are just way more efficient and effective than the norm. Whether it's drive, focus, creativity, or knowledge (usually a combination of them), there are people who solve more problems and solve them better than average.
I don't know the literal number, but some folks are just better at it than others.
Perhaps you are just in a bubble and haven't met one?
I was at a conference once talking to a recruiter, when he stopped mid sentence and literally ran after one of these 10x engineers that had walked past.
The Net Negative Producing Programmer - G. Gordon Schulmeyer, CDP ( https://web.archive.org/web/20160305234708/http://pyxisinc.c... )
> We've known since the early sixties, but have never come to grips with the implications that there are net negative producing programmers (NNPPs) on almost all projects, who insert enough spoilage to exceed the value of their production. So, it is important to make the bold statement: Taking a poor performer off the team can often be more productive than adding a good one. [6, p. 208] Although important, it is difficult to deal with the NNPP. Most development managers do not handle negative aspects of their programming staff well. This paper discusses how to recognize NNPPs, and remedial actions necessary for project success.
> Researchers have found between a low of 5 to 1 to a high of 100 to 1 ratios in programmer performance. This means that programmers at the same level, with similar backgrounds and comparable salaries, might take 1 to 100 weeks to complete the same tasks. [21, p. 8]
> The ratio of programmer performance that repeatedly appeared in the studies investigated by Bill Curtis in the July/August 1990 issue of American Programmer was 22 to 1. This was both for source lines of code produced and for debugging times - which includes both defect detection rate and defect removal efficiency. [5, pp. 4 - 6] The NNPP also produces a higher instance of defects in the work product. Figure 1 shows the consequences of the NNPPs.
[5] : Curtis, Bill, "Managing the Real Leverage in Software Productivity and Quality", American Programmer July/August 1990
[6] : DeMarco, Tom Controlling Software Projects: Management, Measurement & Estimation (New York: Yourdon Press, 1982)
[21] : Shneiderman, Ben Software Psychology: Human Factors in Computer and Information Systems (Cambridge, MA: Winthrop, 1980)
John Carmack is definitely 10x or 100x (or 1000x) compared to me on low-level graphics programming, just factually, not that it says much about either of us.
But the '10x programmer' idea is soaking up and refueling some fanaticism. Let's all double down on the concept of high performance, for no particular reason we can agree on.
https://dariusforoux.com/prices-law/
> Price’s law says that 50% of the work is done by the square root of the total number of people who participate in the work.
Price's square root law: Empirical validity and relation to Lotka's law https://www.sciencedirect.com/science/article/abs/pii/030645... (behind various paywalls)
If you're on a team of 10 people, 3 of those people will be doing about half the work. And then 7 are doing the other half. If there are 10 work units, the 3 are doing 5/3 of a work unit each and the 7 are doing 5/7 of a work unit each for a ratio of 2.3x
---
Yes, 1/10th developers are common - especially in larger teams. And there are also -1/2 developers too where someone will spend half their time just cleaning up after the -1/2 developer.
And just with the nature of teams, larger teams are going to have a significant fraction of the team doing very little work (note: this applies to consultancy teams too and if you get a team of 20 people to work on the project, you really should just get a team of 4 and stay on top of them as it will be less work to manage).
The idea itself has a certain sultry allure. That probably helped keep it wedged in the back of the mind long enough to find those kernels, rather than being immediately forgotten.
> Programming managers have long recognized wide productivity variations between good programmers and poor ones. But the actual measured magnitudes have astounded all of us. In one of their studies, Sackman, Erikson, and Grant were measuring perfor- mances of a group of experienced programmers. Within just this group the ratios between best and worst performances averaged about 10:1 on productivity measurements and an amazing 5:1 on program speed and space measurements! In short the $20,000/year programmer may well be 10 times as productive as the $10,000/year one.
He goes on to explain that because the primary cost in software development is the overhead of coordinating minds, a company is far better off paying for just a few really good programmers than they are 10 times that number of mediocre ones. Whether or not there's such a thing as a 10x programmer in isolation, this seems pretty self-evident and valuable to keep in mind.
If that’s really the primary cost, surely one of the metrics they studied variation in is the coordination cost imposed by the programmer on the company, they didn’t just naively assume the cost was the same but output different, right?
Because it seems like if it does vary, you’d want to minimize it by hiring the easiest to coordinate, and that might offset output differences in sone cases.
After about 6 months on his team it became apparent that he was very good at communicating to his management, very good at writing excess complexity into his code, and very good at preventing the developers he was leading from taking on large or impactful projects.
My takeaway was that he was more of a "turn-everyone-else-into-1/10X" developer. I jumped onto a different team as soon as the opportunity came up (as did most of the folks who worked with him).
[1]https://twitter.com/skirani/status/1149302828420067328?s=46&...
Note that you'll also need to be from a Western country, or at least have access to both a phone number from one and a VPN. And if you want a plus subscription, you'll also need a US credit card - not stated anywhere but I've been totally unable to sign up for plus with either Irish or Belgium cards, it just immediately flashes "card declined" every time with no further info. There's about a 0.1s delay between clicking the pay button and getting that message so it's not querying my bank.
If my org wouldn't pay for them, I would start looking for other jobs (unless your in an industry that can't use them like national security, etc), but I'm not going to let $30/mo stop me from using a tool that significantly improves my work just because of a middle manager.
Also, many programmers I know bring their own keyboards.. Chefs often bring their own knives, or mechanics with their own tool chests. Ownership of your own tools is powerful.
Have you ever tried getting a company to fund a developer tool? More often than not, it's like pulling teeth, even if the service in question is a paltry $20 a month.
I've learned to pay for things like OpenAI myself because it's not worth the hassle to get a company to pay for it (for the purpose of R&D).
FWIW I am using a UK debit card. It's hard to believe they would have reason to block cards from Belgium or Ireland.
10x engineering comes from someone who is trying to "be a better developer" actively for many years. Most people who just use the tools to create the product won't ever become that much better. They will just become more knowledgeable. It is the same as any natural skill. You can play piano for years and never really get better. Or you can try specifically to do hard work of improving your technique, theory, and performance ability directly. It will be way more emotionally taxing to do so, but the end result after years of doing this is that you're vastly better than most. Not everyone can even do this. You definitely have to be in a growth mindset and not particularly stressed emotionally at the outset and during the process.
My coworker at Amazon was just making macros nonstop for all these things he wanted to automate within his workflow. He would focus specifically on the hardest problems that we haven't solved first and everything else around it would just spring up together rapidly (I don't even know how he did this). He ended up being heavily involved with AWS and later went to Netflix.
I was working on a .pkg MacOS installer and could not figure out how to get my background image @2x.png to render on a retina display. Turns out I needed to create a directory with the .imageset extension. My attempts at searching for a solution online never led me to this, but ChatGPT immediately knew what to do when I described my problem.
It's also tremendously useful as an interactive manpage. I've found it's much faster to ask ChatGPT how a complicated CLI tool works than trying to read the dense --help output to find what I need.
It can even have a conversation about A* pathfinding, and how best to implement it with different nuances. It's crazy.
I wonder how employers will handle this trend, which is happening so quickly. Is it OK for an employee to get advice on their work from ChatGPT? Is it OK to commit AI-generated code? Is it OK to show internal source code to an AI to help them find a bug or refactor it? There are security implications here, and ethical questions. I think most sophisticated companies will only adopt this kind of tech when they can self-host it, or at least have some kind of guarantee that their employees are using a dedicated, isolated model.
With that said, I'm sure many people are already making heavy use of this for their work (as are students). It's already an out-of-control phenomenon.
10x engineer cut the implementation complexity from 10x -> 1x, and then proceed to do that, while still satisfying business/product intention.
Generating 1000x code is more like being a -1000x engineer.
One of the examples from GPT wasn't even responsive and didn't work on mobile.
But...some folks are creating iOS games without any Swift experience, so that's cool.
We need to find a way for the system to iterate, debug, write unit tests, and repeat the loop. In theory it's doable, but we are yet to see the system which does this.
GPT is just Spreadsheets in a way. Lot of companies had business users and SMEs create contraptions based on Excel and then when it's a hairy mess or a rube goldberg machine, there are "digital transformation" projects that are done to make them proper apps.
It's almost useless for high-level design and architecture. Those tasks do take more time than writing code, but they're super-charged by rapid prototyping, which ChatGPT excels at. For example, right now, if I'm picking between three libraries, I can have working versions of my code implemented in all three in a few minutes, and built out a bigger mockup in a few hours. I can make a more educated decision from there. In the olden days, that'd be a few hours and many days, respectively.
It also does really nicely for code reviews, which results in better code. That also speeds up maintenance and, later, debugging. That's especially true in domains I'm less familiar with.
Some maintenance, it excels at. I'm the author of a major piece of open source infrastructure, which is widely used, well-architected, but every technology used is obsolete (JQuery, older Python web frameworks, legacy database, etc.). I feel like I could rewrite the whole thing in React+Node+etc. in a few weeks. There is close to zero architecture work -- it's mostly having GPT write -- or sometimes just translate -- code piece-by-piece.
I don't feel like GPT-4 obsoletes software engineers as it's used yet, but it does really raise the bar on what's possible to build.
Of course I'll try it and see how it does. But it struggled and failed to produce what I would consider trivial code when I tried to create a single usable page to spec in react native, so I'm not terribly optimistic.
What's interesting is that GPT isn't trained to produce high-quality code. It's trained as an autocompletion tool. I'd be curious how smart such a system could be if we knew how to engineer it for quality.
It won't find subtle race conditions, abstraction violations, or deeper issues.
I find it makes code reviews shorter and more productive, so perhaps you'll spend 10% of your time instead of 60%. You won't spend 0% of your time, though.
I just hate how slow it is even with Plus. For analyzing code for debugging you’d ideally have something that moves a little faster.
I guess we’re in the 56k days of LLM watching it print line by line.
"I'm getting this error <error> with the following code. Please add log statements that will test for all the things that could be going wrong <code>"
Saved me about 15 minutes yesterday. Not only does chatGPT generate the probably causes of the error, it knows how to test the code for those errors. The human doesn't even have to read the error message anymore.
Thanks for this article, I found it very inspiring.
Do you think the company will let you just have fun learning for the other 90% of the time that you can free up? They’ll just expect more output. Being able to write code 10x faster will likely not free your time enough to become a 10x problem solver.
>Writes code that others can read.
>Reads the Docs.
>Updates the Docs.
In my experience, half of the differentiation between above average and solidly "1x" or middling devs is reading documentation and obtaining a deeper understanding of the tools, language, platform they're developing on top of.
Those of us that have deep architectural skills, are able to deal with ambiguous requirements, balance UX with technical limitations, etc, will be safe for a good while.
Those of us that were trained in bootcamps to crank out MEAN/MERN/etc apps might be in a bit of trouble.
I say this not to disparage the latter group--many are great engineers who do great, reliable work. But it's ultimately rote work, something LLMs excel at.
There is a huge tranche of SWEs who are going to become redundant soon. Lots of them gave up careers as teachers etc for a more lucrative career in tech. It's unfortunate.
Interesting times.
All they think is "right now it's cheaper so we'll do that".
Wrote about that here: https://simonwillison.net/2023/Mar/27/ai-enhanced-developmen...
I've always been curious what people feel the baseline of a 10x is (i.e. 1x). Is it a fresh developer out of college or what?
I like to improve my engineering skills. However I also like to take nice vacations, do wood carving, play mandolin, go for long walks in the park, see my kids grow up, and all the billions of other things people do in life. There isn't time to do it all: time spend becoming better at one is time stolen from the others.
*most people can solve problems effectively but, imo, it takes true crafts(wo)man to solve it -simply-
It's not a skill you can grind per se, although many try that fruitless path.
Now, here is a question:
George R.R. Martin has been writing on the 6th book of "Game of Thrones", titled "The Winds of Winter" (TWoW) for over 10 years. It still isn't finished.
What do you think, which will sell more copies: TWoW, or all novels of the 1000x novelist combined?
Their net worths show it: $65M for Martin vs $450M for King.
As far as I can tell, chatgpt performs the action of finding and simplifying the information needed to take the next step forward, with caveats for a lack of understanding unless asked and the potential to be inaccurate or wrong. But that's all it does at the moment. No better than asking the "devise expert" on my team how to setup devise from memory. Kinda cool experience, but I doubt it's a titanic shift because it doesn't have the depth of sematic meaning you and I do.
ChatGPT's output is certainly limited by its input. If you ask questions that can be answered by finding and simplifying information, then that's the answer you'll get. However, if you treat ChatGPT like a functioning software engineer, describe the problem space, explain the requirements, and define the boundaries, then ChatGPT will do much more. From that point on, you can ask it to explain why it chose a certain data structure, or to adjust an implementation approach, among other things.
With 15 prompts in 2 hours, I was able to pair program a novel 200-line program in Go that used AST parsing, command execution, pattern matching, file parsing, and custom data structures for specific data processing. All to solve a problem one of my teams experienced in the industry. ChatGPT wrote 99% of the code; I tweaked 2 or 3 lines because it had problems correcting an error in a regex (most likely due to the fact that ChatGPT sees tokens not characters). The code compiled, ran, and produced the exact result I wanted.
Researching, prototyping, and writing this code by hand would have taken me a few days probably. That's why the project sat on my TODO list for months
I'm working on trying to figure out lang chain to basically feed it all my vendor and node module code and their respective docs, so basically hallucinating goes way down and the info is as fresh as my cron job to rebuild the langchain embeddings.
I'm doing this for laravel, filamentphp, livewire, alpinejs, vuejs, nextjs, react, svelte, typescript, and PHP docs as well, mostly just the docs... I'll probably work on limiting the info to the most important things or just docs for some of the things...
I'm then planning on having gpt agents specifically tuned to write unit tests another to write code, another to review the code another to run and validate the tests, another to model data and create migrations, etc ...
naked chatGPT can do a lot, adding custom embeddings changes it into a master of x topic and levels it up 10 fold.
Easy, source out your ideas and work to a team of 1000 people (aka become a subdivision manager).
They enable ten other engineers to double their output.
"10x engineers" are a leadership position, not a development position.
I've repeated the "experiment" here having converted the queries to bash:
https://journals.appsoftware.com/public/76/227/4223/blog/sof...
This article is beautiful and effective. Thank you for the information!
The culture around GPT at the moment is verging on fanaticism.
When car companies close down and people lose jobs, it's sad for the die-hard followers, but most people don't care so much. This closure of trivium-level programmery is being met with a ramp up in fanaticism and repetitive doubling-down on preset views. So odd...
Even when I ask it the "generate a 10 row csv containing sample chipotle restaurant order data with headers" prompt, I get "Sure, here's an example CSV file with 10 rows of sample Chipotle restaurant order data, including headers:" at the top...
I wonder if the web UI has some additional "explain everything you do" prompt injection stuff going on.
Edit: got it to work with
Output an example of a CSV document with 10 rows containing sample chipotle restaurant order data. Do not start with any other text.
If you are using the API try lowering the temperature mb?
I think I will start changing my functional prompts to require a JSON format in responses so that various aspects of the response don't need to be manually parsed, and requests can be more reliably piped to subsequent requests.
> I think I will start changing my functional prompts to require a JSON format in responses so that various aspects of the response don't need to be manually parsed, and requests can be more reliably piped to subsequent requests.
that adds more characters though so maybe hits token limits faster, I was thinking about how to have better contexts then found out about langchain and creating a sort of long term memory using vector databases. still trying to figure this out, but once I do, I think it'll be amazing what I can do with it.
- https://github.com/allen-munsch/hey_gpt_hey_whisper/blob/mai...
I tried GPT and it’s unable to write anything but the most simple code from simple requirements.
As a tiny bit of complexity and it’s lost.
Are people at work dealing with tiny code bases with well defined requirements?
So, I still don’t get it.
it's a slightly better version of google
this has built in history and lots of other features that a serious user needs, like checking token limits and estimating costs, etc.
{ "error": { "message": "The model: `gpt-4` does not exist", "type":
"invalid_request_error", "param": null, "code": "model_not_found" } }I tested the scripts and they work really well, esp like the image generation script. However unlike the web-browser the scripts does not keep the context, which is not great.
is there a way for shell script to maintain context session somehow?
Feed the output of the previous prompt back into itself.
If you look at https://platform.openai.com/playground/p/default-chat?model=... you will see that the prompt for the next line is all of the previous generation conversation.
This can consume tokens at an accelerating rate.
AI: I generally remember the last few conversations and I'm also capable of keeping context in longer conversations."
I don't think there is a way to feed context back to the script efficiently, maybe need a local database or a queue of certain length to sustain the subject. To mimic the web-browser experience at terminal.
the real magic for chatgpt vs old chatbots is that it maintains context, how to keep that via API calls is something I do not fully understand yet. Need read the API reference I guess
For
The following is a conversation with an AI assistant. The assistant is helpful, creative, clever, and very friendly.
Human: Hello, who are you?
AI: I am an AI created by OpenAI. How can I help you today?
Human: I am fine. Working hard.
The following is a conversation with an AI assistant. The assistant is helpful, creative, clever, and very friendly.
Human: Hello, who are you?
AI: I am an AI created by OpenAI. How can I help you today?
Human: I am fine. Working hard.
AI: That's great to hear! Do you need any assistance with anything?
Human: How many peks are in a bushel?
And then I get:
The following is a conversation with an AI assistant. The assistant is helpful, creative, clever, and very friendly.
Human: Hello, who are you?
AI: I am an AI created by OpenAI. How can I help you today?
Human: I am fine. Working hard.
AI: That's great to hear! Do you need any assistance with anything?
Human: How many peks are in a bushel?
AI: A bushel contains 8 US (or 32 UK) pecks. Is there anything else I can help you with?And then...
The following is a conversation with an AI assistant. The assistant is helpful, creative, clever, and very friendly.
Human: Hello, who are you?
AI: I am an AI created by OpenAI. How can I help you today?
Human: I am fine. Working hard.
AI: That's great to hear! Do you need any assistance with anything?
Human: How many peks are in a bushel?
AI: A bushel contains 8 US (or 32 UK) pecks. Is there anything else I can help you with?
Human: What produces does that measure?
AI: A bushel is a dry measure used for measuring fruits, vegetables, and other grain products.
Human:
To do this as part of a command line interface, yep - it would need something like .gpthistory which gets fed in to each message... though this eats tokens.
You can see some of this in the playground if you look at the 'view code' button at the top.
...
..
.
10,000x shitter !
What was the last time we could look back at old HN threads and say that? Bitcoin?
My nuanced 2c on it is that every line of code you write is a liability. Part of what revolutionized pricing mechanisms was FOSS and SaaS, which are 2 strategies of amortizing the cost of development across many many people. Every line you write for your company is a line with amortization across exactly 1. So first consider the value/cost tradeoff of FOSS and SaaS to see if you can compete. 1000x might be simply glueing together a few free/cheap systems which would have cost your company quarters. Putting in nice interfaces/adapters means you can easily cut them out if they become a regrettable decision.