Please don't use AI to get on the global leaderboard
adventofcode.com
adventofcode.com
In other words I feel really bad for bootcamp people right now.
https://rarehistoricalphotos.com/life-before-autocad-1950-19...
How much of (our, as in, programmers') work is the "writing code" part, vs. all of the other stuff? Build consensus, gather feedback, communicate ideas, resolve ambiguities, make non-zero progress in the absence of clear direction, etc.
In my experience, For a junior/entry level, 80%+
For a (actual, not just title) Sr. dev? Typically <= 20-25%.
10–20 lines of code per day means the “writing code” part is about 2 minutes out of an 8-hour day, if you only count the code that ends up being delivered. maybe 10× that including unit tests, drafts that get replaced, duplicate code that gets refactored away, test code to see what a library will do, etc.
but ai can obviously handle a lot of that stuff too
A standardized way of working is sometimes better because if I have to work with someone- then it helps to be able to work on the same thing together. Tools like Visual Studio, Microsoft Office, even Netbeans or Eclipse allow us to converge on a highly productive but standardize working environment that we can work from.
If I'm running off a highly customized setup, and my partner is also doing the same thing - then I am only functional at my own workstation and nowhere else.
You may have standard vim bindings mostly to manipulate text.
But there are bindings in other IDEs to find next definition/declaration of the function under the cursor, debugging, refactoring, and so on. Within the same IDE, you will find most people don't customize those. At least, in my experience, anyways.
You will still have to read the boilerplate of course.
Basically, take some yaml like:
```
stuff: 1
things.and.stuff: 2
```
If you unmarshal this yaml into a `map[string]int`, the string keys will actually be, "stuff" and "things". Completely dropping the ".and.stuff". Wild. Keep in mind, this is specific to the Viper unmarshaller, the stdlib json or yaml unmarshallers actually handle this fine, I just have other similar complaints about those.
Given this is my main complaint, not performance or general syntax, I do enjoy writing applications in the language though.
What's the language you're using these days that you feel accomplishes the jobs you need it for better than Go? And preferably also has strong concurrency builtins and is typed.
As for specific languages, that's not really my area of expertise. Mojo looks promising in some ways. But it may never be suitable for systems programming.
Anyway, with it in mind that I believe that is not a short term goal: in the short term, I still need to make money. So I write Go for a paycheck. And based on what I believe the tools available around me today are capable of, I think by _today's_ standards, it is nice to work with in a wide variety of tasks.
The way out seems obvious. Either actually stop caring about why people think A sucks or to curtail my curiosity and stop clicking on the "A sucks" articles; and to care more about why people think A is great or at least click on the articles.
llms are better at reading and explaining code than at writing it in my experience
So you built an AI that can parse a perfectly written spec and produce code? The only thing that changed is that the person writing the spec is now a programmer.
So I guess there have been no programmers since the 1950s.
of course it has been, even very recently. where do you think all the demand for bootcamps and bootcamp graduates came from years ago? Why does basically every SWE interview consist of a large % of programming tests?
As for the rest of your post, I didn't say anything like that at all - but it seems like a trivial conclusion to say that "coding" will become less valuable as the interfaces become simpler to it (such as human language interfaces). This is hardly controversial and probably axiomatic.
From managers who know nothing about programming.
Yes, but that is not necessarily the software developer of today.
Not just self-proclaimed programmers either. I've had a chap claim 5+ years of C++ experience working on GE nuclear reactor software, who couldn't explain how C++ memory management works, or delete vs delete[].
So yes, programming interviews it is, unless we get some different form of accreditation like doctors or lawyers.
For what it's worth, I like practical programming problems. The problem comes when the programming interviews don't match the job. Why do I need to write a sorting algorithm in an interview for an SRE position? (I worked at Dropbox for a couple of years, and my interview there was probably the best SRE coding interview I've had - detect duplicate files in a filesystem, with follow-ups about improving performance.)
You’ll still need experts who understand the generated network code, and experts who can untangle the generated code, but not so many.
Code will be a tool used by non-experts; fewer special-case experts will be needed. Where we’ll be when the experts die out, I’m not sure.
The difference between the average bootcamp graduate and the average engineer isn't intelligence, it's knowledge. Once AI makes bootcamp graduates obsolete, I see no reason why engineers in general won't be obsolete too.
Not necessarily a completely different job, just one where AI won't be able to help all that much without another technological breakthrough, like working with large (several million LOC) codebases.
Coding is a means to an end, and when someone only cares about writing a bitchin' line/method/program struggles to identify and understand (or more importantly communicate) the end, what good is the means?
You can pass job description for Advent of Code task to these curve fitting tools and it will return solution. It is not just about coding. Other technical skills may have become less relevant too.
Rust implements some algorithms [1] for the same API with the sole purpose to be interchangeable and readily replace one with another. Does a computer know how to do that? I find that not to be the case at the moment.
Algorithms were important in the past, but a lot of programming effort had to be put on mastering every compiler/OS/architecture and their quirks in every case. Now the focus of programmers, will shift more to knowledge about algorithms and data structures.
we'll just go back to old school raffle ticket contests.
Maybe they could have leaderboards for smaller groups that you arrange with your friends and colleagues that you trust.
Which Advent of Code has.
Reminds me of the early arguments around automation in racing and whether humans should still be required to shift the car manually once the automatically actuated gearbox became faster and better than a human operator.
In all of these things, humans are transitioning from a place of needing to do work to get the task done, to a place of doing things because they want to do them, because it’s fun, or they learn or it benefits them.
Will people still write code by hand once the machine is clearly superior at it?
Probably not for a living, though one can still do it for some sort of intrinsically-motivated fun.
Folks still enjoy doing woodworking and make boxes, tables, benches, etc. -- even though they could easily buy them new, used, or for pretty cheap (say, at IKEA).
Folks still enjoy making photorealistic sketches/paintings, more than a century after the invention of photography.
Folks still enjoy playing musical instruments and/or singing, even though recordings exist of the world's best musicians performing all the songs you might want to play.
Your observation is correct, though IMHO restricted to a professional context -- in the hobby/personal enjoyment space, I see no reason this should change things.
For what it's worth, many woodworkers prefer making their own stuff because it is of far higher quality than what you can buy at IKEA (and similar stores). With the exception of extremely high end and expensive stuff, most furniture is complete junk. Making a coffee table out of a solid slab of walnut, for example, will last a lifetime compared to the cheap MDF table at IKEA.
Obviously I nor anyone else can predict the future, but I wouldn't be surprised for a similar divide to appear with AI generated software vs. completely hand written software where the hand written software is of higher quality for those that desire such things.
At least for now, we still have the unsolved 'robotics problem' where the data to train humanoid robots to do woodworking better than a human doesn't exist in sufficient quantity to make it work using current AI training techniques. This too may soon change via synthetic training data or some modified technique.
Conversely, my own experience shows that even with context window limitations, ChatGPT is already better than 90% of working software engineers for generating small practical programs. Here's why it's better:
1. It's better because it not only knows how to code but is equally knowledgeable about product management, UI design, business practices, etc which makes it more like interacting with an 'architect level' engineer 2. It knows a good if not 'the best' way to do most things
A CNC?
> Conversely, my own experience shows that even with context window limitations, ChatGPT is already better than 90% of working software engineers for generating small practical programs.
Well if we're doing anecdotal experiences, I find ChatGPT (both with 3.5 and 4) useful for replacing quick search queries to look up documentation but for anything more complex than that and it quickly starts making up blatantly incorrect examples even for small programs making it more than useless most of the time for me at least. But you're welcome to disagree; I don't have the energy anymore to get into my 1000th internet discussion full of anecdotal evidence trying to validate the future merits of generative AI for software engineering purposes.
Not CNC, right?
Yeah I think it's fun to discuss. I think we're saying the same thing. I'm not saying ChatGPT could replace a professional software engineer for most things at this stage. That clearly isn't practical yet.
As you said, it does make mistakes, though it can often correct them if the context window is there. The differences in how we use it for programming tasks and what we find useful is probably related to our SWE skill levels. Yours is likely much higher than mine, as I've never been a professional SWE.
That said, for the many little automation tasks that business people often bug a real engineer for, like iterating through a CSV and hitting an API to do something, then writing a result, it's ability to write simple and useful python scripts has been amazing for me.
I believe it's getting close to the place where I could use to to prototype simple MVPs but it failed during my last attempt.
Not sure how you measure this. However, if by "small practical programs" you mean "trivial programs", then I see no reason to doubt your assertion.
That said, at least where I work, ChatGPT hasn't been a real benefit for real work. By the time you manage to craft the right prompts, then check and correct the code produced, then incorporate it into the larger project, it all takes a bit more time and effort than if you just wrote it yourself in the first place.
It’s the stuff that you should not bother an engineer to do, but it can’t be done (in a reasonable timeframe) by a business person or even a technical IT person who isn’t proficient at code.
There’s a space there, a practical need for things that are boring or obnoxious for an actual SWE but that fill some business need or use case. ChatGPT fills these in nicely.
It probably costs roughly the same as those cheap tables in terms of materials costs and time/labor/skill. I doubt many if any people are doing it if they hate woodworking; meaning the cost/quality does not justify it.
I think it turns out that humans will want to work or event need to work in many ways.
I still want to garden (ripping out weeds is oddly gratifying), practice guitar, work on my old Porsche, hack on various projects, etc.
As the saying goes, I could make that myself in six months for twice the price
Well, I certainly will because writing code by hand is fun. I'd probably stop developing professionally, though, because using AI to do this stuff is decidedly unfun for me.
I the switched copilot to the mode where it only suggests things when I ask for it and that was a lot more fun, figuring it out myself and of I got really stuck I could sometimes get a hint from someone who had a similar approach.
The leaderboard? I guess I can see it for a very small minority. Cheating may be more of an issue, but again, doesn't affect the majority of the participants.
Go buy some AoC swag :-)
We do private leader boards, which are a good bit of fun. A couple of our EVPs opted to use AI to help them with it which certainly greased the wheels on getting it rolled out to our development teams. A fun way to learn how to use the AI in a project. The code reviews that came later were hilarious.
Using AI seems like it will take the fun out of it, but what do I know, I am yet to be interested in incorporating AI into my workflow.
On a side note, just checked out their swags, considering getting some this year.
Unfortunately, they don't get the non-satisfaction you think they get.
Cheaters just want to win and will do anything to achieve that goal. The satisfaction is from the winning, no matter the means. They just like to see their name at the top of the leaderboard.
Of course, in the future presumably AI will be able to solve even hard top-coder contests.
People need to understand that coding is just a tool for a job. The great thing about programming as a tool is new tools are constantly coming out to make you faster. If these people are using the best tools possible, how can one hate? Maybe you need to design the questions alittle better if it goes against the "Spirit" but it's not cheating....
No, but it's reasonable to say "we want to get a sense of your baseline understanding of things so please don't cheat with LLMs or otherwise."
ChatGPT will, more often than not, come up with a stupidly complicated solution like to my above example that doesn't work. It takes an actual engineer to figure out why it is impossible and solve it correctly.
If I was hiring people, things like that would be great questions. It’s also so simple - keep track of the times GPT-4 has no idea of what it is doing, and use those as your questions.
In ask people to describe me the code they are going to write verbally. Or reason through an algorithm verbally. Depending on the role, I sometimes start with FizzBuzz, with a spoken answer.
They can use a notepad. They can take time to think. But they will get "why" questions and need to understand the code or constructs they suggest.
This feels safe from ChatGPT, at least for the moment.
> Can I use AI to get on the global leaderboard? Please don't use AI / LLMs (like GPT) to automatically solve a day's puzzles until that day's global leaderboards are full. By "automatically", I mean using AI to do most or all of the puzzle solving, like handing the puzzle text directly to an LLM. The leaderboards are for human competitors; if you want to compare the speed of your AI solver with others, please do so elsewhere. (If you want to use AI to help you solve puzzles, I can't really stop you, but I feel like it's harder to get better at programming if you ask an AI to do the programming for you.)
I feel like "automatically" and "handing the puzzle text directly to an LLM" imply that copilot-style, really-good-autocomplete AI is permissible, but passing the entire problem into the model isn't.
There is some nuance though: sticking the problem in a comment and then letting Copilot complete it is a clear "handing text" violation, so Copilot is not 100% approved.
(Edit: missing word)
If the writer wants to pollute their own event with this weak distinction, they can. My concern is that it's just feeding confusion and rationalizing about cheating in other areas.
Perhaps throughout the next 5 years it will be a cliche to hear "I didn't use the AI to cheat; I only used it as an aid." from cheaters.
In this case it would involve creating a second leaderboard and letting participants register for either the manual or AI leaderboards.
There will still be trolls, it's not perfect. But the majority of these players simply disagree with your definition of cheating and will happily self-select in return for anointing their playstyle and giving them an arena to square off in with other like-minded players.
Plus you'll get lots of good data, tagged by your users, as to what a cheating score looks like, what IPs they come from, etc. You could use this to prune your other leaderboard if you really cared.
In 2023, "Getting betting at programming" means learning how to use an AI.
What is to stop the owners of these ai systems from denying service to users for trying to make a product that competes with them? Or just straight up taking your work and using it themselves?
Why risk it?
That is. If your goal is to quickly get out a prototype that may or may not work (even though you don't understand it very well), using AIs is great. But if you want to improve as a programmer, it may not be the best (or only) path.
If people are mad that Twitter gives a megaphone to everyone, including the ignorant masses, then they'll love the auto-spam that LLMs are going to create.
Anything that can be said will be said.
You want a Reddit with humans? Ha. On the regular non-human-verified discussion platforms of tomorrow, you'll be lucky if 4% of the comments you are replying to and arguing with even have a human on the other end, but the good news is the rebuttal comment you posted after having too much coffee will be ingested and used for training of the next version of the model you're arguing with. So your original content human-input may be parroted much more broadly than it would've been on the pre-LLM web.
If LLM spam really does flourish and spread misinfo and hallucinations everywhere and we don't develop good automated means to prevent it or to verify content, it may be necessary for a central authority/business to maintain hardware terminals at distributed, centralized locations for interacting with the humanweb that you can't install or control the software on and where a human or a camera is watching you physically type on the keyboard to make sure you aren't just automating the inputs physically with some software->machine->keyboard interface or connecting some virtual keyboard. Think a locked-down public library computer but you're watched while you interact with it, and they're deployed and administered across the planet by a trusted multinational for sensitive usages where you absolutely need to ensure the inputs are from humans.
You wanna get real fun and cyberpunk novel thought-experimenty, picture prison-like security, physical pat-downs or even a requirement that you use the terminal naked and are body-searched for devices. Maybe x-ray scanned for implanted hardware.
Of course the whole thing falls apart if the trusted authority that administers the hardware is compromised but at least you stop some of the non-state actors and script kiddies.
Or imagine a Costco Metaverse Verification Center. You can play in a VR metaverse with other verified humans at other Costcos around the world. AR cameras on the headset will ensure you can see your $1.50 hotdog and soda combo so you never have to leave the metaverse. Costco would also provide you a sleep pod at cost if you want to plug back into the matrix right after waking up.
At least not in a non-dystopian way.
Sam Altmann's WorldCoin tries to achieve this using retina scanners which I believe falls in the "dystopian" camp.
What we really want is certain types of content, and to ban others. If we get that certain type from a bot, that's fine; if the type of content we don't want is coming from a human, it should still be removed.
By "type" of content, I mean very broadly. For instance one could create a community in which there's a limited number of posts/characters/etc. per day, not just be looking at the characteristics of the content itself. I mean all aspects of the content, data, metadata, all of it, as part of the analysis of "desirable."
If you want a pure-human community, put constraints on the community only humans can meet; heavy-duty, unscalable identity verification may play a role there.
As a bit of a "how do you build communities online" hobbyist, I think another trend we're going to see is communities getting faster on the draw to evict participants (originally wrote "people" here, but it's actually generically "participants"), for reasons beyond mere spam or active antagonism. Historically, I think it's a thing that most communities have done; the American/Western zeitgeist has disfavored that idea for a while in favor of expecting every community to take everyone who wants to join, but regardless of the ethics or philosophy behind that idea, I think that's just going to become simply impossible online. If the standard for participation in some community includes bots that won't be evicted no matter what they do, that community will rapidly become just another bot congregation ground and look like all the rest of them. With people roaming the internet for new communities to infiltrate with their bots, community building will become a subtractive process rather than an additive one. That's going to be a big change, it isn't going to be smooth or all good.
I predict that this requirement would only decrease the amount of community and further increase the already high levels of isolation and alienation in society.
But I also predict that conversational AI will inevitably do this anyway, so perhaps we're just doomed.
Once the community gets going, though, well, we have experience with that. The web used to have a lot of actual communities, where you might know someone for 10 years and perhaps meet up for picnics or something. Larger sites took a huge chunk out of them, and there's actually some disadvantage to the Internet being completely geography-agnostic... it's hard to meet up with my community of 50 people spread more-or-less evenly across the world, or even the US. But they have existed before and they may exist again. I said it won't be all good in my original post, but it won't be all bad either. Some of what is going to be excluded in the botpocalypse is the worst of what exists today. Of course, there's going to be all kinds of incentives to create new pathologies, so who knows which way it will go in the end.
If the goal is to use the internet to produce interesting discussions and arguments, IMO it would be neat to try embracing the fact that bots are going to exist and get in the dataset. If bots produce outputs, and we pick the “good” output, that output can be smarter than the model, and go back to train the model, right?
I guess that "always on" verification or short duration authentication could make this strategy less useful.
But assuming we could, wouldn't the "human-verified" web just function as a data source to further train LLMs (or whatever)?
All the best non-commercial content will be created somewhere where creators don't need to rely on "hardware terminals at distributed, centralized locations for interacting with the humanweb that you can't install or control the software on and where a human or a camera is watching you physically type on the keyboard to make sure you aren't just automating the inputs physically with some software->machine->keyboard interface or connecting some virtual keyboard.", while on the other hand, commercial content farms will have no problem hiring a thousand minimum-wage employees to spend 8+ hours in those locations creating authentic, verified human-entered astroturfing spam.
edit: curious about the downvotes. AI is here to stay, but the Advent of Code site specifically states that “[t]he leaderboards are for human competitors; if you want to compare the speed of your AI solver with others, please do so elsewhere.” So creating a new site in the same spirit, but with progressive rules, seems like exactly what they’re advocating for themselves.
> more progressive and modern rules
which comes off as a bit of a dismissive insult.