These aren't mutually exclusive. I pay for ChatGPT. It sucks fat balls at coding but it's okay to do things like "Bash: ensure exactly two params are passed, 1st one is a dir, 2nd is a file". This is slightly faster than writing it myself so it's worth $20 a month but that's about it.

"from now on no explanation, code only" also helps.

Does is still stuck? Definitely. But it's one more tool. I can understand why one wouldn't even bother though.

Maybe this is the problem ? I quite like using LLMs for coding, but I don't think we are in a position where a LLM is able to create a reasonable commit.

For me using LLMs for coding is like a pair programming session where YOU are the co-pilot. The AI will happily fill you screen with a lot of text, but you have the responsibility to steer the session. Recently I've been using Supermaven in my editor. I like to think of it as 'LSP on steroids', it's not that smart but is pretty fast and for me this is important.

Another way I use LLMs to help me is by asking open-ended questions to a more capable but slower LLM. Something like "What happens when I read a message from a deleted offset in a Kafka topic?" to o1. Most of the time it doesn't give great answers, but it generally gives good keywords to start a more focused Google search.

The latest models (o1, Claude sonnet new) are decent at generating code for up to 1k lines. More than that, they start to struggle. On large code bases they lose the plot quickly and generate gibberish. I only use them as code summarizers and as a Google replacement in that context.

So using Ai isnt just about learning some stack or tinkering with a model. Its learning how to communicate with the AI under the current constraints.

Perhaps this goes against the ultimate goal. Speak plainly and AI understands. That's AGI. Today, we need to speak AI English, and that's a new knowledge domain all its own. Hopefully it will be shortlived.

1. Generally you want to give the LLM one well-specified task at a time. If you weren’t specific enough initially try clarifying a bit maybe, and if it makes a mistake maybe try one round of fixing it in the same conversation. Otherwise I always recommend putting followups and separate microtasks in a separate, new conversation (with some context carried over and some no-longer-relevant context pruned).

Every time you call an LLM it takes the entire conversation history as a parameter and generates the most likely response, which at least last I checked was O(n^2) for leading models. Long conversations force it to sift through tons of junk, bias responses towards what the model did previously, and often confuse it regarding objectives and instructions.

2. Don’t let the model make you forget that you know how to write software, and don’t believe everything it says. Make sure you actually read and understand the code it spits out, and try to think at least a little about any errors it causes. If it got most of the way there it’s usually easier to just do the last bit of code yourself IME, and you can still Google your errors as engineers have done for decades now.

3. Treat the model like the “doer” and don’t let it do the thinking. It’s great at converting instructions and code to more code, and for knowing lots of stuff about most things on the Internet, and to use as a sounding board. Anything more intellectually challenging than that you probably want to scope down into simpler stuff.

TLDR is you need to build a Theory of Mind for how to interact with LLM coding tools and know when to take over.

A little background...

I got my undergrad in physics (where I fell in love with math), spent some years working, and got really interested in coding and especially ML (especially the math). So I went to grad school. Unsurprising I had major impostor syndrome being surrounded by CS people[0], so I spent a huge amount of time trying to fill in the gap. The real problem was that many of my friends were PL people, so they're math heavy and great programmers. But after teaching a bunch of high level CS classes I realized I wasn't behind. After having to fix a lot of autograders made by my peers, I didn't feel so behind. When my lab grew and I got to meet a lot more ML people, I felt ahead, and confused. I realized the problem: I was trying to be a physicist in CS. Trying to understand things at very fundamental levels and using that to build up, not feeling like I "knew" a topic until I knew that chain. I realized people were just saying they knew at a different threshold.

Back to ML:

Working and researching in ML I've noticed one common flaw. People are ignoring details. I thought HuggingFace would be "my savior" where people would see that their generation outputs weren't nearly the quality you see in papers. But this didn't happen. We cherry picked results and ignored failures. It feels like writing proofs but people only look at the last line (I'd argue this is analogous to code! It's about so much more than the output! The steps are the thing that matters).

So there's two camps of ML people now: the hype people and "the skeptics" (interestingly there's a large population of people with physics and math backgrounds here). I put the latter in quotes because we're not trying to stop ML progress. I'd argue we're trying to make it! The argument is we need to recognize flaws so we know what needs to be fixed. This is why Francois Chollet made the claim that GPT has delayed progress towards AGI. Because we are doing the same thing that caused the last AI winter: putting all our eggs in one basket. We've made it hard to pursue other ideas and models because to get published you need to beat benchmarks (good luck doing so out of the gate and without thousands of GPUs). Because we don't look at the limitations in benchmarks. Because we don't even check for God damn information spoilage anymore. Even HumanEval is littered with spoilage, and obviously so...

There's tons of uses for LLMs and ML systems. My "rage" (as with many others) is more about over promising. Because we know if you don't fulfill those promises quickly, sentiment turns against you and funding quickly goes away. Just look at how even HN went from extremely positive on AI to a similar dichotomy (though the "skeptics" are probably more skeptical than researchers.[1]). Is playing with fire. Prometheus gave it to man to enlighten themselves but they also burned themselves quite frequently.

The answer is:

you evaluate in more detail than others.

[00] of course it is. LLMs replicate average human code. They're optimizers. They optimize fitting data, not fitting optimal symbolic manipulation. If everyone was far better at code, LLMs would be too. That's how they work

[0] boy, us physicists have big egos but CS people give us a run for the money

[1] I have no doubt that AGI can be created. I have no doubt we humans can make it. But I highly doubt LLMs will get us there and we need to look in other directions. I'm not saying we shouldn't stop perusing LLMs, I'm saying don't stop the other research from happening. It's not a zero sum game. Most things in the real world are not (but for some god damn reason we always think it is)

At first Cody generated a single test case, then I asked if to consider all possible scenarios in that function (around 10), it generated 5 cases, 3 of which were correct and 2 scenarios were made up (it tried to use nonexistent enum values).

In the end I used it to generate some mocks and a few minor things but then I copy pasted the test myself a few times and changed the values manually.

Did it save me some time in the end? Possibly, but also it caused a lot of frustration. I still hope it gets better because generating tests should be a great use case for LLMs

* It’s actually easier to do TDD or black box testing with LLMs. Yes, the lazy approach is to feed it a function implementation and tell it to make a unit test. But you can instead feed it the function definition and a description of its behavior (which may be what you used to generate the implementation too!) and have it generate a unit test with no visibility to the spec.

* Unit tests tend to have a lot of boilerplate sometimes, often not copy-pastable (eg Go table test cases) and LLMs can knock that out super quickly.

* Sometimes you do actually want to add a ton of unit tests even if they’re a little too implementation-focused. It’s a nice step towards later having actually-good tests, and some projects are so poorly tested and plagued with basic breakages/bugs that it’s worth slowing down feature development to keep things stable.

Personally I hate when people try to automate this stuff though, because it does trend towards junk. I find it better to treat writing tests with LLMs tactically, basically the same way you use them to write code.

I also think traditional tools are better for this. The test, generation methods include path-based, combinatorial, concolic, and adaptive fuzzing. Fire and forget tools that do each one, suppressing duplicates, would be helpful.

What would be easier to train developers on are contracts (or annotations). Then, add test generation either from those or aided by static analysis that leverages them. Then, software that analyzes the code, annotates what it can, and then asks the user to fill in the holes. The LLM’s could turn their answers into formal contracts or specs.

That’s how I see it happening. Maybe add patch generation into that, too. At least two tools that predate LLM’s already find errors and suggest fixes.