Egregoros

LLM generated comments:

# Create connection to i3
i3 = i3ipc.Connection()

...makes me think of this ancient blogpost from the 2000s about not writing useless comments that's probably disappeared from the Internet by now:

/*****************************
*     Adds 1 to x                         *
******************************/
x = x + 1

@djsumdog To be fair, when it's a bot writing the code you first have to confirm that the line does what it's supposed to.

So what's interesting is that if you tell an LLM to "show its work," you sometimes get a more accurate result:

https://www.lesswrong.com/posts/XNBZPbxyYhmoqD87F/llms-and-computation-complexity

The comments are probably not so much for verification as they are a required part of the emergent process .. and it still makes up function names that don't exist anyway 😩

@djsumdog @Zergling_man humans make up function names that don't exist too, if you don't let them run the code, don't let them look anything up and ask them to code a few thousands LOC on paper. you can solve most of these issues by giving the LLM ground truth that it can check its work with (i.e., a compiler it can run, a test suite it can run, for network software: a few server implementations it can test against).

I usually refuse most agents when they ask to run commands. they don't make i easy to just give the LLM feedback if you run the command (or something close) yourself. The few times I've allowed an agent to run build/test commands on a difficult problem, it usually goes into a big loop that just eats through tokens (even with Claude Opus vi Copilot .. because it's free paid for by my job)

@djsumdog @Zergling_man have you tried recently with frontier models? (chatgpt 5.2/3, opus 4.5/6) i really don't have this problem.