This PR adds split_lock_detect to the preserved_arguments list in anaconda.conf.
Currently, on recent kernels, if an x86 split lock is detected, it can cause the kernel to crash (#AC: crashing the ...
Not just automated testing but, for CodeGen to really work ‘agentic’ like:
You need that automated test case to trigger the misbehavior 100% of the time (often, the act of figuring out how to trigger the misbehavior means you already know the fix, but not always)
That automated test case needs to be succinct and as much as possible, feed only the problematic output back to the CodeGen. CodeGen can easily get distracted by irrelevant input
That automated test needs to be very quick from time to code change to test case completion. Even with everything just right, expect the CodeGen to basically thrash around guessing things that sound right but to no avail. Most attempts summed up as:
"Ok, the problem is absolutely caused by <plausible but useless prose>, and here is the definite fix <code changes that do nothing at all for the error> and it is complete but just double checking… <test fails> Ok, that didn’t quite fully fix it… see next attempt. So a long test case can make it take an eternity as the CodeGen has to wait and run it over and over and over again, while a human might actually reason through it.
You need to let the token hose go. It’s guessing and it can take quite a few guesses to get right.
Be prepared for pointless code changes along the way. It makes guesses and often leaves the wrong guesses in, doing nothing at all to help the problem, but potentially having side effects. It decides that while it didn’t work, it must have been a part of the solution, and that it must be left in.
Consequently, you better have an amazing test suite to capture the likely side effects of those spurious changes, or be prepared to unwind the progress and extricate the result manually.
Absolutely 100% all of this, though with a lot of other tricks like caveman mode and careful skill files and helper scripts to help the agent quickly surgical extract out just the useful output, you can substantially reduce token burn and improve its memory.
As well as carefully having it rollback changes everytime a fix doesn’t work, and having ut keep a markdown file log of each fix it tried and the results, so it can review each thing it tried previously.
All this and my ultimate conclusion has been that it’s nicer to just use it in targeted scenarios.
If I know it’s the shirt of thing it can knock out in one go without too much code to review while probably not failing, sure. If I’m not sure and there’s a misbehavior that I know will be tedious to sort out, but that it might sort out, then I’ll at least give it a couple of attempts.
If it would take more than a couple of tries, I’ll just go ahead and do it myself. It can complete along the way and I might prompt up a few lines here and there, but I’ll just think through an observed bug rather than endure the codegen grasping at straws.
Also, I’ve seen where it avoided the behavior, but upon review the fix illustrated to me what was wrong, but also that it was absolutely the wrong fix that failed to address a broader underlying issue that would crop up later.
So I’m not on board with the “agentic pipeline” approach. For the same reasons I’ve almost never found a human developer I trusted. Of course the developer behaviors that inspired that mistrust also help me understand why so many people are relatively comfortable with the concept.
Not just automated testing but, for CodeGen to really work ‘agentic’ like:
Absolutely 100% all of this, though with a lot of other tricks like caveman mode and careful skill files and helper scripts to help the agent quickly surgical extract out just the useful output, you can substantially reduce token burn and improve its memory.
As well as carefully having it rollback changes everytime a fix doesn’t work, and having ut keep a markdown file log of each fix it tried and the results, so it can review each thing it tried previously.
All this and my ultimate conclusion has been that it’s nicer to just use it in targeted scenarios.
If I know it’s the shirt of thing it can knock out in one go without too much code to review while probably not failing, sure. If I’m not sure and there’s a misbehavior that I know will be tedious to sort out, but that it might sort out, then I’ll at least give it a couple of attempts.
If it would take more than a couple of tries, I’ll just go ahead and do it myself. It can complete along the way and I might prompt up a few lines here and there, but I’ll just think through an observed bug rather than endure the codegen grasping at straws.
Also, I’ve seen where it avoided the behavior, but upon review the fix illustrated to me what was wrong, but also that it was absolutely the wrong fix that failed to address a broader underlying issue that would crop up later.
So I’m not on board with the “agentic pipeline” approach. For the same reasons I’ve almost never found a human developer I trusted. Of course the developer behaviors that inspired that mistrust also help me understand why so many people are relatively comfortable with the concept.