Refactoring code suits LLMs well, but setup requires a bit of effort. Creating automated jobs lets me perform repository-wide refactoring that would be too difficult or tedious otherwise.
Here is a concise summary of the process.
- Environment: I setup an agent that has memory and can read my code and create pull requests.
- Exploration: I describe my request with an interactive session. I do iterations until I’m convinced the agent understands what it should do.
- Automation: I ensure the scope is valid and let it run to completion.
Besides the overall process, I have some extra tactics:
- Optimization: Getting the cost of the job down
- Keep it small and simple: Address blocks while keeping the jobs simple
- Start fresh: Being aggressive on junking bad code
- Dashboard: Managing many jobs at a time
Environment and framework memory
I use a combination of Claude locally and a custom interactive web session at work. The agent can view the code and follows global guidance for tasks like creating PRs. Our environment at work is quite complex now, but it started from something quite simple — and I started from a simple terminal Claude session with no extra tooling. You can expand the environment as needed.
Besides reading the code, the agent needs memory. This is the same concept that chat prompts offer on web sessions. In my environment this is a set of files — files that the agent knows how to read and write. I commit these to git a repo. This is different from the main source code repository, allowing me or the agent to make changes easily without integration control.
In there, I have a specialized refactoring memory. This use of memory is related to skills but is not explicitly invoked. Instead, I have global instruction triggers to pull in this concept when talking about refactoring. The instructions contain guidance I use repeatedly on all of my migrations.
If you’re uncertain how to create memory, just ask the agent. The concept is simple and common, so the agent can guide you on what to do. You might also try pasting in this article and then prompting it to “create memory.”
Some examples of the notes I have in my refactoring memory are:
- don’t create the PRs until prompted
- create PRs in draft mode initially
- batch migrations at a size of around 50 files to a max of 500 lines
- be succinct and direct
I always tell LLMs to be “succinct” even though it doesn’t always work. I’m unsure why, but they are verbose, especially when talking about code.
While working on my refactoring I update the memory often. This captures a lot of small details, improving the results and ensuring it goes into future effort. I will not however say that the results are ideal: as instructions grow, the LLMs become ever more liberal in their interpretation.
Exploration
My initial prompts are only one or two sentences. Though sentence fragment is perhaps a better term. I type a stream of thoughts quickly, which becomes a list of points, partial ideas, filenames and code snippets. I have seen no evidence that prompt eloquence improves the results.
For example:
frontend code can show a value with unit. currently ad-hoc, I want to abstract shared component. also drop moment; use browser standard.
I keep it simple since the agent can infer a lot from the code itself. This prompt should allow it to identify the patterns I’m referring to and recommend a component that abstracts them.
Or maybe it gets confused, finds the wrong thing, or suggests something inappropriate. This is why I start with an interactive mode. I now have a “conversation” with the agent. This again is minimalistic on my part, comprising bullet points, sentence fragments and strongly worded imperative instructions.
Tell me in code
As I mentioned before, agents tend to be verbose, emitting lots of prose about code. I try to home in on things of interest and act on partial readings. It’s a stark contrast to how I’d work with colleagues, where I listen and try to understand everything they say. Despite sounding friendly, the agent is just a tool. It doesn’t need validation nor politeness. It does not get upset when I ignore half of what it said or ask about something it already answered.
If I think the agent is on the same page as me, I’ll ask it to put up a draft PR. Honestly, I’ll often ask it to do this even if I’m not sure it understands. Somehow, it can often emit acceptable code despite its unintelligible prose.
I’ll review the code and instruct how to change it. When I’m happy with the first PR, I’ll merge it and tell the agent to do the next one. I’ll do iterations here until I get a couple PRs that don’t need any comments.
Then it’s time for automation.
Automation
I have a framework that will run an agent job on a regular schedule. When I’m satisfied my migration is working, I’ll add it to this framework.
The framework runs every half-hour, but it cycles through all of my refactoring jobs. They also have a light / heavy marker, so that only light ones can run every half-hour and heavy ones every hour. Initially I could fill the entire pipeline, but as my migrations become more nuanced I rarely fill all the slots. I only get bursts as a job reaches a highly mechanical bit that needs to apply to a lot of code.
During the iterative process it has likely become clear to the agent what is considered in scope, but it’s good to validate now. It should know what parts of the code it is working on. It should understand which patterns apply, the appropriate level of lenience, and what to avoid. Starting restrictive here is the best approach. It’s easy to expand the scope later, and often the agent itself can give recommendations.
For clarity, I don’t spend a lot of time talking explicitly about scope. Mostly, the agent figures out the scope through the description and iterations on the draft PRs. It emits work lists and scoping guidance to the job, but I rarely look at this directly.
As the framework chugs away, the agent produces PRs that need to get merged. At work we have an automated approval process for certain kinds of PRs. Many of the mechanical refactorings can be approved and merged automatically.
Others still require manual approval. I have a separate tool skill that lists which PRs I need to review for which jobs. I review this only once a day, so while my jobs do run every half-hour, they stall out waiting for my response. An explicit instruction tells the agent to not keep piling up draft PRs. For refactors that need a lot of PRs, like 50-100+, I may instruct it to manage four lanes of PRs for the one job. This helps prevent one draft from blocking ones that could be automatically merged.
If the agent encounters a situation it can’t deal with, I’ve requested it mark the refactor as needing attention. I then start a new conversation and ask it to explain the issue. Sometimes it requires a lot of guidance to get by a block.
Here again, I find it more useful to talk in code than text. For example, instead of having the agent explain what is blocking it, I can instead ask it to put up a PR that would unblock it. Then if that PR is fine I’ll let it continue. Or I may ask it to produce a PR that highlights the bad pattern it would produce.
I use PR drafts to look at code diffs. I would prefer if I my tools have built-in comparison views. Having a dedicated way for a tool to propose code, along with markers for attention and comments, would be ideal.
Eventually the refactoring job exhausts its scope. If the pattern is something that can appear again, the job will demote itself to an idle state. Every week, the idle job scans new code to find matching patterns. For more specific one-off jobs, or where something has been eliminated from the code, the job will switch to a done state.
Appendix: Strategies
A few recurring themes arise when using LLMs. They don’t change the overall flow, but they are significant enough to mention here.
Optimization
Having a complex prompt run on a regular schedule is expensive. It can also be quite slow. Instead of doing productive work, the agent is busy reprocessing the same information repeatedly. Optimization improves both cost and time efficiency.
Much setup work stays consistent across refactoring job iterations. I’ve saved over half of the total token count by eliminating this redundancy. It isn’t even hard. I simply instruct the agent to optimize as much as possible, putting the information into the job itself, or the nearby memory.
Optimization is highly beneficial for the scope. The job must identify the code chunk to process in each run. This involves code scanning, worklist scanning, or a combination of both — a large waste of tokens. Instead, I can instruct the agent to produce a script, usually python, that does the scope aspect. Compared to tokens, even inefficient scripts are cheap.
I’ve cut down token costs by over 80% on some jobs. This process usually cuts the number of turns in half as well, which benefits the speed. I have yet to find a job where optimization doesn’t make a large difference.
Aside from refactoring, in my interactive coding session, I also have the agents produce many scripts and skill knowledge on how to use them. Instead of the agent repeatedly fighting with the same output and setup, I issue similar instructions to optimize. This results in the memory containing pointers to several scripts. Usage isn’t optimal, but it is enough to make a difference, especially in avoiding repeated errors scenarios.
Keep it small and simple
Many refactoring jobs get stuck on something that is too far out-of-scope for them to address. I avoid having an individual job do too much, as it confuses the agent. It’s instead easier to park the current job and create a new job addressing the blockage. Once the new job is done, the original job can continue.
Much of this side-tracking comes from code that doesn’t follow current best practices. At work, as we improve the code, we add a lot of deprecated patterns and linters. Instead of having to migrate all existing code at that time, we create exceptions for pre-existing code. Refactoring this code often runs afoul of these restrictions. We could migrate the exceptions, or add some new ones, but it’s often easier to just fix-up the old code first. Some of our rules are strict, so adding new exceptions isn’t even an option.
Blockages are not always from deprecated patterns. Often the agent will find unforeseen issues with my planned migration. This could be a forgotten use-case, a missing technical feature, an import ordering issue, or a potential deviation in behaviour. Nonetheless, the main job remains blocked while I fix this problem.
I find this approach of side-tracking beneficial as it reduces the size of each individual refactor. The wackiness of LLM results seems to increase exponentially with the complexity of the prompt. Thus keep it small and simple.
Start fresh
I close a lot of PRs. LLM agents often produce nonsensical code. This is why I always start interactively and with some draft PRs. Until I see a pattern of producing the code I want, I distrust the agent.
I often close agent conversations without having produced any output. LLMs have this terrible quality of becoming stubborn. The conversation history ingrains a single thread of thought so heavily that you can’t steer it in another direction anymore. If I get two or three junk PRs, it’s time to start fresh.
If the agent seems close, but is stuck on something, it is sometimes better to start a new conversation. I got the agent conversation tool at work to do this with a /spinup skill. I give a short request to pull out some part of the current conversation and move it to a new one. This compact representation is often free of the anchoring history bits.
Dashboard and status tracking
I can get from 10 to 50 refactoring PRs merged per day across all of my jobs. It depends on which ones get blocked, and which ones need manual review. I also have from 5 to 15 automation jobs active at a time. I additionally have several manually stepped jobs I progress each day. On top of that I have many one-off agent threads. I need good tools to manage this.
Given that number of jobs, you can understand why I don’t want agents sending me prompts or vying for my attention. We use Slack at work and I try hard to avoid having any agent send me notices. They’re out-of-context and draw my immediate attention, messing with my chosen prioritization of work. And again, agents fail to produce helpful and concise notices, so they up working as only a message saying “hey, look at me”.
Instead, I have an agent emit a table of the current status of all my jobs. It notes the recent PRs and whether they need my review or are blocked. I have immediate context. I can just click a link to either review the PR or start working on unblocking the code.
We also have a web dashboard at work which I continue modifying to support a streamlined workflow. As my refactors got harder, the number of blocked ones increased. So instead of looking at one-at-a-time, I have a tool skill that spins up conversations for every blocked task and immediately instructs the agent to look for a solution. These conversations are added to my sidebar which lists all my active conversations.
Recap
Refactoring is perhaps the easiest place to get good automated results from an LLM. You find a pattern in the code, create a better pattern, and tell the LLM agent to migrate everything. Syntax patterns are the easiest to do, but it’s also possible for semantic and contractual patterns. As long as the change is mechanical, an automated agent can handle it.
This isn’t to say things haven’t gone wrong on me. Usually I catch problems early on, but LLMs aren’t precise tools and have no problem going on tangents and finding “clever” workarounds. Be attentive to what is happening. And if something goes awry, then just create a second refactoring job to fix the first one!
