Let me start with a hypothesis:
For human developers, using coding LLMs magnifies their difference in skill levels.
I am one that rarely thinks things are always black and white. Back in March, I wrote Artifial Intelligence: Shades of Gray. Since then, I’ve had more of a chance to experiment with LLMs myself. I also happen to work for an employer that is taking a very pragmatic approach to LLMs: teams and individuals use it as they see fit, but if they are causing considerable expense, they have to justify it.
In various settings, I have seen the egregious examples of AI slop we all know about. As I wrote in March, “I have seen it both waste more time than it saves, and save a ton of time.”
I have come to see that, as a tool, it is most valuable when it is running under the supervision of an experienced engineer. It is at its worst when it has no such supervision; the “vibe coding” and other low-quality slop we see.
A coding agent is like a junior developer or research assistant. When properly supervised, they help projects move along more quickly by letting a senior developer focus on the more difficult, less mundane aspects of the project. But one couldn’t expect a junior developer to consistently deliver high-quality code and architecture on their own.
Let’s put a pin in this idea and look at the story in Debian.
LLM use in Debian
There is a vote happening in Debian around the use of LLMs. In typical Debian fashion, there are 8 options to choose from, many of them similar. Most of these proposals acknowledge there are different types of tasks done in Debian, but the proposals don’t differentiate between them well. Let me do so here. These are some of the LLM-relevant tasks people in Debian perform:
- Packaging upstream software for Debian (by far the largest task)
- Writing Debian-specific code (eg, apt or the Debian installer)
- Maintaining Debian infrastructure (build systems, for instance)
- Writing documentation and translations
I’m going to focus my remarks here on packaging upstream software for Debian, since this is by far the most time-consuming developer task project-wide.
It matters to our users that we get this right, and packaging quality is one of the things that sets Debian apart from other distros. Packaging things for Debian requires knowledge of some specific tools, such as debhelper, that aren’t widely used anywhere else. In most cases, it is fairly rote time-consuming work. In other words, by its design, it requires people with senior-level skills to do grunt work.
I can’t overstate how massive a burden this grunt work is. I maintain some packages for Go and Rust. By Debian policy, all of those packages’ dependencies must also exist as Debian packages, and be used to build against. When upstream adopts a newer version of some library, it can unleash cascading dependencies that can take hours to sort out. Worse, the Rust team and the Go team use entirely different ways of managing packages (Go uses one Git repo per package, while Rust has a monorepo with specialized scripts to import Cargo packages and generate Debian ones). On top of that, we can’t just modify things like usual; we have to use quilt. And on top of that, I’m also a backports maintainer, so all the work (and usually even more) has to be done there also.
Now let’s pull on that pin from the earlier conversation. This is exactly the kind of scenario that a well-supervised coding LLM is most effective in. I could see a seasoned developer saving hours, maybe even days, by turning over the mundane tasks of managing trees of cascading dependencies over to a coding tool — and verifying and directing the process. (Yes, I have been using em-dashes for years; LLMs have copied people like me, not the other way around! This post was not written with any AI assistance.)
Actually, this is almost a dream scenario for a coding assistant. The result is time-consuming to formulate but easy to review, which is the opposite of the way these things often go.
I can assure you with 100% certainty that humans aren’t adding a lot of value in this process. It would be wrong to believe that a human is carefully reading every line of code in dozens of updated or new library packages. The problem set is too big, the time too short, and the code too varied and complex.
Coding agents seem to be most effective when there are strong test suites that they can test changes against. Debian builds, especially of modern packages, tend to have this property. Many packages have test suites that are run during build. And, if the package builds in an isolated environment (and especially if its downstream dependencies do also), then there is a decent chance that it’s fairly correct. Maybe needing some manual tweaking here and there, but generally a successful build is a reasonable indicator.
You can argue that it would make more sense for Debian to just include dependencies in source packages, along with some version information to support security rebuilds, and I’d tend to agree with you. But we are where we are. This would be one of the more significant leaps forward in developer productivity, but it complicates things like copyright reviews.
Where are LLMs run? What is the environmental impact?
Most of the proposals seem to make the assumption that LLMs must always run in some large, hosted datacenter. As I noted in my March article, I have had credible results on even an older GPU running on solar power.
That said, it is undeniable that LLMs are fueling a datacenter boom, and this in turn is producing a significant new demand for resources. Most notably for the global scale: electricity, which is sometimes generated using carbon-emitting technologies.
Bill McKibben, who has been a leading voice in the fight against climate change since the 1980s, has made some interesting points recently: he’s noted that solar power is the fastest kind of generation we can build, and a number of large AI companies are investing heavily in solar, even to the point of fully offsetting new datacenter’s needs. On the other hand, he’s also noted that some companies are buying inefficient and dirty gas turbines. It is decidedly a mixed bag. The heavy investment in solar can have knock-on positive effects for infrastructure. Obviously, not every picture here is rosy. This analysis doesn’t touch on the real land and water use situation, either.
On the other hand, if an LLM allows me to do in an hour what I would have done in a day, that’s a day of not heating or cooling the work area — generally not sustaining a human for the purpose of writing code for Debian. HVAC energy consumption dwarfs my GPU, and I’d imagine probably also the slice of LLM energy used.
Holistically, I would have to conclude the picture is mixed. It is possible to use LLMs in a pretty green way, and also in a pretty dirty way.
Assuming Conditions Never Change
A flaw in most of these proposals is they assume that the conditions at this present moment will always hold. In fact, that the conditions at the present moment will not continue is something both AI cheerleaders and AI skeptics agree on.
For instance:
Ed Zitron has done a ton of research into the financing side of AI, and has concluded that the current model is unsustainable and headed for a significant bubble burst. I’m not positioned to personally evaluate those claims, but if that happens, what is the result? Perhaps it is a steeply increasing cost of inference for the frontier models, slower pace of training/evolution for them, etc.
In a recent episode of Oxide and Friends, Simon Willison discussed the open weight models that are now available. They have been making remarkable strides in efficiency and capabilities, to the point where $50,000 of hardware can now run high-end open weight models with capabilities that are at least in the same ballpark as the American frontier models. This puts running high-end models locally squarely within reach of universities and small- to medium-sized businesses, with power requirements that can be met with standard commercial solar and wind installations.
The lack of nuance in the more restrictive proposals is particularly concerning. Proposal A doesn’t allow “the use or assitance of… LLMs”. So it bans my solar-powered GPU. It bans using LLMs to find security issues. It bans all sorts of things that don’t seem to be ban-worthy, alongside the things that do. And it codifies it in the very hard-to-change social contract.
That proposal, and some like it, seem to imply that all LLM output is bad. I grant you that AI slop is a real and legitimate concern, and many Open Source projects have to deal with it. On the other hand, we have all seen first-hand how the security of the Linux kernel has benefited dramatically from AI analysis. It is certain that black hats are using these tools. If we refuse to use modern security tools, our security will be compromised (and what is the environmental and social impact of THAT?)
I find the statement “Generative AI is characterized by producing output of a nature that would ordinarily be produced and consumed by humans” to be particularly interesting. The same was once said of compilers.
The Real Concerns
You might think from reading this that I am some AI cheerleader. I’m not. I share the ethics of the FLOSS movement, and have for decades. I abhor the power and lack of ethics that many big names in the field are running with at the moment. I’ve had to put up Anubis on this blog, for instance.
I have personally experienced the effects of AI slop, especially at review time. This is a real problem, though I don’t think the more draconian policies are likely to help (the looser “you must disclose” stand a fighting chance, but I’m not sure they would help, either.) Done poorly, AI threatens developer burnout by overwhelming them with poor code and verbose but useless explanations. Done well, AI can help prevent developer burnout by automating tedious and low-value tasks.
Shouldn’t our goal be that humans submit work to Debian, using tools they prefer, and take responsibility for it? Does it matter if someone uses ed, vim, emacs, or vscode? If they use LSP or just run gcc manually? I’d say we benefit from the diversity. Wouldn’t we be better off to benefit from the diversity here, and judge work as we always have: on its merits, not what tools were used to create it?
Fundamentally, a GR is a long and arduous process. It’s not easy to reverse later. Amending the Social Contract is even longer and more arduous (I should know; I may have been the first one to try). The LLM landscape is fast-moving. None of us can really predict where it will be in a year. Will the current market leading companies even still exist? Will it be at all credible to refuse to use AI-assisted security tools? What is the most effective way to deal with AI slop? What level of utility will we be able to achieve with models run locally?
Some of these proposals would make sense if drafted in some way short of a GR, which would allow more maneuverability as the landscape changes.
Brief analysis of the options
Considering the proposals:
- Proposal A: seeks to amend the social contract, which I am opposed to for reasons already laid out above. It names some real concerns about AI that I agree with, but implies that all LLM uses and models are guilty of the problems, which is not the case with all of the claims. It also sets us behind the curve on security and stability by forbidding the use or assistance of those tools, even if run by others. It requires us to ignore reports of actual security bugs, or correct fixes, if those reports were generated with the assistance of an LLM, which I find to be absolutely untenable.
- Proposal B: This is the “AI with accountability” approach. It notes the real concerns with LLMs without painting with an overbroad brush. It strikes me as level-headed and sensible.
- Proposal C: It paints with an over-broad brush and makes some non-binding requests. Then it winds up largely like proposal B, though while it is worded more strongly, has fewer binding requirements (for instance, it lacks proposal B’s prohibition on transmitting sensitive information to untrusted providers)
- Proposal D: Seems broadly similar to proposal B, an “AI with accountability” approach. I’m not really clear why we need both.
- Proposal E: Largely the status quo. It is like proposals B and D in that it says humans are accountable for their contributions. It encourages disclosure of LLM use, but does not mandate it. Like proposal B, it prohibits disclosing sensitive information to third-party AI services. Note that both proposals B and D have an appropriate nuance: a local model is fine, a third-party one is not.
- Proposal F: This seems really similar to proposal E. I’m not sure why we have these two.
- Proposal G: Disallows “the output of generative AI as direct contributions to Debian.” This is something of a weakened proposal A; it doesn’t seek to amend the social contract, nor does it ban all use; it simply bans the use as a direct contribution.
- Proposal H: Ban due to climate impacts. “How is this even an argument” is disrespectful to reasoned conversation. I have already noted that LLMs can be and are used in ways that are not climate-harming. It explicitly contains no binding requirements at all, and is effectively a rant. While I agree with the sentiment that climate change is an urgent problem, and that some LLMs are exacerbating it, I disagree with that all LLM usage does so and therefore disagree with the conclusion.
In favor of nuance
I find that black-and-white thinking is almost always something to be avoided. I see it too often. I see it in politics, I see it in our software, I see it in discussions around AI. Are there deeply unethical things happening in AI? Absolutely. Are they doing some impressive things? Also yes.
We have accepted this nuance in other areas. For instance, almost all the hardware Debian runs on has closed-source hardware, and has components manufactured or assembled in countries with some of the worst human rights records on the planet. I’m not saying this is a great state of affairs. It is something we should speak up about and act upon. But the worse state of affairs would be “no Debian because the hardware is impure”.
Log in
