After spending many hours on reading all the proposals and discussions the best choice for me is NOTA (None of the above).
We do not need to create new rules for LLM usage, we already have our DFSG and our social contract.
Keep it simple, stupid. Avoid more rules!
A couple of weekends back, I upgraded my home sever. It failed to restart after running apt dist-upgrade
The only update that was performed was to the kernel, it went from 6.12.88+deb13-amd64 to 6.12.100+deb13-amd64. I had previously performed an apt-get upgrade, and rebooted the machine, so I was pretty sure that this was to blame. This blog entry (is a late) attempt to document how I went about finding a fix for this issue so that next time I don’t need as much hand holding as I did this time around :-)
(1)
Having my machine not boot following an upgrade is pretty rare, but has happened before. Usually it is because I have done something wrong so as always confirming I haven’t broken something by accident is always my first step…
I plugged in a keyboard an monitor to the machine and watched it boot. Being a server this takes a long time (I guess because at this stage of system initialisation we want to test things sequentially)
Watching the system boot I see the usual BIOS/UEFI stages for this machine, followed by the grub menu and the the local screen showed:
Log in Loading Linux 6.12.100+deb13-amd64 ...
Loading initial ramdisk ...Nothing else. That was it. OK that looks like I have a broken system all right, and at very early stage of the boot process process.
(2)
Breaking into the grub menu and removing the quiet option yields a little more information (but not much):
Loading Linux 6.12.100+deb13-amd64 …
Loading initial ramdisk ...
EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID d
evice path
EFI stub: Measured initrd data into PCR 9and nothing else.
(3) Initial debugging
6.12.88+deb13-amd64 (During boot select Advanced options from the grub menu followed by the kernel image wanted)
EFI stub: Measured initrd data into PCR 9 apart” other than the usual rantings to “turn off secure boot” (on this server that currently isn’t turned on – bad me)(4) Triage
Start looking for where the fault first occurred. At this point I needed help, and given that Sledge was visiting I asked if he would sanity check what I was doing. His initial thoughts were that that /boot had run out of space, but replaying my step (3) with him acting as a ‘rubber duck’ showed that this was something other than PBKAC
Sledge had a quick look, then informed me that between kernel images 6.12.88+deb13 and 6.12.100+deb13 Debian stable has only had shipped .90 .94 .95 and .96 kernels. We could easily try them all:
wget each kernel package then install (dpkg -i) followed by an update-grub, checking that there was sufficient space on disks especially my small /boot partition)6.12.95+deb13 and this worked6.12.96+deb13 yielded the same lock up on boot as 6.12.100+deb13OK I now have the first kernel image that doesn’t boot on my system, time to raise a bug…
Up until now I have been walking to my garage where the server is located and standing in front of a rack
with a monitor and keyboard plugged into the machine. However this machine supports IPMI so I spent a little time getting that up and running so that I can continue from the relative comfort of my desk (with lights, a chair and not needing to hold the keyboard with one hand)
Great I can now grab screen shots from the confort of my desk (unfortunatly they are only screen shots not text files, but at least we can seen the early stage of boot, Post, grub menu and then initramfs before system log happens)
(5) Collating information for the initial bug report
Sledge had mentioned my problem in irc/#debain-kernal where iam_tj suggested that we try appending
‘debug earlycon=efifb’ to the kernal command line. This yielded 15 seconds worth of messages before the system locked up the last few messages being (vmlinuz-6.12.96+deb13-amd64):
[ 14.663477] RCU Tasks: Setting shift to 5 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=20.
[ 14.750474] RCU Tasks Rude: Setting shift to 5 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=20.
[ 14.838024] RCU Tasks Trace: Setting shift to 5 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=20.
[ 14.929752] NR_IRQS: 524544, nr_irqs: 584, preallocated irqs: 16
[ 15.016814] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[ 15.104011] Console: colour dummy device 80×25
[ 15.191236] printk: legacy console [tty0] enabled
[ 15.278249] printk: legacy bootconsole [efifb0] disabled
Booting the working kernel with the same kernel options yields the SAME messages with slightly differing times, but then continues to login prompt:
[ 14.697466] RCU Tasks: Setting shift to 5 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=20.
[ 14.784936] RCU Tasks Rude: Setting shift to 5 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=20.
[ 14.872067] RCU Tasks Trace: Setting shift to 5 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=20.
[ 14.964000] NR_IRQS: 524544, nr_irqs: 584, preallocated irqs: 16
[ 15.051482] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[ 15.226079] printk: legacy console [tty0] enabled
[ 15.313751] printk: legacy bootconsole [efifb0] disabled
[ 15.400831] ACPI: Core revision 20240827
[ 15.401415] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635855245 ns
[ 15.401464] APIC: Switch to symmetric I/O mode setup
... and so on
iam_tj also suggested adding keep_bootcon – with ‘debug earlycon=efifb keep_bootcon’ on vmlinuz-6.12.96+deb13-amd64:
We get a LOT further – and we see a crash / trace-back:
[ 34.285342] BUG: kernel NULL pointer dereference, address: 0000000000000000I raised bug #1143721 and followed it up with screen captures of the boot sequence (captured from the IPMI client) and files containing the output of dmidecode, lscpu and lspci to kive the kernel team as much information as possible:
[6.12.96+deb13-amd64 debug earlycon=efifb keep_bootcon.tar.gz (application/gzip, attachment)]
[dmidecode.txt (text/plain, attachment)]
[lscpu.txt (text/plain, attachment)]
[lspci.txt (text/plain, attachment)]
(6) Tracking down the bug Git Bisect
The problem with this type of bug is that it is hardware (class) specific, whilst the kernel doesn’t boot on my system, it clearly has worked on machines used by the kernel team, the Debian test and build infrastructure, (otherwise this kernel would never have been released) and everyone else who has upgraded to the newer kernel before I did (otherwise we would be drowning in fails to boot bug reports). Carnil’s excellent response to my bug: Message #15 (and help in IRC) provided me with a detailed step by step guide in how to track down the individual git commit that fails on my system. I had already (with Sledge’s suggestion) made a clone of the stable branch, but was struggling to follow the steps in the Debian Linux Kernel Handbook to re-build a duplicate kernel because I didn’t understand how to obtain the same configuration that Debian used to build the kernel; Carnil’s email provided me the missing steps (Highlighted).
git clone --single-branch -b linux-6.12.y https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git cd linux-stable git checkout v6.12.95 cp /boot/config-$(uname -r) .config yes '' | make localmodconfig make savedefconfig mv defconfig arch/x86/configs/my_def test 6.12.96 to ensure this is "bad" git checkout v6.12.96 make my_defconfig make -j $(nproc) bindeb-pkg … install the resulting .deb package and confirm it fails to boot and triggers the NULL pointer dereference.
Right I can now start to Bisect the problem:
git bisect start
git bisect good v6.12.95
git bisect bad v6.12.96Rather than use the half step point’s git bisect suggested I was advised in irc to jump straight to the a given commit that from the git log was suspected as the culprit:
git checkout 977855894bca4b87afa50d21e3f3e85a5a0e901f
build and install….
fails…
git bisect bad
git checkout 977855894bca4b87afa50d21e3f3e85a5a0e901f~1 ## ~1 is the commit beforehand
build and install….
fails…
git bisect good
The entire test tree can shown with git bisect log and this was submitted as an email to the bug report, we have found our smoking gun :-)
Finally I would like to thank Carnil, Iam_tj for their time patience and fantastic support in guiding me through finding this regression. Right now kernel bugs are coming in thick and fast with a lot of AI assisted bug hunting, the increased numbers of bugs mean that the kernel team are especially busy. Hopefully our paths will cross and I’ll be able to buy you some beers (or whatever) soon. thank you. Sledge also deserves thanks for putting up with me and pointing me in the right direction (as ever). Lucky for me that he lives nearby so I can provide beers on a regular basis :-)
Let me start with a hypothesis:
For human developers, using coding LLMs magnifies their difference in skill levels.
I am one that rarely thinks things are always black and white. Back in March, I wrote Artifial Intelligence: Shades of Gray. Since then, I’ve had more of a chance to experiment with LLMs myself. I also happen to work for an employer that is taking a very pragmatic approach to LLMs: teams and individuals use it as they see fit, but if they are causing considerable expense, they have to justify it.
In various settings, I have seen the egregious examples of AI slop we all know about. As I wrote in March, “I have seen it both waste more time than it saves, and save a ton of time.”
I have come to see that, as a tool, it is most valuable when it is running under the supervision of an experienced engineer. It is at its worst when it has no such supervision; the “vibe coding” and other low-quality slop we see.
A coding agent is like a junior developer or research assistant. When properly supervised, they help projects move along more quickly by letting a senior developer focus on the more difficult, less mundane aspects of the project. But one couldn’t expect a junior developer to consistently deliver high-quality code and architecture on their own.
Let’s put a pin in this idea and look at the story in Debian.
LLM use in Debian
There is a vote happening in Debian around the use of LLMs. In typical Debian fashion, there are 8 options to choose from, many of them similar. Most of these proposals acknowledge there are different types of tasks done in Debian, but the proposals don’t differentiate between them well. Let me do so here. These are some of the LLM-relevant tasks people in Debian perform:
I’m going to focus my remarks here on packaging upstream software for Debian, since this is by far the most time-consuming developer task project-wide.
It matters to our users that we get this right, and packaging quality is one of the things that sets Debian apart from other distros. Packaging things for Debian requires knowledge of some specific tools, such as debhelper, that aren’t widely used anywhere else. In most cases, it is fairly rote time-consuming work. In other words, by its design, it requires people with senior-level skills to do grunt work.
I can’t overstate how massive a burden this grunt work is. I maintain some packages for Go and Rust. By Debian policy, all of those packages’ dependencies must also exist as Debian packages, and be used to build against. When upstream adopts a newer version of some library, it can unleash cascading dependencies that can take hours to sort out. Worse, the Rust team and the Go team use entirely different ways of managing packages (Go uses one Git repo per package, while Rust has a monorepo with specialized scripts to import Cargo packages and generate Debian ones). On top of that, we can’t just modify things like usual; we have to use quilt. And on top of that, I’m also a backports maintainer, so all the work (and usually even more) has to be done there also.
Now let’s pull on that pin from the earlier conversation. This is exactly the kind of scenario that a well-supervised coding LLM is most effective in. I could see a seasoned developer saving hours, maybe even days, by turning over the mundane tasks of managing trees of cascading dependencies over to a coding tool — and verifying and directing the process. (Yes, I have been using em-dashes for years; LLMs have copied people like me, not the other way around! This post was not written with any AI assistance.)
Actually, this is almost a dream scenario for a coding assistant. The result is time-consuming to formulate but easy to review, which is the opposite of the way these things often go.
I can assure you with 100% certainty that humans aren’t adding a lot of value in this process. It would be wrong to believe that a human is carefully reading every line of code in dozens of updated or new library packages. The problem set is too big, the time too short, and the code too varied and complex.
Coding agents seem to be most effective when there are strong test suites that they can test changes against. Debian builds, especially of modern packages, tend to have this property. Many packages have test suites that are run during build. And, if the package builds in an isolated environment (and especially if its downstream dependencies do also), then there is a decent chance that it’s fairly correct. Maybe needing some manual tweaking here and there, but generally a successful build is a reasonable indicator.
You can argue that it would make more sense for Debian to just include dependencies in source packages, along with some version information to support security rebuilds, and I’d tend to agree with you. But we are where we are. This would be one of the more significant leaps forward in developer productivity, but it complicates things like copyright reviews.
Where are LLMs run? What is the environmental impact?
Most of the proposals seem to make the assumption that LLMs must always run in some large, hosted datacenter. As I noted in my March article, I have had credible results on even an older GPU running on solar power.
That said, it is undeniable that LLMs are fueling a datacenter boom, and this in turn is producing a significant new demand for resources. Most notably for the global scale: electricity, which is sometimes generated using carbon-emitting technologies.
Bill McKibben, who has been a leading voice in the fight against climate change since the 1980s, has made some interesting points recently: he’s noted that solar power is the fastest kind of generation we can build, and a number of large AI companies are investing heavily in solar, even to the point of fully offsetting new datacenter’s needs. On the other hand, he’s also noted that some companies are buying inefficient and dirty gas turbines. It is decidedly a mixed bag. The heavy investment in solar can have knock-on positive effects for infrastructure. Obviously, not every picture here is rosy. This analysis doesn’t touch on the real land and water use situation, either.
On the other hand, if an LLM allows me to do in an hour what I would have done in a day, that’s a day of not heating or cooling the work area — generally not sustaining a human for the purpose of writing code for Debian. HVAC energy consumption dwarfs my GPU, and I’d imagine probably also the slice of LLM energy used.
Holistically, I would have to conclude the picture is mixed. It is possible to use LLMs in a pretty green way, and also in a pretty dirty way.
Assuming Conditions Never Change
A flaw in most of these proposals is they assume that the conditions at this present moment will always hold. In fact, that the conditions at the present moment will not continue is something both AI cheerleaders and AI skeptics agree on.
For instance:
Ed Zitron has done a ton of research into the financing side of AI, and has concluded that the current model is unsustainable and headed for a significant bubble burst. I’m not positioned to personally evaluate those claims, but if that happens, what is the result? Perhaps it is a steeply increasing cost of inference for the frontier models, slower pace of training/evolution for them, etc.
In a recent episode of Oxide and Friends, Simon Willison discussed the open weight models that are now available. They have been making remarkable strides in efficiency and capabilities, to the point where $50,000 of hardware can now run high-end open weight models with capabilities that are at least in the same ballpark as the American frontier models. This puts running high-end models locally squarely within reach of universities and small- to medium-sized businesses, with power requirements that can be met with standard commercial solar and wind installations.
The lack of nuance in the more restrictive proposals is particularly concerning. Proposal A doesn’t allow “the use or assitance of… LLMs”. So it bans my solar-powered GPU. It bans using LLMs to find security issues. It bans all sorts of things that don’t seem to be ban-worthy, alongside the things that do. And it codifies it in the very hard-to-change social contract.
That proposal, and some like it, seem to imply that all LLM output is bad. I grant you that AI slop is a real and legitimate concern, and many Open Source projects have to deal with it. On the other hand, we have all seen first-hand how the security of the Linux kernel has benefited dramatically from AI analysis. It is certain that black hats are using these tools. If we refuse to use modern security tools, our security will be compromised (and what is the environmental and social impact of THAT?)
I find the statement “Generative AI is characterized by producing output of a nature that would ordinarily be produced and consumed by humans” to be particularly interesting. The same was once said of compilers.
The Real Concerns
You might think from reading this that I am some AI cheerleader. I’m not. I share the ethics of the FLOSS movement, and have for decades. I abhor the power and lack of ethics that many big names in the field are running with at the moment. I’ve had to put up Anubis on this blog, for instance.
I have personally experienced the effects of AI slop, especially at review time. This is a real problem, though I don’t think the more draconian policies are likely to help (the looser “you must disclose” stand a fighting chance, but I’m not sure they would help, either.) Done poorly, AI threatens developer burnout by overwhelming them with poor code and verbose but useless explanations. Done well, AI can help prevent developer burnout by automating tedious and low-value tasks.
Shouldn’t our goal be that humans submit work to Debian, using tools they prefer, and take responsibility for it? Does it matter if someone uses ed, vim, emacs, or vscode? If they use LSP or just run gcc manually? I’d say we benefit from the diversity. Wouldn’t we be better off to benefit from the diversity here, and judge work as we always have: on its merits, not what tools were used to create it?
Fundamentally, a GR is a long and arduous process. It’s not easy to reverse later. Amending the Social Contract is even longer and more arduous (I should know; I may have been the first one to try). The LLM landscape is fast-moving. None of us can really predict where it will be in a year. Will the current market leading companies even still exist? Will it be at all credible to refuse to use AI-assisted security tools? What is the most effective way to deal with AI slop? What level of utility will we be able to achieve with models run locally?
Some of these proposals would make sense if drafted in some way short of a GR, which would allow more maneuverability as the landscape changes.
Brief analysis of the options
Considering the proposals:
In favor of nuance
I find that black-and-white thinking is almost always something to be avoided. I see it too often. I see it in politics, I see it in our software, I see it in discussions around AI. Are there deeply unethical things happening in AI? Absolutely. Are they doing some impressive things? Also yes.
We have accepted this nuance in other areas. For instance, almost all the hardware Debian runs on has closed-source hardware, and has components manufactured or assembled in countries with some of the worst human rights records on the planet. I’m not saying this is a great state of affairs. It is something we should speak up about and act upon. But the worse state of affairs would be “no Debian because the hardware is impure”.
