The diffoscope maintainers are pleased to announce the release of diffoscope version 321. This version includes the following changes:

[ Chris Lamb ]
* Fix compatibility with Ocaml 5.4.1.

You find out more by visiting the project homepage.

The diffoscope maintainers are pleased to announce the release of diffoscope version 320. This version includes the following changes:

[ Chris Lamb ]
* Support androguard 4 and previous versions. Thanks, linsui!
  (Closes: #1140016)
* Use --long-form arguments when calling apktool in order to support apktool
  version 3. Thanks again to linsui. (Closes: #1140015)
* Update copyright years.

You find out more by visiting the project homepage.

Through work, I have paid license to windsurf (recently renamed to "devin"), an application for LLM-based (aka, "Agentic") development.

I hadn't been using it that much, but in an effort to more clearly understand how this whole AI development thing works, I decided to give it a closer look recently.

My conclusions:

In its current form, this whole LLM wave is problematic for multiple reasons. But ignoring that, and looking at the technology only, I can say that:

• it is a paradigm shift;
• it is, at the technological level, a positive evolution;
• and it is a threat to free software.

Problems

Lest someone (incorrectly) assume that I am arguing in favour of the current state of affairs with regards to LLMs, let me state this first.

The way LLMs are built today is highly parasitic. Websites are downloaded in whole, at unsustainable rates, regardless of the consent of the people who made the original content. The result is predictable: servers get overloaded, server administrators attempt to implement various mitigations. Some of these mitigations work; some do, for a while; some are entirely useless. In actual fact, the mitigations are an arms race -- if too many people implement the same mitigation, then the people who try to build yet another LLM so they can extract rent will just try to work around the mitigation, eventually they will succeed, and you'll just have to come up with another mitigation. It's a bit like spam; you introduce regex-based spam filters, they introduce spelling mistakes, you introduce bayesian filters, they add a large batch of markov chain-generated semi-nonsense words made invisible by markup, you add filters to block emails with such markup, they move the text into an image. We have working mitigations today, but eventually we'll run out of ideas.

LLMs glob up everything they can while ignoring the license of the source material. The people who push those LLMs claim that pushing the source material through the machine learning algorithms makes the output of the algorithm distinct enough from the source material that the license no longer applies; I'm not so sure that this is true. I guess the New York Times v OpenAI lawsuit will teach us some of the answer to that question here, but even so the ethical questions about "is it OK to bring down another server just so we can download the internet for another for-pay LLM" are still open. And regardless of what the law states, my opinion on "you're using my copyleft code to generate code under a different license" is not something you might like if you agree with the rent seekers' opinion on the subject.

That all being said and true, the technology works. You can have a "conversation" with an LLM that resembles a human one. If you pass it some data, you can use plain english to ask it questions about that data, which is a lot easier than to ask it about that in a formal way. You can request it to generate some code, and it will generate something that looks like what you need and that will be mostly correct for like 95% of the time.

Now, yes, 95% of the time is not 100% of the time, and no, you can't ask it to "write me a piece of software that implements this 300-page requirements document and get back to me when you're done", because it will fail, and you won't know where it has failed, and you'll take it into production and expect everything to be fine because it won't and this one minor logic bug will cause half your servers to spin and consume credits with your infrastructure provider with nothing to show for it.

But that doesn't mean you can't use an LLM to build a large piece of software. It just means you have to understand the LLMs limitations and strenghts, and use them correctly.

Here's what an LLM is good at:

• Generating plausible text
• Interpreting text to figure out what a plausible meaning or summary of that text is
• Giving vague indications as to what the probable context of a given body of text is.

It turns out that that's enough to use the LLM to build a reliable piece of software, provided you do it right.

Paradigm shift

An LLM can generate text by the truckful. The generated text could be code. Given a good enough LLM, the generated text might even run and do something useful.

You can try to blindly run the code, and if it doesn't run correctly, you can paste the error message to the LLM, and it can tell you what went wrong and how you could possibly fix it. This creates a feedback loop: you ask it for an amount of code, you run the code, you receive an error, you tell it that the code is problematic and give it the error message, it makes changes to the code, now you have something that at least no longer fails at startup.

If you ask it to add tests to make sure that your code acts as per your specification, now you get an error if and when the code doesn't act as per your specification. Or, well, at least not as per the part of the specification that was correctly turned into a unit test by the LLM.

LLMs have a context window, so if the error message is pasted in the same conversation as where the code was generated, it is able to reuse the earlier prompts to refine how it should interpret the error message that you received.

You can't really paste the source code of an entire application into the prompt of your LLM, that would quickly overrun its context window. But LLMs also allow you to provide some form of background information -- a document, say -- on which you ask it to reason. It will interpret that document, but doing so uses less of the LLMs context window. So providing the LLM with your application's source code as background information can help it understand better how your code interacts. This is especially helpful if you only provide the LLM the background information relevant to the actual question.

So now if you are able to:

• Create background context with your application's source code
• Have the LLM generate a first draft of your requested change, plus the tests to make sure it works
• Compile (if applicable) the generated code (and tests) and run said tests
• Return any error messages to the LLM with a request to correct the error

Then the combination of "getting it 95% right off the bat" and the above feedback loop means you can generate syntactically correct code, that probably does what you need, in minutes.

I say "probably" for a reason. There are going to be cases where you specify a request without a number of details (because they are implied), and the LLM will get most of those details right but just not implement the one bit because it's an automaton and it doesn't think. Or you will ask it to make sure that two bits of the application look exactly the same, without specifying that they must act the same, now and in the future, and it will just generate the same block of code twice and then in a future change it will change one but not the other.

But if you review the changes, and you have experience as a programmer, you will be able to spot most cases where the LLM got it wrong. And so it's possible, if not necessarily easy at first, to use an LLM to generate mostly correct code.

There are certain places where "mostly correct" code is not desireable. But equally, there are also cases where, "mostly correct" is good enough.

After all, most of the software you run today -- the bits of it that weren't, yet, generated by an LLM -- is only "mostly correct", too, because to err is human and we all make mistakes. If not, there wouldn't be any CVEs and your software would never do anything wrong.

Now, doing the feedback loop described above is certainly something you could do manually. You could open an account on one of the LLM websites, upload the source code of your application, ask it to generate some new feature, download the newly generated feature, run it, and then copy/paste any error messages back into the LLM.

But that's a lot of manual work of the type that computers are pretty good at. So that's what the "windsurf" tool helps you with: you run it inside your IDE -- either a VSCode-based tool that you download from their website which comes with their product preinstalled, or a separate JetBrains plugin that you can install. You can then open your entire relevant codebase in a workspace in your IDE. You then ask the LLM, through the IDE, to generate a new feature in your codebase, and to also generate the test while it's at it. It will use a mixture of LLM interpretation and non-LLM functionality to scoop out the relevant bits of your codebase to send to the LLM as background information, will send it your prompt, will download the generated code and patch or create files, will compile (if required) and run the newly generated code and tests, and will refine the generated code if the tests produce any errors. All mostly automatic; by default, running anything requires explicit confirmation. You can turn that off completely (probably not a good idea), or you can give it a whitelist of things that you don't want to confirm (perhaps OK), and the tool also passes standing instructions to the LLM to never generate any command that deletes a file (which, like with any LLM, can be overridden, but it requires you to be very stubborn and to use more credits than you'd probably like).

All this put together means you can build something without writing any piece of code, provided you do it right.

A technically positive evolution

Don't go and say, "here's a 300-page document, read it and write whatever the document says". It will get it wrong, it will write a massive test suite that it will only run at the end, it will choke itself up trying to interpret the massive amount of failures it encounters, it will fill up its context window and it will start to forget some of the requirements. That won't work.

But what you can do -- what I did, in fact -- is this.

First, create an empty workspace. Don't put any code in it.

Then, tell the LLM to generate a backend framework using technology X and a frontend framework using technology Y that initially only says "hello, world". Also add tests to it, and run the tests.

It will do that. You'll not get much, but it will work.

Then, ask it to add some UI elements. A login page, perhaps. A navigation bar. Small things. Most of it doesn't have to be functional -- but tests must be there for the bits that are, and have it run the tests and evaluate the results.

Rinse, repeat, until you have a working application.

Importantly, in between the steps, you should also run the application yourself and see if the change was implemented correctly. Sometimes it won't be. Sometimes there will be a subtle bug -- I at one point had a the application hang after a few minutes. Sometimes you tell it that there's a subtle bug, and it will discover it more quickly than you could, and it will fix it, and in implementing the fix it will uncover another bug, and then you have to fix that one -- the fix it came up with for the hang was to move something to an async process on the server, which caused the application to start spinning while trying to create hundreds of async jobs (this is when I realized that the hang was a deadlock due to some part of the codebase doing something that indirectly triggered itself). Sometimes it will try to fix the bug you tell it about, and you'll see that it's going off on a tangent that has nothing to do with what you're seeing. It's important to keep an eye on what it's doing, so you can guide it back on track when that happens -- when I told it about the hang, it started investigating the part of the code which sends out emails, thinking that it could hang while waiting for sendmail to finish, but the hang was happening when the application was idle, not when it was sending out emails, and only when I told it about it happening when it was idle did it find the deadlock.

So it's not a fully automatic process, and it needs to be guided by someone who knows what they're doing. But if that is the case, you can come up with something that works. I spent evenings and breaks for about a week, and I managed to create a working application which, had I written it by hand, would have taken me a few months of full-time work to come up with. And I now have a side project, fully complete and working, that I had been thinking about doing for more than a decade, but never got around to actually doing, because of all the work that would be involved and I just didn't see myself having the time for.

It's not perfect code. But it's mostly good enough, and it will perform the job it needs to. And it looks far slicker than most of the side projects I've done in the past, because in the past I would prioritize between implementing new features or making something look slick, and I would decide that the new feature was more important because it's only for me and there's only me and nobody cares if it looks good or not and I don't have three weeks to come up with something that looks better. But here, I found myself sometimes spending 10 minutes writing a prompt with instructions on making things look better. Because what's 10 minutes when you just spent an hour writing down and refining specifications for functionality and tests?

There are a number of other things in which an LLM can help a programmer.

For instance.

I received a bug report recently in a project I'm paid to maintain that I couldn't make heads or tails of. I opened the source code in my windsurf IDE, pasted the bug report in the prompt, and then requested the tool to analyze the source code and the associated logs and tell me how the described behavior could be happening. It turned out that I had overlooked something, but with the help of the tool, I found the bug in minutes.

I was trying to understand a particular part of a large codebase that I didn't really grasp very well. I loaded the codebase in the tool, and asked it to explain to me how a particular action is performed by the code. I requested specific functions and line numbers. I now have a far better understanding of how the code works, and will be able to write that patch that I've been wanting to write for years -- without using the LLM.

I have been struggling for, literally, years with understanding why another tool that I maintain was misbehaving in a particular way but only in Firefox. I opened the codebase in Firefox, explained the buggy behavior in plain English, and asked it to explain how this could be happening. It picked up some obscure corner case behavior of ffmpeg and mp4 containers that I was not aware of and that perfectly explained why things were misbehaving in the way that they were.

At the same time, there are limitations. Giving an LLM a codebase that was originally generated by an LLM (either the same one or another one) seems to work well. Giving it a codebase that was written by a human and expecting it to correctly update it seems to be more error-prone. I did one or two of those as a trial, and it is more problematic than anything.

An LLM is also not intelligent, notwithstanding the popular term of "Artificial Intelligence". On multiple occasions, I've asked it to write a test case for some code that was not set up to do so; and rather than suggesting a refactor is required, it would instead copy the code that needed to be tested and then test the copy, rather than the original. The tool has made multiple similar errors. I have sometimes people describe agentic coding as "similar to interacting with junior programmers", but that is not the case. A junior programmer will either fill in the gaps in your specifications, or ask for clarification when something seems off. The LLM will not do that; it will do what you ask, exactly that and nothing more. If you missed a corner case in your specification, then all bets are off.

I remember learning about programming language generations in college. A first-generation language is "machine code", a second-generation language is "assembler", a third-generation language is any high-level language such as C, Perl, or Pascal. I've forgotten what set a 3rd-generation language apart from a 4th-generation language. But I remember the definition they gave me for a 5th-generation language: "you tell the computer what to do, and it will do it". At the time, I thought it was ridiculous. Nobody could ever write something like that.

But it's here.

And it's a threat to free software.

A threat to free software?

Yes.

There is the obvious part where most of the well-known LLMs are non-free software. I mean, there are some "open source" LLM models. The windsurf tool that I used doesn't allow you to use them (directly), but they're there. There are also open source applications that implement what the windsurf editor does. So it's definitely possible to work like this without resorting to non-free software and non-free services, even though the non-free LLMs might be a bit ahead of the curve of the free ones. But that's not what I mean.

And there is also the obvious thing which I mentioned earlier in this post, which is that the people who try to build LLMs are doing it in unethical, disgusting ways, causing downtimes and disregarding licenses for whatever they can get their grubby hands on. Ideally we wouldn't be in that situation, and ideally this wouldn't be a problem, but we are where we are.

And there's the obvious thing where the OSI sold itself out and declared that a machine learning program can be open source even when the very things it was built from -- the training data -- is not available. That's a major issue that the free software community needs to fight against, but there's not really anything that that is a threat to free software. You just build your own, free software, LLM, and you're done.

The actual threat is in funding and developer support.

Most large businesses do not care about free-as-in-freedom software. They like the free-as-in-beer part, and they appreciate that the free-as-in-freedom bits can make the software more customizable. They are (mostly) happy to do sponsorships of the free-as-in-freedom projects that they use if that means their free-as-in-beer usage of the software gets improved.

But why would you care about all that when you can just generate the code you need, rather than interacting with an open source community that may or may not care about your business's interests?

Where to go from here

Although I think the moral and environmental issues with LLMs are real and problematic, given the experiments I did I am not convinced that the concept of interacting with a computer system in natural language and to use it to generate code is necessarily deficient. There are pitfalls, but they can be managed. It is possible to use such a system to create throwaway, proof-of-concept type "good enough" code bases. It can be used to interpret code bases and to understand bug reports.

I believe that the major issue with LLMs has to do with that saying about hammers and nails:

If all you have is a hammer, then everything looks like a nail.

LLMs are an outgrowth of machine learning, pushed by large corporations. These large corporations have a lot of money. If all you have is money, then every problem can be fixed by throwing more money at it. The initial language models were promising but not (yet) good enough, and it seemed that one way in which they could be improved was to increase the scale of the statistics: throw more hardware (and thus money) at it, and rather than improving the efficiency of the models, just scale up.

Scaling up is something that megacorporations are very good at. It's only a money problem, after all. Does that mean that "scaling up" is the only way to improve the models, though? I'm not convinced.

Some hardware, such as most modern Apple and Samsung devices, ship with accelerator hardware for machine learning algorithms. There are some models that are small enough to be able to run on these devices. I don't see why it should not be possible to create a small(er) language model that can do some useful part of the above-described use cases; if not locally, then at least on a server that one can run on-prem rather than requiring that you pay rent to one of the LLM companies.

The Software Freedom Conservancy has published an aspirational statement on machine learning-assisted programming that, I think, gets a lot right. It's not quite a definition, but it's something to keep in mind.

Perhaps that's the way forward?

More questions than answers at this point, anyway.

There has been a video blog post recently published with a review of Ubuntu Touch as an option to opt out of the Android world: https://www.youtube.com/watch?v=wTK6TS3pXgc

Thanks to @SwitchandClick for spending time on this and publishing that video. Much appreciated.

Many Issues amended in upcoming 24.04-2.0 Release

When I watched that video referenced above, I continuously thought: ah... this is fixed in the next major release of Ubuntu Touch, or: ah... this is a known issue that we have on the roadmap..., or: ah... this is done in this ways by design (so it's a feature or basic functionality)...

Let me just state, that most of the criticized aspects will be resolved in upcoming Ubuntu Touch release 24.04-2.0 (the tests in that video blog post have been run on Ubuntu Touch 24.04-1.x):

• Camera notch and rounding corners get honoured now by the UI
• Ubuntu Touch's default webbrowser (Morph Browser) has been bumped from Chromium engine v87 (Qt5 based) to v134 (Qt6 based), installing another browser should not be necessary anymore (note that the privacy level in Morph Browser is pretty high, so using other browsers could mean a loss of privacy).
• Bluetooth pairing agent got added to the bluetooth indicator
• Ubuntu Touch now supports Snaps on CLI level and in the OpenStore app
• Libertine has received fixes, but no substantial improvements. It mainly targets users who want to use their Ubuntu Touch device as desktop daily driver. Libertine-provided desktop apps UI-wise are often not usable on a phone-like device.

The full feature preview of the 24.04-2.0 release can be found here: https://ubports.com/blog/ubports-news-1/ubuntu-touch-24-04-2-0-beta-is-n...

Ubuntu Touch App Ecosystem

The app ecosystem of Ubuntu Touch is quite specific, because many apps in Ubuntu Touch have been explicitly developed for Ubuntu Touch using a widget toolkit called Lomiri.Components. However, in Ubuntu Touch we also encourage developers to provide apps written with other convergent-capable toolkits, such as QQC2-based apps or Kirigami-based apps.

One reason for the very different app ecosystem in Ubuntu Touch is that many service providers don't have Ubuntu Touch on their radar when investing in app development for their services. Some Ubuntu Touch App Developers work around this by either implementing unofficial client apps for web services (e.g. the Flow app for Deezer by Sander Klootwijk), others provide the web service via implementing a web app (will not work when offline, but at least will show up as an app in the launcher).

The overall solution for making Open-Store.io more familiar to users who migrate from Android is that commercial service providers start honouring digital sovereignty and start providing apps for Linux. Not just for the Linux desktop, but also for mobile Linux platforms. This dual use case can easily achieved with an app development that bears convergence in mind.

App Ecosystems are also a Matter of Perspective

And one more minor note: whenever I open an Android appstore or can peak over someone's shoulder using an iOS device: I always wonder: what are all these apps about??? Never heard about them.

So, familiarity really depends on perspective. And perspective depends on what you are used to. Change what you do and your perspective will follow.

Ubuntu Touch's root filesystem (rootfs) is Immutable

Only thing from that video blog post that we haven't fixed and won't do so in the midterm future is apt-get not working on the command line.

The reason for this is: the Ubuntu Touch root file system is an immutable file system and thus shall not be changed via apt-get & friends by ordinary users.

There are various discussions ongoing such as dpkg-divert'ing apt-get to a wrapper shell script that spits out an error message if rootfs is mounted read-only and someone tries to install packages the Debian/Ubuntu way. Other approaches are to mount some RAM disk over the rootfs, so apt-get can be used at runtime but changes to the system get reset at reboot.

However, it is possible to mount the root filesystem read-write and test newer package versions (as UT core developers do regularly, in fact). If you tinker with this, it is recommended to reflash your device (don't wipe user data, when you reflash!) from time to time, because adding packages or package upgrades to your rootfs may over time corrupt the integrity of the rootfs.

One reason for apt-get breaking the rootfs and thus your Ubuntu Touch development device is that the upgrade process of the rootfs image is incremental, so update tarballs sometimes contain only those parts that got changed between this and your previous upgrade (sometimes, upgrades contain a complete rootf image, depending on the interval between upgrades). If files from an incremental update tarball mix into a rootfs that got tinkered with via apt-get, you really end up on your own. Re-flashing will grab the complete rootfs tarball and wipe the whole rootfs and reinstall a fresh version of the newest rootfs image. Developers also do this in regular intervals to ensure their test device is clean again before running more/other tests.

This is somehow the featured website on https://earlyweblinks.com/ this week.

Read all about my web site here! https://earlyweblinks.com/site-of-the-week/joey-hess

Kind of reminds me of back in 1995 or so when my website would randomly end up picked by some best of the web list that I never heard of. The web is still a small place I guess.

Maybe I should join a web ring or something?

Very happy to share that a new package rspdlite arrived on CRAN today in its inaugural version 0.1.0-1. It wraps and provides the (header-only) C++20 library spdlite which its author describes (aptly) as tiny, fast, capable. Just like its bigger sibbling spdlog (which we wrapped as rcppspdlog), it is written by Gabi Melman. However, with a focus on C++20 and compile-time configuration, it is lighter, nimbler and faster. It is also still a fairly young project so changes may occur.

I have been working on this for about a month, and it is ready for use by R and C++. It contains the initial upstream release 0.1.0, and I plan to follow the upstream versioning making this first release as 0.1.0-1.

The package itself provides the headers for use from other C++ projects (i.e. mostly other packages), as well as a simple R wrapper so that logging can occur from either C++ or R. It will generally access the single logger instance in a compilation unit. So for a package built against these header it would be shared library of that package. At present we provide the basic logging level setters and getters, formatting accessors, and two (compile-time) options of a ‘null logger’ and a file-based logger. More options are availble from the C++ level, multiple logging sinks are but one example. Some examples are provided in the package as an R example and a C++ example; these are probably best examined from the sources.

The NEWS entry for this release is simply and just announces that we have a release. More details are in the ChangeLog and the GitHub repo.

Changes in version 0.1.0-1 (2025-06-08)

• Initial complete version and CRAN upload

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub. You can also sponsor my Tour de Shore 2026 ride in support of the Maywood Fine Arts Center.

The next Ubuntu Touch major release is approaching rapidly, yesterday we reached a major step in the preparation of the upcoming Ubuntu Touch 24.04-2.0 release: The branching-off (see below on what that is).

Ubuntu Touch 24.04-2.0 Beta is Now Available

Part of this development release step is the publication of the 24.04-2.0 Beta release images, for more details and information see:
https://ubports.com/blog/ubports-news-1/ubuntu-touch-24-04-2-0-beta-is-n...

And additionally, find below some background information on how we maintain various Ubuntu Touch releases in parallel via Git(Lab). In fact, the release model of Ubuntu Touch has partially been adopted from how we in Debian maintains our various Debian versions in parallel, only that in Ubuntu Touch we use Git(Lab) for maintaining the different package versions and not, like in Debian, the APT archive itself.

What does 'Branching-Off' Mean?

Last Saturday, in the UBports Q&A, I explained Ubuntu Touch's "branching-off", an aspect of the Ubuntu Touch release workflow based on Git(Lab). To make this accessible to even more people, here it comes as a write-up:

We host many Git repositories on GitLab, and our primary work is done on the main branches, which contain the bleeding-edge code. When a merge request is deemed critical for stable versions of Ubuntu Touch, we cherry-pick it into a release series branch.

Currently, we land our changes in the main branches and then cherry-pick them to the ubports/24.04.1.x branches. The 'branching off' process for the upcoming 24.04-2.x release means that our current main branches will be copied over to create new branches for this release cycle, namely ubports/24.04-2.x.

This has two major implications. First, any item that hasn't been translated by the time of the branch-off will not receive any more translation updates during the 24.04-2.x cycle. This is why it is crucial that translation work is completed before the branching-off.

Warning of Breaking Changes arriving soon in 26.04-1.x Daily Development UT Images

Second, looking ahead to the release after 24.04-2.x, we will be approaching 26.04-1.x. The OS base will change to Ubuntu 26.04 LTS, hopefully being ready for release to Ubuntu Touch users before the end of the year. We already have a list of features we want to land there. Because we plan to include various major changes, such as the switch from Mir 1 to Mir 2, new calendar and contacts backends, Qt6-based core apps and service components, etc., the likelihood of breaking changes at the beginning of the 26.04-1.x release cycle (which will become the next main branches' target) is very high.

The Ubuntu Touch 24.04-2.0 Release Schedule

The current release schedule is estimated to be:

25 May 2026 [done]
Platform stability freeze 24.04-2.x

25 May 2026 [done]
String freeze 24.04-2.x

15 June 2026 [done]
Branching-off (and unfreeze 26.04-1.x development), UT image release: 24.04-2.0 Beta

22 or 29 June 2026 [coming]
Final freeze for 24.04-2.x, UT image release: 24.04-2.0 RC

6 or 13 July 2026 [coming]
Release version 24.04-2.0

In 2008, I landed my second job, in the network team at Orange Portails¹, the division behind the websites and search engine of the French telecom operator Orange. The place ran like clockwork: a comprehensive technical setup, a dedicated team for every part of the business, and room to focus on what I do best. A few years later, none of that mattered: thanks to an obsession with the numbers, we could no longer deliver new services on time.

The first years

During my first years, the department operated like a startup. Its cradle was the French company Echo. They built a search engine. France Télécom bought it and renamed it Voila. It was the most visited search engine in France in the early 2000s. France Télécom consolidated the portal activities into the Wanadoo Portails division, later renamed Orange Portails.

The technical environment was excellent. We had many internal tools:³ a ticket system, an RRD-based graphing tool, an IPAM, a reporting tool, and an SNMP-based alerting tool.⁴ We deployed our Linux servers with CFEngine. We installed systems and applications from internal Debian repositories. We documented everything in a private MediaWiki instance. Supervision was performed with an ancestor of Xymon. The network architecture was clean and scalable with little legacy. We onboarded new people in a day.

It was a nurturing environment for me. I developed several tools: lldpd, an 802.1AB implementation, Snimpy, a pythonic binding for Net-SNMP, Wiremaps, a layer-2 discovery tool with a time machine to know which device is connected where, Kitérő, a tool to simulate network conditions, QCSS-3, a controller for load-balancers, and ipoo, a service available through a Jabber chatbot and a Greasemonkey script to expose IP-related information. I added SNMP support for Keepalived and Quagga. I also started this blog, with articles like “Anycast DNS,” TLS-related articles like “TLS computational DoS mitigation,” SNMP-related articles like “Integration of Net-SNMP into an event loop,” Linux-related articles like “Tuning Linux IPv4 route cache,” and an article about VXLAN long before it was cool.

The collapse

When we needed new servers, the on-site team would take a set from the inventory, install our base Linux distribution on them, put them in the datacenter, and cable them to the top-of-the-rack switches. We opened a ticket describing the servers we needed, and one week later, our servers were available. 💫

Orange wanted to know if this team was performing well, so they asked for KPIs. They decided to use the number of tickets completed in a year. They asked to double this number. So instead of one ticket for a new service, we would open six tickets—one per server. By the end of the year, the KPIs had more than doubled.

Everybody saw it as a success for performance management. So, they asked to do the same for the next year. Now, we needed to open a ticket per server and per step. Again, the KPIs doubled. Behind the scenes, the tickets went to different people and were no longer handled in order. So, for the next year, it was decided to have meta-tickets and meetings to follow the progress of these tickets. Of course, all these extra steps pushed the KPI even higher.

This performance management method spread to the other teams.⁵ Everything became slower. Instead of a couple of weeks, a new service now took six months. We built a Soviet nail factory. But the KPIs were good, and we stopped caring.

Let me give you another example. We had to estimate the impact of each night operation. We weren’t half bad: we declared most operations “without any expected impact.” Most of the time, there was no impact. One time out of five, there was a 5-second impact. We were told to try harder to meet our expected impact. What did we do? We started declaring a 5-second expected impact. One day, we got a 30-second impact and were told we failed to match the expected impact. In the end, we declared most operations with a 10-minute expected impact, and we stopped caring: instead of carefully shifting traffic around, we allowed ourselves a 5-minute impact. And our KPIs were never better.

An artist's rendering of the evolution of impacts over the years.

KPIs are not bad, but they are easy to break. Use them carefully: let the people doing the work help choose the metrics, and tie those metrics to the quality of the service—for example, with service level objectives. Otherwise, even dedicated people stop caring, game the system, and eventually quit. 📊

I have proposed the deletion of an obsolete script, but it makes me feel complicated feelings so I’m going to try and express those. This particular script was written in 2014, but the concept goes back much further – before git was invented.

When I started university in 2003, I seem to remember the computing society used to run tutorials for first-year students on how to use Apache Subversion for your group project – a vast upgrade on CVS (or worse, no version control at all). Back then, the idea of viewing your changesets in a web browser was relatively new – while it was possible to look at an SVN repository through a web UI, features were limited unless you installed something compicated like Trac.

Figure 1: Data flow when distributing commits via a mailing list

Perhaps because reading email on your desktop computer (I don’t think I could afford an IBM ThinkPad?) was the only vaguely real-time notification system available at the time (except I guess SMS, which cost 10p per text), a common pattern seemed to be to use a post-commit hook to send every single commit to a mailing list, named something like ‘foo-commits’. Indeed, for a long time Fedora had an scm-commits list which appears to be a topic of recent discussion.

I can’t really explain why people wanted to have every commit sent to a mailing list except as a way of getting notified of activity – I can’t believe people would import raw patches from those lists, ala LKML, rather than run actual version control commands to fetch the new source directly. Maybe you’d have to go back to NNTP for this.

I do like the vendor-neutrality of the “everything-as-text” approach, building on the open ecosystem of SMTP. But I doubt we’d see a widespread resurgence of commit lists now – most code hosting must allow anyone to subscribe to email notifications, I assume, and I don’t see a huge benefit in a mailing list archive of commit messages.

In the case of seL4, I’m even more confused about why this script was committed in 2014, shortly after the kernel was put on GitHub. I can only assume it was imported from previous infrastructure. I do know that the implementation is quite Python 2 heavy, with the conversion between unicode and bytes featuring heavily. So rather than risk breaking its logic with patching, I think it’s time to “thank it for its service” and let go.

Debian Contributions: 2026-05

Contributing to Debian is part of Freexian’s mission. This article covers the latest achievements of Freexian and their collaborators. All of this is made possible by organizations subscribing to our Long Term Support contracts and consulting services.

Go default compatibility, by Helmut Grohne

At the MiniDebConf Hamburg, Andrew Lee had prepared a talk on how Debian accidentally chooses Go compatibility. Helmut joined Tobias Quathammer and Andrew Lee in looking into the problem. Go has a compatibility system where modules declare a desired Go version to be compatible with. This influences various features such as whether RSA keys smaller than 1024 bits are accepted. Unfortunately, Debian’s way of building Go packages is unique in setting GO111MODULE=off, which practically implies a very old compatibility version that enables a number of insecure settings. Most Linux distributions use the default GO111MODULE=on and therefore consult a go.mod file that often declares a sensible version. While doing so is the way for Debian longer term, getting there involves major changes so we also sought a more short term workaround. We developed a patch to the Go compiler that would enable it to pick up a compatibility version from the environment. Tobias uploaded it to unstable. The next step is communicating the declared compatibility version from go.mod to the compiler via the new variable. Then, rebuilding the archive resolves the immediate symptoms. This does not save us from having to perform the larger transition to GO111MODULE=on, but this shortcut can be backported to trixie.

Trimming build-essential, by Helmut Grohne

One of the harder problems of the architecture cross bootstrap is correctly expressing the Build-Depends of glib during the toolchain bootstrap. It implicitly depends on build-essential, which happens to depend on libc6-dev. This poses a cycle. It applies even for cross building, because it is interpreted for the host architecture and that there is no way of satisfying this dependency during the toolchain bootstrap.

Given discussions at MiniDebConf Hamburg with Jochen Sprickerhof and others, a seemingly stupid idea evolved: Let’s delete build-essential. What looks insane on the surface might deserve a second look. Given how we moved away from C, C++ and autotools, what is in build-essential no longer is required by much of the archive. With the rise of debputy, debian/rules no longer has to be a makefile. While the task would be huge, those packages relevant to architecture bootstrap could explicitly support building without the implied dependency making their dependencies explicit. In a number of cases, this amounts to issuing a dependency on g++-for-host. This dependency requires the use of architecture-prefixed tools. Therefore, Helmut wrote a debhelper change that makes it always pass build tools to various build systems. This also enables more packages to honour environment variables such as CC and CXX.

Python upstream engagement, by Stefano Rivera

Stefano attended PyCon US (at personal expense) to improve upstream relations and ensure Debian’s voice is heard where it needs to be. On Friday there was a packaging summit (notes) with good discussion on the future of the wheel format, and some discussion of the new abi3t shared library format for free-threaded python.

In preparation for the event, Stefano did a complete review of the current patch stack.

Stefano’s primary goal was to get some of Debian’s patches merged during the sprints, and results were mixed. Some trivial patches (e.g. GH-150098, made progress and merged, but the most consequential patch Debian is carrying is still blocked. Stefano will continue to try to drive progress on this.

Miscellaneous contributions

• Carles worked on po-debconf-manager: Reviewed Catalan translations for 6 packages, submitted 10 packages to maintainers, and removed 3 packages from po-debconf-manager.
• Carles worked on check-relations: Continued improving the backend, including importing source package build dependencies to better support analysis of Debian blends. Added support for ignoring packages using regular expressions and source package names in response to user feedback. Used the tool to report 5 new bugs and followed up on previously reported issues.
• Helmut sent a cross build patch on behalf of a customer.
• Helmut uploaded debvm and guess_concurrency both featuring improved reproducibility and documentation.
• Helmut continued maintaining rebootstrap and made it correctly handle binNMUs of gcc-defaults. Additionally, he poked at existing gcc patches giving answers, rebasing or closing them.
• Helmut supported the video team in Hamburg mixing audio.
• Helmut continued to report undeclared file conflicts of various kinds and corresponded with maintainers about them.
• Antonio attended a debate during the Brazil Internet Forum about the impacts of the child protection regulation (ECA Digital) on free software operating systems.
• Antonio worked on Debian CI to improve the system transparency for users. This included listing any pending jobs explicitly in the job lists for each package/architecture/suite page, as well as adding a queue status page that users can check for an estimate of test latency.
• Antonio worked on several Debian CI maintenance tasks, including but not limited to some monitoring improvements, replacing usage of fonts-font-awesome with fonts-fork-awesome, and adding the ability in debci to configure a global notice (which is being used in Debian CI to point to the system status pages).
• Antonio started doing some tests related to the change of default Debian CI backend from lxc to incus-lxc. This helped identify an omission in the creation of incus-lxc images. It was missing dpkg-dev, which caused a few packages that assumed its presence to fail. In the end, the incus-lxc backend will be fixed to include dpkg-dev by default in the image, but that uncovered an undeclared dependency in gem2deb (Ruby packaging helper) and in ruby-byebug, both already fixed in unstable.
• Stefano did some minimal work on debian-reimbursements to get it working with current versions of django-allauth.
• May included the discovery of several high-severity Linux kernel root exploits. Stefano updated kernels and rebooted debian.social infrastructure several times.
• Stefano supported the Hamburg miniDebConf’s wafer website during the event, and set up an instance for the 2027 edition too.
• Stefano supported the bursary team issuing bursaries for DebConf 26.
• Stefano uploaded routine updates of python-pip, pystemmer, snowball-data, snowball (making up a mini, uncoordinated snowball transition), python-authlib, python-discovery, python-installer, python-mitogen, python-pipx, python-cachecontrol, platformdirs, and python-virtualenv.
• Stefano fixed a small number of bugs in dh-python, culminating in the 7.20260524 upload.
• Thorsten finally managed to upload a new upstream version of hplip. He also uploaded a new upstream version of epson-inkjet-printer-escpr. Last but not least with the help of other contributors he could fix bugs in lprng.
• Lucas and Santiago contributed significantly to the DebConf 26 Content team; helping to organize the team, review and rate talk proposals.
• Lucas also supported a packaging sprint held in India by rebuilding and publishing the latest results of the Ruby 3.4 transition effort.
• Santiago continued contributing to the efforts to organize DebConf 26, especially supporting the local team with different tasks.
• In collaboration with Emmanuel Arias, Santiago is mentoring Aryan Karamtoth, a GSoC participant that is working to introduce linux live-patching support in Debian. The GSoC project started in May, with community bonding and coding. Santiago reviewed a merge request to prepare the clang-extract package for debian. clang-extract is one of the building blocks that will help to extract specific functions from large C code, so only relevant code can be patched, without recompiling the whole original basecode.
• Anupa assisted Jean-Pierre Giraud with the point release announcements for Debian 13.5 and Debian 12.14.
• Colin backported various security fixes from OpenSSH 10.3 to all supported releases (including LTS and ELTS).
• Colin backported IP quality-of-service fixes to OpenSSH in trixie. The situation there had been unsatisfactory for some time, and upstream reworked their QoS support in OpenSSH 10.1 in a way that typically produces much better results.
• Colin imported new upstream versions of 26 Python packages, and fixed around 25 RC bugs for the Python team.

Quick note to share that rbenchmark is back on CRAN! The rbenchmark package makes it easy to benchmark (and compare) simple R expressions.

This package has been on CRAN for many years. At one point fourteen years ago it appeared to be rudderless so I offered help but things realigned. Now it was just tossed off CRAN, taking a number of packages depending on it with it (as shown in this CRANberries skeet listing a set of removed packages) so I offered again to help, and CRAN agreed. So here we are.

So far I just made a number of small ‘editing’ changes, added CI support, and enable dbsr-universe coverage . I do not expect to change the package materially. So far the package has no NEWS file either so maybe glance at the ChangeLog at the git repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub. You can also sponsor my Tour de Shore 2026 ride in support of the Maywood Fine Arts Center.

Today we went to see a theater play in UNAM’s Cultural Center, very near our home. No, not inside any of the theaters — in the square just between Sala Nezahualcóyotl, Foro Sor Juana and Sala Carlos Chávez.. So, yes, not only we had fun, but we had fun for free!

UNAM’s El Carro de Comedias is an itinerant theater company that often presents in this same spot (but you can see the stage is foldable, and they do have presentations elsewhere, of this same play even). I went with my family, and we enjoyed a very fun adaptation of this great play (written by teenager Alfred Jarry in 1894). One of those plays that could be inspired any day by current geopolitical events…

I know most of the people that happen to stumble upon my blog are not in Mexico City. But if you happen to be here, do consider going to their function. Check their schedule; being it an itinerating show, they can also be found at other places, but they are scheduled at the same place we saw them, every Saturday and Sunday until June 28, 11:00AM. They mentioned they will likely continue during August, but AFAICT it is not confirmed (or, at least, announced) yet.

Some pics, shot randomly by me throughout the play:

First Light box

My youngest daughter and I recently started playing the tabletop game HeroQuest. Specifically, the recently-issued, cut-down variant HeroQuest: First Light. This is quite advanced for her age, and I'm a little surprised she's taken to it, but she's really loving it, It's pushed her to read bits of lore on cards and quest books that is way above her expected reading level, and we've been exercising her maths by adding up the gold we find on our quests and calculating what the heroes can buy with it in the store afterwards.

Originally from 1989, Hasbro re-issued HeroQuest in 2020. I read about it at the time but didn't buy it. I wasn't sure who I would play it with. It also seemed expensive to me. It probably wasn't unusually expensive in 2020, nor now, for the sheer volume of finely-sculpted miniatures included. I also knew I had the original game in the loft, and I wasn't that keen on buying something I already had, although untangling the contents from several similar boxed games would take me many hours, and I wasn't sure how much of the game I would find.

mix of old and new

First Light was compelling because it is much, much cheaper than the full remake, so I was happy to take a punt. It's cheaper because it doesn't have any plastic monsters or furniture: instead cardboard cut-outs that stand up on plastic stands. For us, that is a significant drawback: 3D miniatures are much more immersive, But I can re-use the plastic miniatures I can find from the original game. First Light has a newly written adventure, better suited to beginners than the original game.

The re-issue(s) have new art and new model sculpts that look fantastic. They've changed anything which tied into Games Workshop's IP and I'm really happy about that. They've made an effort to add women, almost entirely absent from the original. I'm certain my daughter wouldn't have tried it otherwise.

For almost two decades, the PackageKit package management abstraction layer has shipped with pkcon as its command-line client. pkcon does its job, but it was always kind of a “testing” front-end for the PackageKit daemon rather than a tool designed for everyday use. The focus has instead been on the GUI tools, automatic system updates, GUI application managers and other front-ends. Its command names mirror the D-Bus API almost one-to-one (get-details, get-updates, get-depends), output is very plain, and there is no machine-readable mode for scripting. Most importantly though, there has been no development on it at all for almost a decade, so pkcon was stuck in its rudimentary state from that era.

Since a lot of changes will be coming to PackageKit, and testing the daemon and working with it from the command-line was not very pleasant anymore in 2025/2026, I decided to modernize the tool as part of my work as fellow for the Sovereign Tech Agency last year. pkgcli is the new command-line client for PackageKit. It is built from the ground up to be pleasant to use interactively and easy to drive from scripts.

Why a new tool?

Of course, instead of introducing a new tool, I could have just expanded pkcon instead. The problem with that approach is that the pkcon utility has been around for so long and its command-line API had ossified so much, that rather than changing it and potentially breaking a lot of scripts relying on its quirks, I decided to introduce a new tool instead. pkcon can still be optionally compiled for people who need it in their scripts and workflows.

The goals for pkgcli, and the features it now has are:

• Human-friendly command names. Verbs that read the way you’d describe the task, instead of mirroring the D-Bus API 1:1: show, search, list-updates, what-provides, instead of get-details and friends.
• Readable, colored output by default (still respecting NO_COLOR and degrading gracefully).
• A real scripting mode. A global --json flag emits JSONL instead of fully human-readable output when possible, to make it easier to use the tool for scripting purposes.
• Sensible defaults. A few defaults have been changed, such as the metadata cache-age, or automatic cleanup of unused dependencies being enabled by default. This is more in line with current defaults by other tools and frontends. We also print package information in a slightly different, more readable way.
• Better handling of internationalized text. Text should now align properly in the terminal window, and we should no longer have completely chaotic text output on non-English locales (especially Chinese/Japanese).

Why not pkgctl?

Originally, this tool was called pkgctl, to match other common cross-distro tool names. However, that name was already taken by an Arch-specific distro development tool. When this issue was raised, we decided to just rename our tool to pkgcli with the next release, to avoid the name clash on Arch Linux.

Examples!

Here are some examples on how to use the new tool (some of which include the abridged output pkgcli prints).

Search for anything containing the string “editor” in name or description, then look at the details of one result:

$ pkgcli search editor
Querying                  [████████████████████████████████████████] 100%
▣ ace-of-penguins 1.5~rc2-7.amd64 [debian-testing-main]
▣ acorn-fdisk 3.0.6-14.amd64 [debian-testing-main]
▣ ardour 1:9.2.0+ds-1.amd64 [debian-testing-main]
✔ audacity 3.7.7+dfsg-1.amd64 [manual:debian-testing-main]
✔ audacity-data 3.7.7+dfsg-1.all [auto:debian-testing-main]
▣ augeas-tools 1.14.1-1.1.amd64 [debian-testing-main]
▣ emacs 1:30.2+1-3.all [debian-testing-main]
▣ gedit 48.1-9+b1.amd64 [debian-testing-main]
▣ gedit-common 48.1-9.all [debian-testing-main]
▣ gedit-dev 48.1-9+b1.amd64 [debian-testing-main]
[...]

$ pkgcli show nano
Package: nano
Version: 9.0-1
Summary: small, friendly text editor inspired by Pico
Description: GNU nano is an easy-to-use text editor originally designed as
 a replacement for Pico, the ncurses-based editor from the non-free mailer
 package Pine.
[...]
URL: https://www.nano-editor.org/
Group: publishing
Installed Size: 2.9 MB
Download Size: 646.0 KB

Search only within package names rather than descriptions:

$ pkgcli search name python3

Check for updates. refresh updates the metadata, then list-updates reports what’s available:

$ pkgcli refresh && pkgcli list-updates
Loading cache            [████████████████████████████████████████] 100%
▲ cme 1.048-1.all [debian-testing-main]
▲ gir1.2-gdm-1.0 50.1-2.amd64 [debian-testing-main]
▲ imagemagick 8:7.1.2.24+dfsg1-1.amd64 [debian-testing-main]
▲ imagemagick-7-common 8:7.1.2.24+dfsg1-1.all [debian-testing-main]
▲ imagemagick-7.q16 8:7.1.2.24+dfsg1-1.amd64 [debian-testing-main]
▲ libdlrestrictions1 0.22.0.amd64 [debian-testing-main]
▲ libfftw3-bin 3.3.11-1.amd64 [debian-testing-main]
▲ libfftw3-dev 3.3.11-1.amd64 [debian-testing-main]

Explore relationships between packages:

$ pkgcli list-depends inkscape  # list what inkscape depends on
$ pkgcli list-requiring libappstream5  # list what requires libappstream5

Find the package that provides a capability, here the AV1 GStreamer decoder:

$ pkgcli what-provides "gstreamer1(decoder-video/x-av1)"
✔ gstreamer1.0-plugins-bad 1.28.3-1.amd64 [auto:debian-testing-main]

You can also have JSON output for most commands! Attach --json to any query and pipe the result straight into jq. Each line is a self-contained JSON object:

$ pkgcli --json list-updates | jq -r '.name'
cme
gir1.2-gdm-1.0
imagemagick
imagemagick-7-common
imagemagick-7.q16
libdlrestrictions1
libfftw3-bin
libfftw3-dev
libfftw3-double3

Try it

pkgcli is built by default alongside the rest of PackageKit since PackageKit 1.3.4. If your distribution ships a recent enough PackageKit, it should already be on your PATH. You can read its man page man pkgcli for more information. Feedback, bug reports, and patches are very welcome.

In the process of preparing a major Ubuntu Touch release (v24.04-2.0, coming soon...) we will also update Ayatana Indicators in Ubuntu Touch.

Last week various new features have been added to some of the indicators (toggle switch to keep the display switched on permanently, blue tooth pairing agent, redesign of the keyboard indicator, etc.) and those changes require translation updates.

If you can, please visit [1] this weekend and help translating Ayatana Indicators into your native language. Thanks so much!!!

light+love
Mike

[1] https://hosted.weblate.org/projects/ayatana-indicators/

I have been looking at seL4 some more recently, and had a small patch merged today to remove a legacy Python module from a helper script. (I was trying to run the script on a system without that module installed, and it was almost easier to patch it out.)

However, the more I think about this code and how it’s used, the more it seems wrong on at least five other levels.

The patch itself is quite uninteresting; this script was importing the past module (part of future?) to use the xrange function. Python 2 used to have separate xrange and range functions, where range returned a list in memory while xrange generated an iterator. Because this seL4 script is iterating over a large range of values, it’s important the list is not generated in-memory. But Python 3 removed the xrange function and just has range return an object, so it’s trivial to avoid the module import.

Having thought carefully some more about the specific line, there’s surely an off-by-one error in it - range iterates over 0 to n-1, so this line shouldn’t be subtracting one if it’s looking to test all 32-bit values:

    for i in range(2**32-1):

But then again, this is being used for a ‘sanity check’ of a magic bit shift algorithm that speeds up division operations to convert CPU ticks to microseconds on 32-bit arm platforms. Surely if the algorithm’s good, it shouldn’t be necessary to validate it exhaustively against every possible 32-bit value?

Also, 32 bits isn’t enough, because this is 64-bit division. include/api/types.h shows that ticks_t is always a uint64_t, so if this were a proof by exhaustion it should run to 2**64 (though that would take infeasibly long).

As discussed in issue #1352, lots of people have been running this code with the wrong divisor anyway. But because the bit shift path is only used on 32-bit platforms, it’s not clear to me that there’s even any point in specifying CLK_SHIFT/MAGIC on platforms which are 64-bit only (e.g. the tx2 port).

And to follow this rabbit hole to the very end, in comments on PR #1435 and issue #1509 it’s clear that the future of this code is to remove it, as it’s 1. unnecessarily clever (on 64-bit platforms the equivalent code just uses a division, so performance can’t be that important), and 2. the entire concept of converting to microseconds breaks the seL4 principle of not abstracting away details of the hardware.

So this has left me unclear on whether my small patch was a good thing or not, but I certainly learnt something about this corner of seL4 timer handling. And I’ve ordered a copy of “Hacker’s Delight” on the recommendation of a code comment.

The diffoscope maintainers are pleased to announce the release of diffoscope version 319. This version includes the following changes:

[ Jochen Sprickerhof ]
* Improve header detection for Sphinx documentation projects.

You find out more by visiting the project homepage.

I’ve been asked a few times if it would be possible to use virtnbdbackup as some kind of “replication” utility, to keep cold standby virtual machines on other libvirt hosts.

Usually i would tell to use underlying filesystem features (such as zfs send/recv, with incremental snapshots) to keep cold, standby copies on other hosts.

As for qcow based virtual machines, using the dirty bitmaps is not only a valid feature to create backups, but to (incrementally) replicate virtual machines, too.

I’ve released vmsync. A small golang utility that implements a simple replication tool using the NBD protocol to sync virtual machines to other hosts.

Some of you might have noticed that the recent (or rather: previous) version of libayatana-appindicator (v0.5.94) notified users and developers of the library being deprecated.

This short post is to notify you, that with today's libayatana-appindicator v0.6.0 release [1] this deprecation warning has now been removed again. Another new feature (added to AppIndicator without ABI breakage) is tooltip support. The new package version has just been uploaded to Debian experimental. Please test if your application (if it gets linked against libayatana-appindicator) continues to work flawlessly. Thanks!

libayatana-appindicator will receive continued support until GTK-3 becomes end-of-life (because libayatana-appindicator has a baked-in GTK-3 dependency which should not be ported to GTK-4 imho). That said, in the future, GTK-3 applications can continue using libayatana-appindicator for sending AppIndicator-like icons and menus over DBus to KStatusNotifierItem-based system tray renderers.

If you are looking for an AppIndicator implementation for GTK-4 applications (or other), I'd like to encourage you to help making libayatana-appindicator-glib [2] a new standard (can be used in GTK and Qt applications alike, implementation is using pure Glib-2.0). Currently, there is only one renderer (ayatana-indicator-application), so more work needs to be done on the renderers' side. (One of the next work items here is to get AppIndicator-Glib support working in Lomiri's desktop/windowed mode).

[1] https://github.com/AyatanaIndicators/libayatana-appindicator/releases/ta...
[2] https://github.com/AyatanaIndicators/libayatana-appindicator-glib/

My Debian contributions this month were all sponsored by Freexian.

You can also support my work directly via Liberapay or GitHub Sponsors.

OpenSSH

I backported various security fixes from 10.3 to trixie, bookworm, bullseye, buster, and stretch. For trixie, I also backported several IPQoS fixes to line up with upstream’s traffic management settings and drop a rather hacky Debian-specific patch; this needed a quick follow-up fix.

I upgraded trixie-backports to 10.3.

I fixed openssh uses pidof but does not depend on procps.

PuTTY

I upgraded from 0.83 to 0.84.

Python packaging

New upstream versions:

• bitstruct
• ormar
• pdm (fixing a build failure)
• pydantic
• pydantic-core
• pydantic-settings
• pyglet (fixing a build failure)
• python-asyncssh
• python-bitarray
• python-btrees
• python-build
• python-certifi
• python-charset-normalizer (fixing a build failure)
• python-fakeredis (contributed supporting fix upstream)
• python-holidays
• python-jsonschema-path
• python-memray (fixing a build failure and CVE-2026-32722)
• python-openapi-schema-validator
• python-pathable
• python-persistent
• python-pyftpdlib
• python-pytest-run-parallel
• sorl-thumbnail
• twisted
• zope.interface
• zope.proxy

Other build/test failures:

• beets
• buildbot (contributed upstream)
• dep-logic (contributed upstream)
• diskcache
• khard
• matplotlib
• mkdocs-rss-plugin
• ormar: compatibility with fastapi 0.125 and pydantic 2.13
• pgzero
• py7zr
• pydantic-extra-types (contributed upstream)
• pydata-sphinx-theme
• python-invocations (contributed upstream)
• python-localzone
• python-maturin
• python-nacl
• python-pampy
• python-treq (contributed upstream, including fixing some CI bitrot)
• python-txrequests (contributed upstream)

Other bugs:

• buildbot: (Build-)depends on deprecated module python3-pkg-resources (contributed upstream)
• pysodium: Depends on cruft package libsodium
• python-fakeredis: lua support not working, breaking django-redis cache locking
• python3.14: Drop libnsl-dev build-dependency

I updated python-treq upstream to stop vendoring multipart, now that the packaging issues with that have been sorted out.

Code reviews

• debmirror: User-Agent blocked by Ubuntu/Launchpad repositories (uploaded, and cherry-picked into trixie)
• pydantic: Fix CVE-2024-3772 in bookworm (merged and uploaded)
• pyodbc: Run SQLite tests (merged and uploaded)
• python-jsonschema-path: Transition to starlette 1.0 (merged and uploaded)
• python-maison: FTBFS with the nocheck build profile (followed up to fix the nodoc build profile as well)
• python-openapi-core: Transition to starlette 1.0
• python-openapi-schema-validator: Transition to starlette 1.0 (merged and uploaded)
• python-openapi-spec-validator: Transition to starlette 1.0 (merged and uploaded)
• python-pathable: Transition to starlette 1.0 (merged and uploaded)
• python-rich-argparse: New upstream version 1.8.0 (merged and uploaded)

Other bits and pieces

I contributed a debian-policy patch to fix several links related to build profiles.

AI slop is invading the web. A recent story about disallowing LLM-generated submissions on Lobsters triggered a lot of debate. My personal worst offenders are LinkedIn articles with AI-generated images and uninspired articles filled with emojis from people trying to masquerade as experts on a subject they don’t care enough to write themselves. While I am unhappy about this situation, I rely on LLMs for grammar, copyediting, and translation. I don’t see this as a contradiction.

I am a native French speaker, but I blog in both English and French. When I started writing this blog in 2011, I was composing in French and translating to English, but I found it was better to work in the reverse order to avoid unnatural and non-idiomatic constructions. One of my goals is to write “good” English but I never felt it was my strong point.¹ For example, verb tenses are often an issue, even if I mostly stick with the present tense. I learn the rules and forget them right away. I also don’t feel like hiring an editor for something I see as a hobby.

As an example, I have kept the history of the successive iterations when writing “Scaling Akvorado BMP RIB with sharding”:

1. the first draft, authored with the help of a thesaurus,²
2. the edited copy revised by the copyediting skill,
3. the translation to French generated with the translation skill, and
4. the human proofread of the French translation, with minor edits to the English version.

I know that LLMs may alter the author’s voice when editing, but the corrections in the second step are minor. The prompt asks to “apply light stylistic edits,” with some guidance around avoiding passive voice, long sentences, bland verbs, and filler words. It also defines the target audience: technical with a B2 level in English.

In the following excerpt, I used “long time” instead of “long-standing.” The former is missing a hyphen and applies to people—a long-time friend, while the later relates to a situation—a long-standing agreement. I had a hard time understanding the reason of the second change: the LLM prefers a defining relative clause to provide the definition of “RIB sharding.”

As the Internet routing table contains more than 1 million routes, Akvorado needs to scale to tens of millions of routes. This has been a long time long-standing challenge, but I expect this issue is now fixed by using RIB sharding, a method to split that splits the routing database into several parts to enable concurrent updates.

In the next modification, the LLM puts “device” instead of “equipment.” This is correct as “equipment” is an uncountable noun. I know that, but I still fall into this trap.

When Akvorado does not find a route from a specific device, it falls back to a route sent by another equipment device.

I ask the LLM to use “descriptive verbs” and it complies by replacing a multi-word predicate with a lexically rich verb:

The benchmarks demonstrate it has better performance than outperforms other packages, both packages for lookups, insertions, and memory usage.

It also fixes grammar errors. In the next excerpt, a “list of routes” is a singular expression. Moreover, “stored” is a state and I should not use “into” as it expresses a change.

The list of routes for each prefix are is not stored directly into in the prefix tree.

As a last example, consider the following snippet. The “require” verb accepts a noun or an object followed by a to-infinitive. I can’t use it with just a to-infinitive.

An alternative would be to have one prefix tree for each peer but it would require to configure configuring all routers to export their routes.

As someone who didn’t grow up speaking English, I struggle with these grammar rules despite reading a lot of English material.³ French is more complex to get started but more systematic. English is full of irregularities.

On each page, I disclose in the footer whether an AI modified the content. There are three levels:

• 🧠: no AI or almost no AI (e.g., grammar corrections)
• ✨: enhanced (e.g., copyediting)
• 🤖: generated (e.g., translated from another language, even if human-edited)

Hover or tap the icon to reveal the AI’s name and its role in the document.

Example of AI usage disclosure: Claude Sonnet 4.5 edited this article.

The graph below shows which tool altered each post, year by year. Recently, I applied the grammar skill to past articles. Since 2018, French articles have been translated with the help of DeepL first, then of an LLM. Since 2024, English articles are copyedited.

If you are strongly against any usage of LLMs specifically for writing, I hope you accept my more nuanced position on the usage of these tools as a trade-off to provide clearer and more engaging articles. Years of literature on improving English told us it is important to choose the right word to keep the reader engaged.

[…] Good writing consists of mastering the fundamentals (vocabulary, grammar, the elements of style) and then filling the third level of your toolbox with the right instruments.

― Stephen King, On Writing

Official announcement

European Voxit community strengthens digital sovereignty: shared codebase completed.

Read the official announcement at:
https://www.voxit.org/european-voxit-community-strengthens-digital-sover...

The Voxit community and platform development

The Voxit participation platform is originally based on the open source Polis platform developed by The Computational Democracy Project in the United States, but since its establishment in autumn 2025, the European Voxit community has been developing an independent solution, adapted to European needs.

The aim is to create an open source, interoperable and scalable participation infrastructure suited to Europe’s regulatory environment and aligned with democratic values. Through this development work, Voxit is becoming a clearly distinct fork of the original Polis platform – allowing Europe to develop participatory infrastructure at its own pace and according to its own governance needs, while the original Polis project continues to break new ground. This enables Europe to build its own open and trustworthy digital democracy tools, rooted in public governance and European democratic traditions.

Voxit 1.0 source code is now available

The source code for version 1.0 of the European community edition of the Voxit platform has now been published and is openly maintained on GitLab.com at: https://gitlab.com/voxit/voxit#

The RISC-V CPU architecture has been gaining a lot of popularity since it launched in 2014, and now that the industry is standardizing on the RVA23 level that includes vector support as a mandatory extension, we are likely to see a lot more edge- and IoT devices with the ability to run local LLMs at reasonable speed, and most importantly at very compelling prices.

SpacemiT is a Chinese RISC-V CPU manufacturer that launched on May 11th, 2026, their long-anticipated next-gen RISC-V AI chip K3. It is among the earliest RISC-V CPUs that adhere to the RVA23 standard and performance-wise it is quite capable, providing 130 KDMIPS general computing power, 60 TOPS on INT4 which translates to about 15 tokens per second when running a 30 billion parameter large language model.

The aspect that really makes it stand out is:

• the RISC-V CPU architecture is open source,
• the price point is within reach of home and small business users and
• the overall feature set makes it an ideal platform to build local and offline AI systems.

SpacemiT also develops their own Debian-based Linux distribution Bianbu OS, and seems to have collaboration going on with the wider community. Their community site seems active, and they also have a dedicated X account @spacemit_riscv and Reddit account r/spacemit_riscv posting relevant progress info on Linux kernel upstreaming activities. The X account is also responsive, as evidenced by its replies to my questions.

Canonical lists the SpacemiT K3 pico-ITX and K3 CoM260 Kit on its official Ubuntu for RISC-V partner-built hardware page, which strengthens the perception that upstream Linux support is being taken seriously. The SpacemiT folks also gave an interesting talk at the 2026 Ubuntu Summit that includes a peek into their roadmap with future K3, K7 and K9 models.

For technical details, see SpacemiT’s K3 pico-ITX documentation, the Jetson Orin Nano-compatible K3 CoM260 board documentation and documentation of the K3 processor itself.

Comparing the resellers

SpacemiT does not sell anything directly to consumers. Instead you need to buy a board that includes the K3 chip from an integrator. Currently the main resellers are:

• Milk-V
• Sipeed
• Banana Pi
• Firefly
• DeepComputing

All of the above are Chinese companies that ship to customers both inside and outside China. DeepComputing stands out as the only one that actually has done real integration and ships the K3 on a custom board, while the others simply resell the SpacemiT-produced K3 pico-ITX and K3 CoM260 Kit.

Milk-V

Milk-V is a RISC-V specialized integrator, as the name already implies. They sell the K3 under the name Jupiter2. Of all the K3 pico-ITX reseller product pages, the Jupiter2 presentation is the nicest and most detailed. Unfortunately their order page at arace.tech only states that it is a “pre-order” with no information about shipping schedule, taxes, or other details like what SSD is included (if any). Based on the pictures it does ship with a Milk-V branded case. The 32 GB RAM lists at 504 EUR, which is a very reasonable price. The @MilkV_Official account on X recently promoted the K3.

Documentation and support

As of this writing, the Milk-V Jupiter2 documentation site is just a stub and has no actual content, and only two links to the SpacemiT K3 documentation site. For support there is a web forum with a dedicated Jupiter2 section. There is also a Matrix space, but unlike their other products, there is no dedicated Jupiter (neither v1 nor v2) channel.

Community size and open source involvement

At least one prior Milk-V product was certified by Canonical, which indicates there is some collaboration in progress. Canonical also lists the Milk-V Titan on its official Ubuntu for RISC-V partner-built hardware page.

Sipeed

The Sipeed K3 announcement is well written (in English) with all the relevant details and links to additional PDF manuals. However, their main page at sipeed.com says nothing about the K3, so one must know the subpage URL to access it. They offer both the K3 CoM260 kit compatible with Jetson Orin Nano carrier boards, and the stand-alone K3 pico-ITX-sized motherboard. The CoM260 kit is only 10 USD cheaper than the full pico-ITX motherboard, so choosing the latter is a no-brainer if starting from scratch. The pico-ITX model with 32 GB DDR5 RAM sells for 639 USD. The product page does not mention anything about hard disk size, so you don’t really know exactly what you will be getting if placing an order. There is no indication about case, Wi-Fi antennas or power supply either, so most likely they are not included.

Their store.sipeed.com website does not work at all, and their Taobao and AliExpress stores are not public and only accessible to registered users. The order page also says nothing about shipping time, delivery time, or taxes. The X account @SipeedIO is active and recently posted pictures of shipments in progress.

Documentation and support

The main documentation wiki does not yet have any K3 content at the time of writing. There is a Discord channel for general RISC-V discussion, and their MaixHub also has a discussion board, but I didn’t find anything K3-specific.

Community size and open source involvement

Sipeed has had at least one of their previous devices certified by Canonical, which indicates they are active in the community.

Note that the other RISC-V company SiFive that also has had hardware certified and officially supported by Canonical is a different company, despite the very similar name.

Banana Pi

Banana Pi announced that they offer both the K3 CoM260 kit and the K3 pico-ITX motherboard version. Their product page for the K3 confusingly shows a MediaTek product in the page banner rather than the SpacemiT K3. Based on the product description and the fact they renamed the product as BPI-SM10, it seems to ship with some carrier board. The product pictures look identical to the SpacemiT documentation and there is no picture of the carrier board, and details are very sparse. The pico-ITX version with 8 GB RAM and 128 GB SSD sells for 293 USD and the CoM260 developer kit with the same specs sells for 287 USD and the 32 GB RAM with 128 GB SSD model sells for 595 USD. The shop page shows only five orders so far and items are currently out of stock. As there was no 32 GB RAM version of the pico-ITX available at all, this isn’t an option for me as I want to run 30B parameter models that need the larger memory version.

Of all of these resellers, the Banana Pi website seems the most outdated. It does not have a search feature, it is not mobile-friendly, pictures can’t be pinched to zoom in and so forth. Product names are also almost all identical, and as the product listings only show the beginning of the product name, figuring out what product is what requires extra effort that just makes the online purchase experience plain bad.

Documentation and support

I was only able to find the documentation page for the CoM260 kit, but none for the pico-ITX version. For support there is a forum, but the category list does not show any section for K3, and the forum search prohibits using the search term “k3” as too short.

Community size and open source involvement

Banana Pi has a long history in the ARM single-board computer market, but their presence in the RISC-V ecosystem is still growing. Their X account @sinovoip has posted only once about the K3 and otherwise promotes their ARM boards. However, their community culture page does express a commitment to open hardware in general, but there is no visible K3-specific community activity.

Firefly

Firefly’s K3 product page is comprehensive. Based on the details, they do not offer the K3 pico-ITX variant at all, but only the K3 CoM260 board inside the AIBOX-K3 Firefly RISC-V Edge Mini PC product. This is a feature-complete offering with a Jetson Orin Nano carrier board and case. The AIBOX-K3 with 32 GB RAM and 128 GB SSD in a case sells for 689 USD in their own Firefly.store. Unfortunately it only has HDMI and there is no USB-C with DisplayPort support, which is a deal-breaker for me personally.

Interestingly, Firefly also offers rack-mounted servers with K3 as the CPU.

Documentation and support

The wiki link on the product page is broken. The Firefly wiki does have a section for the AIBOX-K3, but it too has a broken link. It seems that as of the time of writing, there is no wiki section for this product yet.

For support there is a web forum, which does have at least one K3 thread covering guides such as Hermes Agent installation, though broader K3-specific sections are still sparse.

Community size and open source involvement

Firefly’s X account @TeeFirefly has had no posts since 2024, and their GitLab/T-Firefly shows mostly 2024 activity, with only one repository updated in 2025 and nothing in 2026. Historically they have built a moderate community around their ARM-based Rockchip boards, with active forums and wiki contributions for those product lines. Their RISC-V K3 offerings are newer, and likely need a lot more polish to be attractive products overall.

DeepComputing

Last, but certainly not least, is the laptop manufacturer DeepComputing that offers a Framework laptop compatible motherboard with the SpacemiT K3 chip. They also sell the plain motherboard, or with the Cooler Master case, which allows one to easily connect it to an external monitor and keyboard and use it as a desktop computer. The plain board with 32 GB RAM and no SSD sells for about 882 EUR. Shipping of the first batch is expected to start by end of June 2026. Their X account @DeepComputingio promotes this DC-ROMA RISC-V Mainboard III as their flagship product, so they seem to put a lot of effort into it.

The overall product design and packaging seems good. Of all the K3 resellers and integrators that I was able to find, DeepComputing is the only one that actually designs their own boards with the K3 processor, while all the other vendors above are simply reselling the vanilla K3 boards with or without a case.

After reviewing all these options I decided to buy the DC-ROMA RISC-V Mainboard III for Framework Laptop 13 with 32 GB RAM, 1 TB SSD and the Cooler Master case, totalling about 1100 EUR.

Documentation and support

DeepComputing maintains product information for their RISC-V hardware at github.com/DC-DeepComputing/Framework, with documentation of the newest Mainboard III (FML13V05) still being finalized ahead of the first batch shipment. They provide community support through Discord and web forum, although the latter has very little activity.

Community size and open source involvement

DeepComputing has established itself as a pioneer in RISC-V laptops, beginning with the DC-ROMA. I have seen their stand at FOSDEM, which shows they are genuinely active in the open source community. Canonical lists DeepComputing’s first mainboard / FML13V01 on its official Ubuntu for RISC-V partner-built hardware page, and it seems likely that they will continue to collaborate with Canonical with the new model once it ships. While the underlying Linux enablement depends on SpacemiT’s upstream efforts, DeepComputing’s involvement helps bridge the gap between reference hardware and consumer-ready products.

Conclusion

After weighing all the options, I ended up placing an order with DeepComputing for their custom K3 board with the Cooler Master case. Despite the premium price, the active community support and the properly documented promise of a complete, working system made it easy to place an order with confidence.

The SpacemiT K3 is poised to be one of the most significant RISC-V chips for local AI workloads, thanks to its RVA23 compliance and high tokens per second potential. Yet the buying experience in mid-2026 remains fragmented and incomplete. Hopefully this is just because the product is new, and they will get the purchase experience polished soon.

What struck me most during this process was how poor the customer experience is across nearly all of these vendor websites: broken links, missing search functions, outdated product banners, pages that show the wrong product entirely, and no information about shipping times, stock levels, taxes, and so on. One wonders why these companies don’t fully invest in their web presence.

Personally I would assume they likely have enough customers already, primarily through domestic channels like Taobao and JD.com, that they do not feel any pressure to improve their international-facing sites. However, I did also review what was offered on Taobao, and the product details were very incomplete there too. Taobao, however, has a built-in live chat with almost all sellers, which can be used to ask questions and thus compensate for missing product details.

I don’t fully understand why the sales process seems unpolished. The websites feel almost like an afterthought – a checkbox to claim global reach while the real business apparently happens elsewhere via closed platforms or via inaccessible reseller channels. It is a frustrating reminder that in the RISC-V hardware world, the technology may be open and global, but the purchase experience is less so.

A new minor release 0.4.27 of RQuantLib, the first in over a year, arrived on CRAN a couple of minutes ago, has just now been uploaded to Debian, and is being built for r2u as well.

QuantLib is a rather comprehensice free/open-source library for quantitative finance. RQuantLib connects (some parts of) it to the R environment and language, and has been part of CRAN for nearly twenty-three years (!!) as it was one of the first packages I uploaded to CRAN.

This release of RQuantLib brings an update to the interface for all equity options, vanilla and exotics as well as implied volatilities. We now support the option maturity via either an actual maturity date, or the (fractional business-day years) numeric. This uses a clever little Rcpp trick I should discuss in a separate blog post. We also re-ran compileAttributes() to re-create the RcppExports.cpp file now using a slightly improved way of calling Rf_error for an ongoing Rcpp transition, and did some more standard maintenance. The details from the NEWS file follow as usual.

Changes in RQuantLib version 0.4.27 (2026-06-07)

• All equity option functions can now take either a (fractional) time span to expiry or a given date, and accept a daycounter setter.

• Two very old schedule helpers had a superfluous try/catch removed.

• The continuous integration setup received a minor update.

• The RcppExports.cpp file was updated to aid a Rcpp transition.

Courtesy of my CRANberries, there is also a diffstat report for the this release. As always, more detailed information is on the RQuantLib page. Questions, comments etc should go to the rquantlib-devel mailing list. Issue tickets can be filed at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

I finally carved out some time today to prepare and release debsecan-mcp v0.1.2 to PyPI. During this release, I integrated PyPI's trusted publisher mechanism, which authenticates directly via GitHub Actions and eliminates the need for manual uploads or static API tokens.

A while back, I got my first subwoofer (a surprisingly nice addition to the movie experience, just like rear speakers were). But I live in an apartment, and I don't want to annoy my neighbors at night (the speaker cone points literally down into the floor, and I have no idea how much my neighbors get to share in my enjoyment). So, what to do?

It turns out my receiver supports a sort-of documented serial protocol; it doesn't have an actual serial port, but you can telnet into it (only one session at a time!) and get the same two-way stream. (It also has a HTTP version which I find less useful.) So this allows me to impose my own policy, and of course, doing it via an existing Home Assistant adapter or something was no fun and also thoroughly frustrating, so I saw it as an opportunity to keep maintaining my low-key Rust skills. (No, no LLM code generation. If I'm going to spend time on this, at least I can learn something myself. I think I asked one for code critique at some point, but I can't remember.)

The policy is roughly: If I'm watching TV after 22:00, then the subwoofer is either turned off (if possible) or turned down -12 dB (the maximum). But if I'm watching a Blu-ray or another input like that, that's presumably a conscious tradeoff I've made and things are left at normal. Everything gets a bit more complicated by the fact that the receiver tends to lose state when doing certain switches, and when it boots, it takes a minute or two before Telnet responds, and when it shuts down, it goes into this weird limbo state where it doesn't respond to anything but the TCP connection seems still up.

And then I figured out I also wanted to dim the display when watching movies (again, only certain inputs), but not for a couple of seconds after making any adjustments. And after doing that, I figured that my access point LED should also be turned off, which happens to be some SNMP writable stuff against the Cisco wireless controller it hangs on.

So, if you have a Denon or Marantz AVR, a Cisco access point on a controller, and my exact preferences about what to do about the subwoofer, then you are free to download and use my software to impose that policy. It is “is distributed in the hope that it will be useful”, as one says. If you have IPv6.

Debian LTS/ELTS

This was my hundred-forty-third month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian.

During my allocated time I uploaded or worked on:

• [DLA 4580-1] exim4 security update to fix one CVE related to remote code execution.
• [DLA 4591-1] rsync security update to fix five CVEs related to local root privilege escalation.
• [#1134340] trixie-pu bug for libcoap3 to fix two CVEs in Trixie; the debdiff was confirmed and the upload was accepted to the proposed update queue.
• [#1126167] bookworm-pu upload of zvbi has been flagged for acceptance
• [#1126273] bookworm-pu upload of taglib has been flagged for acceptance
• [#1126370] bookworm-pu upload of libuev has been flagged for acceptance
• [hplip] upload to sid to fix two CVEs.

This was a rather strange month. The details about the embargoed exim4 issue arrived only after I already went to bed and the embargo lift was 18 hours later. Luckily Stretch was not really affected and the uploads for Bullseye and Buster went out on time.

Something similar happened with the embargoed issue of rsync. The info arrived at 8:00 in the morning and the embargo lift was on 2:00 next morning. From an Europeans point of view, the Australians do have strange time zones. But there is more to this than that. Upstream sent more than 50(!) patches for these five CVEs that needed a backport to Bullseye. As things turned out, there is a regression in the upload to Unstable and investigations are ongoing whether this regression is also available in the backported patches for Trixie, Bookworm and Bullseye. So rsync-updates for Buster and Stretch is in the works, but I am afraid they need some more time.

All good things come by threes. Two critical CVEs of hplip appeared and a new upstream version was released by HP. HP is no longer interested in working with distributions and over time more than 80 patches have been accumulated that need a rebase for a new upstream version. For that reason I avoid this package as much as I can, but two critical CVEs did apply some kind of pressure on the maintainer. So I finally managed to do this update and the latest version of hplip is now in Debian. Nevertheless, this feels good :-). Anyway, it is not over yet. HP does not have a public repository nor do they publish patches for these CVEs. So I am still searching for the correct fixes to backport them to Bullseye, Buster and Stretch. The other distributions have the same problem and a silver lining appears on the horizon.

I also prepared an update of gimp for Buster and Stretch, but due to an accident I only managed to release the corresponing ELA in June. The accident was also the reason for only half a week of FD. Thanks to Daniel who took over.

Debian Printing

This month I uploaded a new upstream versions:

• … lprng to unstable.
• … epson-inkjet-printer-escpr to unstable.
• … hplip to unstable.

This work is generously funded by Freexian!

Debian Lomiri

This month I continued to work on unifying packaging on Debian and Ubuntu. This makes it easier to work on those packages independent of the used platform.

This work is generously funded by Fre(i)e Software GmbH!

Debian Astro

This month I uploaded a new upstream version or a bugfix version of:

• … supernovas to unstable (sponsored upload).
• … virtualgps to unstable.
• … nautic to unstable.

Debian IoT

This month I uploaded a new upstream version or a bugfix version of:

• … pyicloud to unstable.

misc

This month I uploaded a new upstream version or a bugfix version of:

• … visam to unstable.
• … tntdb to unstable.
• … ae56 to unstable.
• … texify to unstable.
• … chktex to unstable.
• … ta-lib to unstable.

I also got rid of gypsy, which no longer makes sense to maintain in Debian, as gpsd is way better.

I previously wrote some advice for developers and distributions about the upcoming Microsoft CA Rollover, and I hope that was useful for people.

I've now also added some user-facing documentation about the CA rollover in the Debian wiki at https://wiki.debian.org/SecureBoot/CAChanges. I've added guidance on managing certificate updates on Debian systems: how to check if a system needs those updates and various ways to make them happen. If you're running Secure Boot systems, this may be important for you.

While the same event is the primary cause for these docs, they're designed for different people. Again, I hope this new doc is helpful!

Introduction

I have just bought a HP Z4 G4 with W-2125 CPU for $320 and I decided it was a good time to do some benchmarks on Debian package building to see which system I should use for that.

The W-2125 CPU scores only 9,954 on the passmark multithread test but scores 2,546 on single thread [1]. Passmark seems to have some limitations as the only DDR3 system that’s important to me at the moment (the HP Z420 workstation my parents use which cost me $750 in 2021) with a E5-2620 CPU scoring 5,325 for multithread and 1,113 for single thread [2]. From the passmark results one would expect that the system is slightly more than twice as fast as the Z420 for operations that involve less than 4 CPU cores.

For the initial tests of the Z4 G4 I ran them with hyper-threading enabled as 4 cores isn’t much by today’s standards and also the machine in question is going to be less exposed to hostile data and contain less secret data than most of my systems so the security risks of hyper-threading are less of a concern.

I did some tests with a couple of tasks that are very important to me, building SE Linux policy packages (something I may do a dozen times in a day) and building Warzone 2100 (which I do less often but is the most intensive build process I regularly run). At the bottom of this post there are tables with the results from building these packages on my Z640 workstation with a E5-2696 v4 CPU [3], the Z420, and the new machine.

For the Warzone 2100 package I tested building on my Z840 dual CPU system [4]. I didn’t test building the SE Linux policy on the Z840 this time because that package can’t take advantage of even 22 cores. When I initially got the Z840 running it built the policy packages faster because the Z640 had an older CPU that was slower for single core operations than the CPUs in the Z840.

BTRFS Compression

For some time I have noticed significant differences in compile time on my workstation, a factor of more than 2. I did more tests and noticed that “top” showed something like the following, those kernel threads are all BTRFS related, except for “gfx” which is probably something graphical caused by running Chrome with about 300 tabs open.

2144316 root      20   0       0      0      0 I  26.6   0.0   0:36.76 kworker/u88:20-btrfs-endio-write                                                                                                                                                                             
2221470 root      20   0       0      0      0 I  23.7   0.0   0:01.85 kworker/u88:12-gfx                                                                                                                                                                                           
2221436 root      20   0       0      0      0 I  15.1   0.0   0:07.48 kworker/u88:8-btrfs-compressed-write                                                                                                                                                                         
2166191 root      20   0       0      0      0 I  12.8   0.0   0:15.80 kworker/u88:23-btrfs-compressed-write                                                                                                                                                                        
2126387 root      20   0       0      0      0 I  10.2   0.0   1:29.11 kworker/u88:4-events_unbound 

I had been running BTRFS with the mount option “compress=zstd:15” which caused much of the performance problems when building. It was also a random performance issue which I think happened due to the BTRFS 30 second write-back sometimes taking more than 30 seconds during the build process which then caused a second write-back.

I did tests on ZSTD compression levels 5, 8, 10, and 15. 15 was never good and often really bad. 10 was not unbearable but consistently slower. 8 was sometimes as fast as 5 and sometimes quite a bit slower. I didn’t test levels below 5 because I need to have some compression and it seemed that the benefits of reducing compression were dropping off below 8.

I found that the BTRFS compression delay is not counted in system time for the process. I think it’s the fsync() system calls in the semodule and dpkg-deb programs that cause the delays related to BTRFS compression waiting for kernel threads.

BOINC

I have all my systems other than laptops running BOINC in the background so that CPU power is used for scientific research when I don’t have any personal use for it [5]. I believe that it’s immoral to waste CPU power when it could be used for research.

In the below table which has test results from building the package with and without BOINC, and with different ZSTD compression levels in BTRFS all the worst entries were from when BOINC was running apart from one where ZSTD level 15 compression was used. The really poor performance with ZSTD level 15 was an outlier, but it wasn’t an uncommon outlier so I left it in.

Running BOINC in the background configured to use all CPU cores caused a significant increase in “user CPU time” (the time a CPU core spent actually running the program). My initial thought was that it’s partly related to “turbo boost”.

The Intel ARK page for the CPU in the Z420 shows that it’s main clock speed is 2.0GHz with a 2.5GHz “turbo boost” [6]. The “turbo boost” is apparently largely based on temperature and apparently limited to one core, so if the other CPU cores are all being used then the CPU will probably be too hot to have the turbo boost and if it happens it might not happen for my compile processes.

The ARK page for the E5-2699 v4 (which is a similar CPU to the E5-2696 v4 that I’m using but is officially documented by Intel) [7] shows that it has a base clock speed of 2.2GHz and a turbo boost speed of 3.6 GHz. 322 vs 244 seconds of user CPU time means running 32% slower which can plausibly be explained by the lack of a 64% turbo boost with a bit of help from the 55MB L3 cache being thrashed.

Turbo boost would only be a noticeable issue for building packages like the SE Linux policy packages which doesn’t take much advantage of multi-core CPUs. For a build process to average at best 362% CPU use there has to be large parts of the process that are limited to one or two cores which can potentially give a benefit from turbo-boost.

When building the Warzone 2100 packages most of the build time is running basis-universal which is a multi-threaded program to compress GPU texture data. This usually causes a load average of 300+ on the Z640 or 600+ on the Z840. But the build time is still increased by more than 50% on both the Z640 and the Z840 when BOINC is running in the background, which seems to be an indication that it’s not related to turbo boost. I verified that BOINC is running at IDLE schedule priority with the following command:

# chrt -p $(pidof -s einstein_O4MD_2.01_x86_64-pc-linux-gnu)
pid 2974874's current scheduling policy: SCHED_IDLE
pid 2974874's current scheduling priority: 0

In theory this means that BOINC won’t affect foreground processes.

Hyper Threading on the W-2125

The best claims I’ve seen about HT are 15% to 30% performance boost. The best I’ve actually seen in the past is about 18%. Seeing a 10% benefit for building Warzone 2100 is at the low end of the range I expected. 8 virtual cores is not many for a build process that causes a load average of 600+ when running on a system with 44 real cores.

I was surprised to see a 6% performance benefit in hyper-threading for building the SE Linux policy as I didn’t think there was enough use of threading or multiple processes to allow that.

Many build scripts use a number of processes that match the number of apparent CPU cores. While “make -j 88” might give a theoretical performance benefit on a 44 core system it will also take a lot of RAM and any paging will outweigh the benefits of hyper-threading. On a system with only 4 real cores there’s less potential for using too much RAM and as security isn’t so important on that system I will leave it on.

Comparing the CPUs

The best results of the Z640 and Z4G4 are only 50% faster than the best results of the Z420.

The Z420 has a E5-2620 CPU which is far from the fastest CPU available for that system – the E5-2687W has 8 cores and rates 10,021/1,669 on passmark [8] which is far better than the 5,331/1,114 the E5-2620. The E5-2687W is the fastest CPU that HP lists as supported by the Z420 and it supports DDR3-1666 RAM as opposed to the DDR3-1333 that is the fastest that the E5-2620 supports. With suitable hardware upgrades the Z420 would probably only take about 20% longer to do builds of the SE Linux policy and other packages that can’t take advantage of more than 8 CPU cores.

The Z4G4 system has 4 RAM channels which means that you should get some performance benefits from having 4 DIMMs, my system currently has 2 and I haven’t yet managed to get more DDR4-2666 DIMMs. But I’d still expected a W-2125 CPU with 2*DDR4-2666 DIMMs outperform any E5-26xx CPU with 4*DDR4-DDR-2400 DIMMs for tasks that average less than 4 CPU cores.

In retrospect I would have been better off getting a HP Z820 (two socket server with DDR3 RAM) than the first DDR4 systems I got. It seems that for reasonable size builds a two socket system comes close to twice the speed of a single socket system. I did briefly own a HP ML350 two CPU system with DDR3 RAM but it was too noisy for my intended use as a deskside workstation so I sold it.

Things to Investigate

I plan to do more investigation on BTRFS compression, how to get the best compression without excessive delays and how to recognise when delays are happening. I have some SSDs that have sustained write speeds as low as 15MB/s (Crucial P1 series) so for those I could probably have very high compression levels without slowing the system down.

The fact that BIONC slows things down so much seems to be a bug. When processes are running with the IDLE scheduling class there shouldn’t be such significant delays. Is it due to cache thrashing? How can I best get BOINC suitably throttled when I’m sitting at my workstation, I don’t want BOINC connecting to the local X server (which it repeatedly tries to do). Do I need to tune my kernel for better handling of IDLE scheduling?

When I get more DIMMs in the Z4G4 I need to do more tests to see if it gives an overall performance boost.

Also the Z4G4 system has a BIOS option for “sub NUMA” which basically means treating the different RAM channels on a single CPU as NUMA zones, I enabled that option which does nothing presumably because I only have 2 DIMMs, the results when I have 4 DIMMs will be interesting. I will also do some NUMA tests on the Z840 to see what benefits it gives.

I have a selection of RAM speeds that will work in the Z4G4, if I have enough spare time I’ll test what difference that makes for CPU bound tasks that matter to me.

For package building fsync() is not helpful, if the system crashes before it’s done then I will just do the build again. For a build cluster it is probably a good feature and probably doesn’t affect aggregate performance when multiple packages are built at the same time, but for the single user case probably not. I will investigate libeatmydata for package building [9].

Conclusion

The progress in CPUs seems to have slowed down a lot recently. The main benefits seem to be in more CPU cores and for newer sockets with more RAM channels.

The CPUs that do have improvements in single core performance are the i9 series (which mostly doesn’t come with motherboards supporting ECC) and AMD CPUs (which is rare in enterprise class hardware). Maybe I should get a server with an i9 or AMD CPU for tasks that need a fast turn around with a small number of cores. That would probably outperform any CPU designed for large core counts for things like building the policy and setting up test VMs (which depends on package installation speed that is single core bottlenecked).

The W-21xx CPUs seem to offer little benefit over the E5-26xxv4 CPUs and not a lot of benefit over E5-26xx CPUs (with DDR3). Even the W-22xx CPUs look like they aren’t going to offer a lot as they are only an incremental improvement over the W-21xx series. I had considered making the Z4G4 my main desktop workstation after the high end W CPUs become affordable, but it looks like that won’t be worth it until such CPUs drop from the current ebay price of $900 to $100.

I think I’ll keep waiting for a decent socket LGA3647 or DDR5 based server [10] for my next significant upgrade.

Tables

Building SE Linux Refpolicy

Building Warzone 2100

Hyper-Threading

• [1] https://tinyurl.com/2ddf7t5y
• [2] https://tinyurl.com/kgmagfs
• [3] https://etbe.coker.com.au/2026/04/10/hp-z640-e5-2696-v4/
• [4] https://etbe.coker.com.au/2025/04/05/hp-z840/
• [5] https://boinc.berkeley.edu/
• [6] https://tinyurl.com/2mopjxgc
• [7] https://tinyurl.com/2r3j4bzg
• [8] https://tinyurl.com/reu2p84
• [9] https://www.flamingspork.com/projects/libeatmydata/
• [10] https://etbe.coker.com.au/2025/08/02/server-cpu-sockets/