❌

Normale weergave

Bernhard R. Link: I learned something new about URLs today

3 Augustus 2026 om 21:39

Today I stumbled over some behavior that I found quite surprising:

$ ipython3 -c 'import httpx;print(httpx.URL("https://example.com/foo/bar/../../baz"))'
https://example.com/baz

Even more surprising that behavior is actually standards-compliant, even mandated by RFC 3986.

The underlying motivation is relative reverences. If some resource reachable by "https://example.com/foo/bar" references another resource relatively as "../../baz" then this is of course the intended result.

Getting from this problem to what RFC 3986 suggests might be surprising in the result, but somewhat understandable if you look at the consequences of that problem:

Giving the path components ".." (and ".") special meaning at the start of the relative reference means that if you allowed them in absolute URLs those would be impossible (or at least very convoluted) to address as relative URLs.

So RFC 3986 describes a way to handle them everywhere: Just join the path of the base URL and the path of the relative reference and normalize the result. Or normalize the absolute on either side if only that is to be taken. This makes things very convenient: Multiple reference URLs can just be joined without special handling for relative references starting with dots, making writing applications handling them easier. Programmers don't have to care how to handle relative references and can just join everything in whatever way they want.

For maximum elegance there is still some corner case left: What happens if an absolute URL has a path starting with double-dot components? Or an relative path starting with more of them then the base URL's path has components. You just ignore them:

$ ipython3 -c 'import httpx;print(httpx.URL("https://example.com/../../baz"))'
https://example.com/baz

With that last point every URL is valid and has well-defined meaning. Handling relative references and relative paths is very easy and convenient.

So this shows a high regard for simplicity, elegance and convenience. And a total and uncompromising disregard of security.

After all the most convenient it is for an attacker; If they are allowed to supply a path component for a request a system does in their behalf, then they can easily escape anything they were supposed to be limited to. The ignoring of dots at the start means they don't even have to know exactly how deep their request is:

$ python3 -c 'import httpx;print(httpx.URL("https://example.com/public/api/public/resources/harmless/../../../../../../../../../internal/data"))'
https://example.com/internal/data

So even if the resource server securely handles request (unless you consider not having any way to lower your permissions for one request to a specific subset), your fully RFC conforming client library will already request the permission they should not have permission for. Even worse dots are usually not characters you can easily forbid so once slashes are to be allowed things get complicated.

There also would have been a simple, elegant and secure way: Consider every path element ".." or "." in an (absolute) URL an error. Define a reference resolution that allows the relative reference to only start with "./" or one or multiple "../" and consider every appearance of a dot or two dots as path components after than an error.

Everything joining two paths has to either use an implementation of that path joining algorithm, but only if they want to joins paths in the potentially dangerous way allowing leading "../". Otherwise they can just use the normal join and even if an attacker gets those dots that will just cause the generated URL to be rejected as invalid.

Of course using a secure implementation is now even more inconvenient thanks to RFC 3986 being around: If you have no control over the generator of relative references, it is always possible that they generate relative references with ".." components after non-dot components.

And if you check all code to properly filter out "/../", keep in mind that convienence does not stop there. After all it is not unheared of for server implementations to helpfully normalize unicode characters, too, or translate them to their nearest ASCII equivalents. Or translate percent escaped characters back before doing path splitting. Or you might think there was some unicode codepoints between those two dots, but they that those were some meaningless control characters that can be omitted. So you need some really restrictive allow lists...

  •  

John Goerzen: Celebrating 45 Years of Kermit with the First New C-Kermit Release in 15 Years (and working with a decades-old C codebase)

3 Augustus 2026 om 17:45

1981 was a different time for computing. It was expensive (both hardware and software), and it was far from a given that machines from one vendor would be able to talk to those from another. In fact, Columbia University had just such a problem, so in 1981, Frank da Cruz and Bill Catchings designed a serial protocol they called Kermit. Because of the many quirks of the DEC-20 and IBM mainframes, the Kermit protocol was highly adaptable from the start: able to handle systems that had trouble processing more than 96 bytes of data at once, able to transfer 8-bit files over 7-bit links, able to translate between character sets (ASCII and EBCDIC then; now also various Unicodes), and of course, handling of error-prone serial links.

Kermit spread rapidly; by 1982, Kermit had been ported to MS-DOS and Unix. Eventually, C-Kermit (an implementation of Kermit in C) became the flagship Kermit. It gained TCP support, an interactive CLI, a powerful scripting language (with features from the shell, Lisp, and expect), and optimizations for today’s high-speed links, such as jumbo packets, sliding windows, and streaming modes. Along the way, Kermit flew on the International Space Station, ran data collection from sensors during hurricanes, and many other uses including postal systems, Boeing 787 manufacturing, and more.

Today, I use it as a powerful ssh wrapper (letting me easily transfer files through multiple nested ssh, sudo, su, etc. commands), a BBS client, to exchange data with me HP 48GX calculator, and so on. It’s also used today to transmit firmware updates to embedded devices. And, of course, anyone that works with vintage systems is likely to use Kermit at some point.

It wouldn’t be until the late 1990s that the TCP/IP stack was finally adopted by most OS vendors, establishing something of a common basis for communication. Of course, we assume this today. Though transferring large files between OSs (say, Linux, Windows, MacOS, Android, iPad, etc.) is still a challenge, even though they all speak TCP/IP! I find that the easiest way to get large files from two computers is to spin up Kermit (see ckwin for a Windows fork of C-Kermit) and just set up a TCP connection over the LAN. In fact, I added a new show interfaces command in C-Kermit 11, making it easy to see your system’s local IPs.

For most of its history, Columbia’s Kermit project was self-funded. Columbia charged for commercial use, which limited its inclusion in Linux distributions. In 2011, 30 years after its founding, Columbia canceled the Kermit Project and released C-Kermit as Open Source under a BSD license. Frank da Cruz, who had still been working with the Kermit project all those years, volunteered to continue maintaining Kermit outside Columbia, and continued development with alpha and beta releases through his retirement from the project in 2025.

I dive into this C codebase

As Debian maintainer of Kermit, I noticed some areas where it wasn’t matching modern expectations. One area was, not surprising for a project of its age, security. Another area was that its character set or line-ending conversions are usually not desired now; we are used to byte-identical binary transfers, and the defaults caused confusion and even some rare instances of data corruption. So I started making a few patches last year.

I’ve worked with old C codebases before, such as Varnish. I’ve generally hated it. You usually find a mix of bad and terrible practices, unclear memory management, and so forth.

But I’ve been living in the C-Kermit codebase for a few months now, and I enjoy it. Yes, this thing is still designed to build on VMS, OS/2, and with compilers that haven’t heard of ANSI β€” and those that require modern practices. (That em-dash was mine; I knew how to use them before LLMs existed and I’m not going to stop just because LLMs have copied people like me! No AI was used for this post.)

The there is an elegance in all of that. As I worked, I fixed a bunch more potential security issues, both with memory safety and with protecting against a malicious remote in roughly the same manner that some patches to scp did a few years back. I added IPv6 support, of course conditionally compiled because some systems C-Kermit builds on have never heard of IPv6 and never will. (And, of course, with fallback algorithms at runtime for systems that have IPv6 support but not IPv6 connectivity.)

I added unit tests and Python-based end-to-end tests, running nearly 2000 test cases in total. Along the way, I found and fixed a number of bugs going back decades. I learned about FIONREAD being broken on macOS, about NetBSD’s bugs in the pty driver, and fixed bugs in the Kermit protocol implementation itself. I added compatibility tests with the gkermit and ekermit (embedded) implementations, as well as the last full release, C-Kermit 9.0.302 from 2011 (which was difficult to get compiled on a modern system).

There is an extensive changelog describing all the improvements in C-Kermit 11.

C-Kermit development had never really used a VCS at any point, though Kermit veteran Jeffrey Altman imported historical releases into a Git repo, along with some patches that hadn’t made it into a release (which I also pulled in.) There was a lot of disabled code behind #ifdef COMMENT, along with commentary describing why it was no longer used. With Git, we would now generally just remove the old code and explain why in a commit message. I went through and did so with a lot of it, meaning that, at last check, C-Kermit actually has fewer lines of code now than it used to.

Towards a new release

It became apparent pretty quickly that I was making more changes than would make sense as a Debian patch series. Not only that, but they would be more widely applicable to more than just Debian and Ubuntu users. As Linux and BSD distributions were running everything from the last non-beta release (2011’s 9.0.302) to the last beta release (about 1.5 years ago), depending on their different policies about running betas, even sharing patches in a useful fashion was going to be quite difficult.

So, I spun up a project at Open Kermit to coordinate future development in the open and keep Kermit going.

With modern CI, I run that test suite on Linux (x86_64 and arm64), macOS, FreeBSD, NetBSD, and OpenBSD. It builds binary releases on all those platforms, plus a statically-linked Linux binary built with musl libc.

You can download the latest C-Kermit release, and of course contribute to C-Kermit and its website.

Dedication

Frank da Cruz was directly involved with Kermit for 44 years. I’m not aware of any other Open Source project founder being involved for so long. Richard Stallman started working on GNU Emacs in 1984, 3 years after Frank started working on Kermit, but Richard hasn’t been in that role since around 2008.

Accordingly, C-Kermit 11 bears this dedication:

I dedicate this release of C-Kermit to Frank da Cruz.

Frank was directly involved with Kermit for 44 years, from its initial design in 1981 all the way through 2025. He maintained Kermit as an Open Source project after Columbia University ended its sponsorship. I know of no other Open Source project where the founder remains so personally involved for so long.

When Kermit was begun, transfers between different hardware and operating systems were difficult or impossible. Frank helped build a bridge. Kermit glued systems together, from the International Space Station to pocket calculators, and set a new standard for interoperability. It continues to do so.

Kermit is still one of the quietly-working pillars of computing today, enabling everything from firmware upgrades to radios. And, yes, it still reliably transfers files over serial lines.

As we start to spend a lot of time in the Kermit codebase, we do so standing on the shoulders of a giant. Thanks, Frank, for your decades of work on Kermit.

John Goerzen, July 2026

  •  

Russ Allbery: Term::ANSIColor v6.0.0 TRIAL release

2 Augustus 2026 om 23:49

Yesterday, I uploaded Term::ANSIColor v6.0.0-TRIAL to CPAN for early testing. This release will raise the minimum required Perl version to 5.12, dropping support for Perl 5.8 and 5.10. When I did the same with podlators a couple of years ago, it upset a few people and one of them asked me to make this sort of test release in the future. Hopefully this will help.

I have not run the normal release machinery and haven't archived this release in the normal places, since I intend it to be transient. It's only on CPAN, where people can retrieve it for testing. Once v6.0.0 is released, few traces of this TRIAL release will be left. This doesn't appear to be how other people use the TRIAL mechanism, but it felt more comfortable to me. If I have to make substantial changes, I'll consider changing my approach.

I plan on turning this into the v6.0.0 release in about a month or two, hopefully with only documentation changes.

Term::ANSIColor is a "very upstream" core module with a lot of dependencies, and CPAN (unlike some of the archives that followed it, such as PyPI) doesn't support conditionally retrieving packages based on the current Perl version. This release may therefore be disruptive for people who are still trying to support Perl 5.8 and 5.10, since CPAN installation tools may attempt to install an incompatible Term::ANSIColor version. I'm sad that this will be the result, since I know some people still care about those versions.

I'm pressing forward with updating my Perl modules anyway, though. I realized that honoring other people's desire for stability to such a degree that I was unable to use Perl features added more than 15 years ago was destroying my motivation to work on these Perl modules at all. So I've decided on a very slow and gradual approach where I'm going to keep pushing the minimum supported version forward but try to give people a lot of warning.

Personally, I think it's time to let ancient versions of Perl go and follow the Lyon Amendment about supported Perl versions. When we're talking installing new modules for software released more than 15 years ago, we're talking about special limited environments and retrocomputing more than what I would consider routine software maintenance. Those tasks should expect to need different tools and a different workflow so that they can pin historical versions. Since this isn't something I'm personally interested in, my willingness to expend time and energy to assist is limited.

As you can probably tell, I still feel nervous about pressing forward in this way, but I think this is the approach that lets me continue to enjoy maintaining these Perl modules. It's been 29 years for Term::ANSIColor, but I still enjoy fixing bugs in it and putting out a new release from time to time, particularly if I can clean up the code a bit each time I touch it.

  •  
❌