Bernhard R. Link: I learned something new about URLs today
Today I stumbled over some behavior that I found quite surprising:
$ ipython3 -c 'import httpx;print(httpx.URL("https://example.com/foo/bar/../../baz"))'
https://example.com/baz
Even more surprising that behavior is actually standards-compliant, even mandated by RFC 3986.
The underlying motivation is relative reverences. If some resource reachable by "https://example.com/foo/bar" references another resource relatively as "../../baz" then this is of course the intended result.
Getting from this problem to what RFC 3986 suggests might be surprising in the result, but somewhat understandable if you look at the consequences of that problem:
Giving the path components ".." (and ".") special meaning at the start of the relative reference means that if you allowed them in absolute URLs those would be impossible (or at least very convoluted) to address as relative URLs.
So RFC 3986 describes a way to handle them everywhere: Just join the path of the base URL and the path of the relative reference and normalize the result. Or normalize the absolute on either side if only that is to be taken. This makes things very convenient: Multiple reference URLs can just be joined without special handling for relative references starting with dots, making writing applications handling them easier. Programmers don't have to care how to handle relative references and can just join everything in whatever way they want.
For maximum elegance there is still some corner case left: What happens if an absolute URL has a path starting with double-dot components? Or an relative path starting with more of them then the base URL's path has components. You just ignore them:
$ ipython3 -c 'import httpx;print(httpx.URL("https://example.com/../../baz"))'
https://example.com/baz
With that last point every URL is valid and has well-defined meaning. Handling relative references and relative paths is very easy and convenient.
So this shows a high regard for simplicity, elegance and convenience. And a total and uncompromising disregard of security.
After all the most convenient it is for an attacker; If they are allowed to supply a path component for a request a system does in their behalf, then they can easily escape anything they were supposed to be limited to. The ignoring of dots at the start means they don't even have to know exactly how deep their request is:
$ python3 -c 'import httpx;print(httpx.URL("https://example.com/public/api/public/resources/harmless/../../../../../../../../../internal/data"))'
https://example.com/internal/data
So even if the resource server securely handles request (unless you consider not having any way to lower your permissions for one request to a specific subset), your fully RFC conforming client library will already request the permission they should not have permission for. Even worse dots are usually not characters you can easily forbid so once slashes are to be allowed things get complicated.
There also would have been a simple, elegant and secure way: Consider every path element ".." or "." in an (absolute) URL an error. Define a reference resolution that allows the relative reference to only start with "./" or one or multiple "../" and consider every appearance of a dot or two dots as path components after than an error.
Everything joining two paths has to either use an implementation of that path joining algorithm, but only if they want to joins paths in the potentially dangerous way allowing leading "../". Otherwise they can just use the normal join and even if an attacker gets those dots that will just cause the generated URL to be rejected as invalid.
Of course using a secure implementation is now even more inconvenient thanks to RFC 3986 being around: If you have no control over the generator of relative references, it is always possible that they generate relative references with ".." components after non-dot components.
And if you check all code to properly filter out "/../", keep in mind that convienence does not stop there. After all it is not unheared of for server implementations to helpfully normalize unicode characters, too, or translate them to their nearest ASCII equivalents. Or translate percent escaped characters back before doing path splitting. Or you might think there was some unicode codepoints between those two dots, but they that those were some meaningless control characters that can be omitted. So you need some really restrictive allow lists...