Log in
Dirk Eddelbuettel: #059: r2u, GitHub Actions, a Tragedy of the Commons, and a Fix
Welcome to post 59 in the R4 series.
How did we get here: A initial words about GitHub.
GitHub Actions
provides (essentially unlimited) compute time. This further boosts a
service already in a market-dominating position: GitHub1 as a code repository.
Those of us old enough to remember the start of git (the
program and protocol) may remember the extremely bare-bones
initial hosting site repo.or.cz (launched in 2006). GitHub
came two years later, and put an enormous amount of focus into design
and user interfaces. To cut a long story short, GitHub won the services
war. And with it git won the platform war. To a first
approximation, everybody and everything is on GitHub.2 So
the repository is already dominant.3 And then free compute
was added.
So given its scale and positioning, and its essentially free provisioning of free multi-core compute setups with generally decent connectivity, widespread adoption happened. And as is goes, some mischief is bound to happen. And it did. More on that below.
A few words about r2u: r2u makes all packages on
CRAN, i.e. the code repository
network for R, install fast, reliably and
easy on Ubuntu by making them available to apt,
the native package manager. It is to our knowledge also the first and
only time an entire open source programming repository is available in
binary form with all dependencies resolved. It is going strongly: the
last monthly use topped
five million packages. See the r2u website for more.
r2u and GitHub: For the first few years, builds for r2u were done locally on my machine, and then uploaded to the primary repositry r2u.stat.illinois.edu. I do not recall systemic outages or connection issues though occassional network timeouts were seen. Once we started to support arm64 (in addition to the default amd64) binaries, building those switched to GitHub Actions simply because … they had runners for arm64 while I had no arm64 hardware. The experience of building packages (in bulk) was rather positive. So we investigated builds for amd64 too. If memory serves we first did this for either one of the semi-annual BioConductor updates. Before long, builds for amd64 followed meaning all of r2u was being built in GitHub Actions.
During these builds, I would regularly encounter builds failures: “cannot connect to r2u.stat.illinois.edu”. I misdiagnosed this as a resource issue on the GitHub side, and consequently made (several) attempts at robustifying the builds via for example longer (download) timeout limits as well as checks for build failures and conditional rebuilds. Needless to say, and given what we know now (more on that below), this did not work. But it went on for a few months this spring and summer. What did work was to simply relaunch under ‘re-run failed jobs’. Given the distributed nature of GitHub Action this generally allocates to a different machine and address and succeeds. In the grand scheme of things a nuisance as we a need second run, but given the fourty (!!) concurrent jobs this tends to be quick. So a minor nuisance.
This discribed the production side. On the consumption side, one prominent user of r2u, especially at GitHub, is our r-ci setup for continuous integration. It too could fail at times, and a simple re-run would fix it. Annoying, if addressable manually. Usage by others I cannot monitor so I can only assume that the random failure nature must have frustrated them too. Potentially a much bigger nuisance.
As users were getting annoyed, some took action. Jeffrey Girard opened discussion topic #159 which contained a thorough investigation of his confirming that only amd64 nodes were affected. This had not been noticed before. Troy Hernandez set up a full harness with tests in an ad-hoc repo designed for repeated remote triggering. This also logged the IP addresses for success or failure. Through both these approaches it became (eventually) clear that the failures were limited to either certain (individual) IP addresses, or IP subnets.
When taking the conversation back to network service at U of Illinois, we realized that the issue was in fact caused by a network policy at the university. And specific to GitHub.
In fact, what happened initially were waves of port scanning attacks originating from GitHub IP addresses. As (essentially) “anybody” can run code there, bad actors can too. The response from the university side was reasonable and swift: Identified IP addresses were added to a ‘null-router’ that (essentially) swallows traffic. And that was the cause of the perceived-as-random outages: Jobs that ended up failing at GitHub Actions were the ones assigned to addresses that have previously been seen as port scanning.
Shifting production: Once this was confirmed, I investiaged alternatives. On the production side using different machines would help. So I tried blacksmith.sh, a competing alternate service offering faster runners as ‘drop-in replacements’ for the GitHub Actions runners. This worked great, until I ran up against my ‘free cpu minutes quota’. In a mere two days (that were arguably overly busy as it was shortly after CRAN reopened after the summer break). Given that the service would not sponsor us a supported open source software project with sufficient quota, we moved off blacksmith.sh after two days.
A first programmatic response: consumption-side: For
the r-ci client side,
it was straightforward to setup a check and subsequent workaround. When
curl fails with a silent HEAD attempt at the primary
repository failed, we take this to be caused by presence of a
null-router entry for the IP we are on, and switch the apt
setup to the secondary repository. Which may be slower, or at rare times
unreachable itself – but still provides a fine fallback when a node is
‘prohibited’ from talking to U of Illinois resources such as
r2u.stat.illinois.edu. Having used this for a few days in r-ci it seems to work.
A second programmatic response: production-side: For the r2u builds, and given that blacksmith.sh would not grant ‘most-favored status’ with sufficient free minutes, we switched our Docker-based setup to switch to the secondary when an initial probe fails. That was added last weekend, and appears to work just swimmingly. Another application to the fundamental theorem of software engineering: another layer of indirection can solve just about any problem.
For completeness, the corresponding code is
webstatus=$(curl --head --silent --no-fail --output /dev/null \
--write-out "%{http_code}" https://r2u.stat.illinois.edu || true)
if test "${webstatus}" = "200"; then
echo "The r2u repository is reachable."
else
extip=$(curl --silent https://ipinfo.io/ip)
echo "::notice::The primary r2u repository is **not reachable** from ${extip}."
fiWe run an initial curl test (without failing) and have
it report the HTTP return code. 200 means no issue, all others are
suspect here—so we run a second curl query to obtain our
external IP and log it. We use the same logic in another spot from
inside the build container and use the else branch to switch
apt to the secondary repository via sed call
on the .sources file.
Logging of ‘bad’ IPs: On both our sides, i.e. production as well as consumption, we now also log the IP addresses of the failing nodes and will ask network security to remove these from the null router. If our jobs can be assigned to them it clearly shows the machines are part of the normal compute pool and are not doing anything nefarious at the moment. So they should be removed from the null-router list. We will see how that fares.
Putting it all together: Providing a free resources can, sadly, lead to an a decline the service experience just as the tragedy of the commons analysis would predict. Restricting, or ‘pricing’ use may be a stock answer but I for one am glad GitHub Actions is still free. But we need to do our bit of upkeep. Just as network security logs bad actors (taking advantage of the free resource) we should make an effort to unlist nodes no longer part of any portscan (or alike) swarm.
For r-ci users, there is hopefully little to do (if you rely on the standard action). We do now catch a node that was assigned a continuous integration job cannot connect to r2u as we can test this easily (and cheaply). Pivoting to the secondary repository is a valid, and working, answer. Hopefully over time we can also work towards restricting the null-router list down to recent entries and fewer overall, thereby lowering the chance of gitting a bad IP. Eventually, we could also overly a CDN proxy to avoid the ‘bad IP’ problem. It is something to consider.
Summing up: We are still chuffed at how successful r2u has become, and how much can be done with GitHub Actions. Sadly, as we found out, there can also be a ‘tax’ on letting compute happen there but as discussed in this note, there are ways to avoid it by pivoting to alternate repository source.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.
Before we really get started, one clarification. GitHub and its services including GitHub Actions have been in the news lately as they suffered a number of high-profile outages. While also arguably a tragedy of the commons problem, it is not what this note is about. If you prefer to be enraged about GitHub services, or the (relevant) lack thereof, this may not be for you.↩︎
The year is 2026 and politics is what it is, of course non-US alternatives emerged and will remain available and used. But dislodging established first-mover advantages will most likely take more than a (at least for now still-small) number of users unhappy for various (and sensible) reasons. We will see how this pans out.↩︎
Entire essays (or book) can be / will be / have been written about the competitive situation, how GitLab did not make enough of a dent, how Gitea remained niche and of course now Codeberg. This is not that essay, and I do not have a strong view but let me mumble a quiet plus ça change, plus ça reste la même chose …↩︎
