Log in
Steinar H. Gunderson: Looking at dpkg startup time
Five years or so ago, I had a look at trying to speed up dpkg's package installation; I concluded that it was probably possible to speed up, but that there was no appetite for this kind of large-scale changes. (You'd probably need to rewrite the transaction system to get rid of a lot of fsyncs, you'd ideally want to reduce the number of syscalls for unpack by io_uring and so on.)
This summer, I've been looking at something related on and off; it is possible to speed up the startup time? That's in a sense the opposite scenario; instead of installing lots of packages in a newly debootstrapped chroot (with very few packages), see how fast you can install one in a much more busy chroot (I just copied my laptop's dpkg dir, with ~6600 packages installed).
Before I show the numbers, I must stress that this is an investigation, not a fair benchmark, and you should not go shout at the dpkg maintainers that they need to get to “catch up”. That said:
> sudo time dpkg --root=root -i hello_2.12.3-1_amd64.deb >/dev/null Not building database; man-db/auto-update is not 'true'. 1.12user 0.49system 0:01.85elapsed 87%CPU (0avgtext+0avgdata 171880maxresident)k 0inputs+14648outputs (0major+72963minor)pagefaults 0swaps > sudo time ./src/dpkg --root=root -i hello_2.12.3-1_amd64.deb > /dev/null 0.04user 0.01system 0:00.15elapsed 38%CPU (0avgtext+0avgdata 6520maxresident)k 0inputs+1080outputs (0major+2705minor)pagefaults 0swaps
How is it unfair? Well, for one, the code to run triggers is messed up so they're not run (but the trigger in question should be very fast). And there's one step at the end with detecting “disappearing packages” that doesn't run properly because it's a bit tricky in my model and I didn't want to deal with, well, difficult problems. But I think both are perfectly doable without really affecting the end time, it just requires engineering. There's a lot of work to be done, though; diving into the code makes me shudder at all the complexities that need to be in place to support all the corner cases of multiarch, for instance.
The code is extremely proof-of-concept, but it runs and can read (and write) metadata from SQLite instead of flat text files, it can resolve dependencies in the most basic fashion, it can keep track of installed files, it should be crash- and powerloss-proof. You know, the very very basic stuff, and without changing the model fundamentally (like e.g. Michael Stapelberg did with distri, fundamentally replacing packages with disk images and ending up in a very fast but rather different-looking system). So it was satisfying to see that it ends up around 10x even on my not-very-new laptop (plus a significant RAM reduction); I believe it should be possible to squeeze under 100 ms, but that would probably require also optimizing the unpacking itself, which I didn't look at this time.
Having a bunch of files being read into RAM and then processed freely was a design that made a lot of sense when dpkg was written (in 1995!) and Debian had ~250 binary packages in total (and you probably wouldn't install all of them). There was no reasonable database available for desktop systems; the closest thing you'd have was probably BerkeleyDB and that wasn't really it, so flat files and fsync made a lot of sense, and was easy to manipulate and persist. But now, SQLite is widely available and probably the most battle-tested code in history, a typical system has thousands of packages (you could easily install tens of thousands if you're doing heavy development), SSDs have replaced HDDs almost everywhere for system disks, and the environment has just changed a lot in general. So I hope that someone at some point will be crazy enough to pick this up and run with it, because it's a lot of work and I don't intend to. :-)
PS: I didn't look at apt; I think what I'd really love to see first and foremost is a package format change so that apt-listchanges can look at (or look for) NEWS.gz without having to unpack the entire package. Perhaps a control field saying “nothing new here”?
