Linux cgroups has been used for a while to group processes and control their access to certain system resources. One of those resources is memory, through the use of memcg. That control is dynamic, allowing for cgroups to setup a minimal and a maximal amount of memory, as well as low and high thresholds, which will control the reclaim process.

But what about device memory that is not available as system RAM? Memory that is not available to the Linux page allocator? One of those device memories is GPU VRAM.

That's where the dmem cgroup controller comes in. Introduced in Linux 6.14, it was first adopted by the Intel Xe driver. In Linux 6.15, AMD amdgpu driver also adopted it. In 7.3, it was adopted by nouveau.

Such a cgroup controller allows the system to be configured such that a workload, be it a browser, a game, or another GPU rendering process can have some minimal VRAM available. Or it can prevent one of these processes from using all VRAM, preventing your compositor from using it, for example.

Let's go through the system knobs that are available to control this and what's new in Linux 7.3.

dmem regions and dmem.capacity

Since a given system might have multiple GPUs, there might be multiple device memory regions. So, on dmem, each such region is described by a name. Every dmem cgroup setting file will contain one line per region. One of those files shows the capacity of each region in bytes and is present only in the root cgroup.

After enabling the dmem cgroup with the following command:

# echo +dmem > /sys/fs/cgroup/cgroup.subtree_control

One can inspect the capacity of any available regions by looking at dmem.capacity at the root cgroup:

# cat /sys/fs/cgroup/dmem.capacity
drm/0000:03:00.0/vram 8573157376

That shows that the GPU at PCI address 0000:03:00.0 has a 8176MiB VRAM dmem region.

dmem settings

When introduced at 6.14, the following settings were available.

dmem.current

In every dmem cgroup, including the root cgroup, the dmem.current is a read-only file that shows the current usage of each region by the given cgroup.

# cat /sys/fs/cgroup/user.slice/dmem.current
drm/0000:03:00.0/vram 292757504

Here, it shows that user.slice cgroup is currently using above 292 MB of VRAM.

# cat /sys/fs/cgroup/dmem.current
drm/0000:03:00.0/vram 311377920

Here, we can check that total system usage is above 311 MB.

dmem.max

Excluding the root cgroup, dmem cgroups can control the maximal number of bytes that can be allocated from a region by processes in that cgroup. If an allocation moves the usage above the max setting, it will be refused. In that case, the allocation might be attempted at system memory or it might trigger eviction, depending on the allocation parameters and the driver.

Here, the max value has special meaning, which is unrestricted usage.

# echo drm/0000:03:00.0/vram 536870912 > /sys/fs/cgroup/user.slice/dmem.max
# cat /sys/fs/cgroup/user.slice/dmem.max
drm/0000:03:00.0/vram 536870912
# echo drm/0000:03:00.0/vram max > /sys/fs/cgroup/user.slice/dmem.max
# cat /sys/fs/cgroup/user.slice/dmem.max
drm/0000:03:00.0/vram max

dmem.min

dmem.min, also not available on the root cgroup, will control the eviction process - when memory needs to be moved out of VRAM to make room for other VRAM allocations. If an object belongs to a cgroup which is using less than its min setting, it will not be considered for eviction. It effectively works as a hard protection such that the cgroup can use at least the minimal setting.

Given multiple cgroups may set a minimal value that exceeds the device memory capacity, it computes an effective minimal depending on the hierarchy.

dmem.low

While dmem.min works as a hard limit, dmem.low is a soft limit that will prevent eviction in case other objects from different cgroups are available for eviction. That is, the system will try to evict from different, possibly unprotected cgroups, before it tries to evict from a cgroup if evicting would leave its usage below the value set as dmem.low.

Recent developments in Linux 7.3

Some new changes have been recently introduced in the dmem cgroup controller and its integration with GPUs.

One of those is a new read-only file, dmem.peak, which was introduced to allow the peak usage of a region in a cgroup to be inspected.

Also, Linux 7.3 fixed certain scenarios where a given cgroup would not be able to allocate VRAM up to its minimal or low limits if other processes had already allocated most of VRAM, even though they had no such limits set.

Changing multiple regions' settings in a single write was possible before 7.3, but it was prone to failures which would not be reported to userspace and might have left the values partially updated. Starting with 7.3, only one region may be updated per write.

And, finally, prior to 7.3, setting dmem.max in a cgroup that was using more than the new setting would silently fail. Now, there is a mechanism that allows a reclaim mechanism to be triggered. That way, when a process writes to dmem.max and sets a maximal value below the current usage, it will wait (unless O_NONBLOCK is used) for the reclaim process to move objects out of VRAM until the usage gets below the maximal value. If that is not possible for some reason, the setting will still be in place, restricting further alloctions by that cgroup.

The Chinese author Cixin Liu [1] is most known for his book The Three Body Problem and it’s sequels. One of the main plot elements in the book is that every species in the universe is cool with genocide and willing to commit genocide as a preemptive strike against any star system found to contain intelligent life.

It’s not unreasonable to think that some planets may be by run societies like that, if one star system in 10,000 had a technological society and one society in 10,000 is run by depraved autocratic governments that would make 1,000+ planets in our galaxy that are run by genocidal groups!

To think that every single society wants to commit genocide at every opportunity says a lot about every person who makes such claims. If you believe that everyone wants to do something then it must be something you really want to do. If there’s something you really don’t want to do then you will be able to imagine a system of government where the majority of citizens vote against doing it.

Even in the more horrible parts of human history there hasn’t been a genocide comparable to destroying and entire star system, genocide is usually based on slavery and large scale armed robbery. To just commit large scale genocide with a side effect of destroying valuable resources without any investigation or attempts at negotiation just based on fear of others is an extreme that I don’t think humans have reached in history.

The Daleks are portrayed as a perverse extreme of xenophobia but even with the low budget SciFi trend of depicting aliens as mostly having a common belief system they managed to show some Daleks going against their system from time to time. Cixin Liu is a very talented writer but his attitude towards genocide has caused him to write SciFi with a major plot point that’s not great by Dr Who standards.

Overall his short stories are much better than the three novels he is most known for. But he shows his ass in some of them too. One notable example is a short story where a significant plot point is declaring a “right to sex” (for men) which involves government sponsored sex workers. A “right to sex” conflicts with a sex workers right to refuse sex for a potential client who is undesirable in some way. Regardless of opinions on whether sex work should be permitted I hope that the vast majority of voters in every jurisdiction can agree that sex workers shouldn’t be forced to do things that they don’t want to do.

Some writers like George Orwell create deliberately fictional scenarios that they hope to inspire opposition to bad ideas, I’m sure that many proposed laws have failed to get enacted after being described as “Orwellian”. But describing a fictional horror situation as inevitable is a way of driving support for such things. It’s particularly noteworthy when written by a citizen of a country that’s involved in genocide.

A possible counter argument would be “but he is living in a repressive country and can’t oppose his government”. But when Hergé (the author of the Tintin graphic novels) was threatened about the consequences of writing another book like King Ottokar’s Sceptre [2] he didn’t write books in favour of authoritarianism, he just avoided the topic by writing about visiting the moon instead. I don’t think that people living in China now should be expected to publicly oppose evil things done by their government, but when they go out of their way to support such things it’s their choice and we should note that they made that choice.

I’m not saying that people shouldn’t read his work as his short stories are definitely worth reading. But the Three Body Problem series really isn’t that good, he prevented many potentially interesting situations from happening by having every interaction between different species be based around genocide or the potential for it. Being a horrible person also made him a less capable SciFi author.

• [1] https://en.wikipedia.org/wiki/Liu_Cixin
• [2] https://en.wikipedia.org/wiki/King_Ottokar%27s_Sceptre