One of the things we’re tackling as part of the STF infrastructure initiative is improving notifications. Other platforms have advanced significantly in this area over the past decade, while we still have more or less the same notifications we had since the early GNOME 3 days, both in terms of API and feature set. There’s plenty to do here

The notification drawer on GNOME 45

Modern needs

As part of the effort to port GNOME Shell to mobile Jonas looked into the delta between what we currently support and what we’d need for a more modern notification experience. Some of these limitations are specific to GNOME’s implementation, while others are relevant to all desktops.

Tie notifications to apps

As of GNOME 45 there’s no clear identification on notification bubbles which app they were sent by. Sometimes it’s hard to tell where a notification is coming from, which can be annoying when managing notifications in Settings. This also has potential security implications, since the lack of identification makes it trivial to impersonate other apps.

We want all notifications to be clearly identified as coming from a specific app.

Global notification sounds

GNOME Shell can’t play notification sounds in all cases, depending on the API the app is using (see below). Apps not primarily targeting GNOME Shell directly tend to play sounds themselves because they can’t rely on the system always doing it (it’s an optional feature of the XDG Notification API which different desktops handle differently). This works, but it’s messy for app developers because it’s hard to test and they have to implement a fallback sound played by the app. From a user perspective it’s annoying that you can’t always tell where sounds are coming from because they’re not necessarily tied to a notification bubble. There’s also no central place to manage the notification behavior and it doesn’t respect Do Not Disturb.

Notification grouping

Currently all notifications are just added to a single chronological list, which gets messy very quickly. In order to limit the length of the list we only keep the latest 3 notifications for every app, so notifications can disappear before you have a chance to act on them.

Other platforms solve this by grouping notifications by app, or even by message thread, but we don’t have anything like this at the moment.

Notifications grouped by app on the iOS lock screen

Expand media support

Currently each notification bubble can only contain one (small) image. It’s mostly used for user avatars (for messages, emails, and the like), but sometimes also for actual content (e.g. a thumbnail for the image someone sent).

Ideally what we want is to be able to show larger images in addition to avatars, as the actual content of the notification.

As of GNOME 45 we only have a single slot for images on notifications, and it’s too small for actual content. Other platforms have multiple slots (app icon, user avatar, and content image), and media can be expanded to much larger sizes.

There’s also currently no way to include descriptive text for images in notifications, so they are inaccessible to screen readers. This isn’t as big a deal with the current icons since they’re small and mostly used for ornamental purposes, but will be important when we add larger images in the body.

Updating notification content

It’s not possible for apps to update the content inside notifications they sent earlier. This is needed to show progress bars in notifications, or updating the text if a chat message was modified.

How do we get there?

Unfortunately, it turns out that improving notifications is not just a matter of standardizing a few new features and implementing them in GNOME Shell. The way notifications work today has grown organically over the years and the status quo is messy. There are three different APIs used by apps today: XDG Notification , Gio.Notification , and XDG Portal .

XDG Notification

This is the Freedesktop specification for a DBus interface for apps to send notifications to the system. It’s the oldest notification API still in use. Other desktops mostly use this API, e.g. KDE’s KNotification implements this spec.

Somewhat confusingly, this standard has never actually been finalized and is still marked as a draft today, despite not having seen significant changes in the past decade.

Gio.Notification

This is an API in GLib/Gio to send notifications, so it’s only used by GTK apps. It abstracts over different OS notification APIs, primarily the XDG one mentioned above, a private GNOME Shell API , the portal API, and Cocoa (macOS).

The primary one being used is the private DBus interface with GNOME Shell. This API was introduced in the early GNOME 3 days because the XDG standard API was deemed too complicated and was missing some features (in particular notifications were not tied to a specific app).

When using Gio.Notification apps can’t know which backend is used, and how a notification will be displayed or behave. For example, notifications can only persist after the app is closed if the private GNOME Shell API is used. These differences are specific to GNOME Shell, since the private API is only implemented there.

XDG Portal

XDG portals are secure, standardized system APIs for the Linux desktop. They were introduced as part of the push for app sandboxing around Flatpak, but can (and should) be used by non-sandboxed apps as well.

The XDG notification portal is based on the private GNOME Shell API, with some additional features from the XDG API mixed in.

XDG portals consist of a frontend and a backend. In the case of the notification portal, apps talk to the frontend using the portal API, while the backend talks to the system notification API. Backends are specific to the desktop environment, e.g. GNOME or KDE. On GNOME, the backend uses the private GNOME Shell API when possible.

How different notification APIs are used today

The plan

From the GNOME Shell side we have the XDG API (used by non-GNOME apps), and the private API (used via Gio.Notification by GNOME apps). From the app side we additionally have the XDG portal API. Neither of these can easily supersede the others, because they all have different feature sets and are widely used. This makes improving our notifications tricky, because it’s not obvious which of the APIs we should extend.

After several discussions over the past few months we now have consensus that it makes the most sense to invest in the XDG portal API. Portals are the future of system APIs on the free desktop, and enable app sandboxing. Neither of the other APIs can fill this role.

Our plan for notification APIs going forward: Focus on the portal API

This requires work in a number of different modules, including the XDG portal spec, the XDG portal backend for GNOME, GNOME Shell, and client libraries such as Gio.Notification (in GLib), libportal , libnotify , and ashpd .

In the XDG portal spec, we are adding support for a number of missing features:

• Tying notifications to apps
• Grouping by message thread
• Larger images in the notification body
• Special notifications for e.g. calls and alarms
• Clearing up some instances of undefined behavior (e.g. markup in the body, playing sounds, whether to show notifications on the lock screen, etc.)

This is the draft XDG desktop portal proposal for the spec changes.

On the GNOME Shell side, these are the primary things we’re doing (some already done in 46):

• Cleanups and refactoring to make the code easier to work on
• Improve keyboard navigation and screen reader accessibility
• Header with app name and icon
• Show full notification body and buttons in the drawer
• Larger notification icons (e.g. user avatars on chat notifications)
• Group notifications from the same app as a stack
• Allow message threads to be grouped in a single notification bubbles
• Larger images in the notification body

Mockups of what we’d ideally want, including grouping by app, threading, etc.

There are also animated mockups for some of this, courtesy of Jakub Steiner .

The long-term goal is for apps to switch to the portal API and deprecate both of the others as application-facing APIs. Internally we will still need something to communicate between the portal backend and GNOME Shell, but this isn’t public API so we’re much more flexible here. We might expand either the XDG API or the private GNOME Shell protocol for this purpose, but it has not been decided yet how we’ll do this.

What we did in GNOME 46

When we started the STF project late last year we thought we could just pull the trigger on a draft proposal Jonas for an API with the new capabilities needed for mobile. However, as we started discussing things in more detail we realized that this was the the wrong place to start. GNOME Shell already didn’t implement a number of features that are in the XDG notification spec, so standardizing new features was not the main blocker.

The code around notifications in GNOME Shell has grown historically and has seen multiple major UI redesigns since GNOME 3.0. Additional complexity comes from the fact that we try to avoid breaking extensions, which means it’s difficult to e.g. change function names or signatures. Over time this has resulted in technical debt, such as weird anachronistic structures and names. It was also not using many of the more recent GJS features which didn’t exist yet when this code was written originally.

Anyone remember that notifications used to be on the bottom? This is what they looked like in GNOME 3.6 (2012).

As a first step we restructured and cleaned up legacy code, ported it to the most recent GJS features, updated the coding style, and so on. This unfortunately means extensions need to be updated, but it puts us on much firmer ground for the future.

With this out of the way we added the first batch of features from our list above, namely adding notification headers , expanding notifications in the drawer, larger icons , and some style fixes to icons . We also fixed a very annoying issue with “App is ready” notifications not working as expected when clicking a notification ( !3198 and !3199 ).

We also worked on a few other things that didn’t make it in time for 46, most notably grouping notifications by app (which there’s a draft MR for), and additionally grouping them by thread (prototype only).

Throughout the cycle we also continued to discuss the portal spec , as mentioned above. There are MRs against against XDG desktop portal and the libportal client library implementing the spec changes. There’s also a draft implementation for the GTK portal backend .

Future work

With all the groundwork laid in GNOME 46 and the spec draft mostly ready we’re in a good position to continue iterating on notifications in 47 and beyond. In GNOME 47 we want to add some of the first newly spec’d features, in particular notification sounds, markup support in the body, and display hints (e.g. showing on the lock screen or not).

We also want to continue work on the UI to unlock even more improvements in the future. In particular, grouping by app will allow us to drop the “only keep 3 notifications per app” behavior and will generally make notifications easier to manage, e.g. allowing to dismiss all notifications from a given app. We’re also planning to work on improving keyboard navigation and ensuring all content is accessible to screen readers.

Due to the complex nature of the UI for grouping by app and the many moving parts with moving forward on the spec it’s unclear if we’ll be able to do more than this in the scope of STF and within the 47 cycle. This means that additional features that require the new spec and/or lots of UI work, such as grouping by thread and custom UI for call or alarm notifications will probably be 48+ material.

Conclusion

As we hope this post has illustrated, notifications are way more complex than they might appear. Improving them requires untangling decades of legacy stuff across many different components, coordinating with other projects, and engaging with standards bodies. That complexity has made this hard to work on for volunteers, and there has not been any recent corporate interest in the area, which is why it has been stagnant for some time.

The Sovereign Tech Fund investment has allowed us to take the time to properly work through the problem, clean up technical debt, and make a plan for the future. We hope to leverage this momentum over the coming releases, for a best-in-class notification experience on the free desktop. Stay tuned

In the 80s and 90s software development landscape was quite different from today (or so I have been told). Everything that needed performance was written in C and things that did not were written in Perl. Because computers of the time were really slow, almost everything was in C. If you needed performance and fast development, you could write a C extension to Perl.

As C was the only game in town, anyone could use pretty much any other library directly. The number of dependencies available was minuscule compared to today, but you could use all of them fairly easily. Then things changed, as they have a tendency to do. First Python took over Perl. Then more and more languages started eroding C's dominant position. This lead to a duplication of effort. For example if you were using Java and wanted to parse XML (which was the coolness of its day), you'd need an XML parser written in Java. Just dropping libxml in your Java source tree would not cut it (you could still use native code libs but most people chose not to).

The number of languages and ecosystems kept growing and nowadays we have dozens of them. But suppose you want to provide a library that does something useful and you'd like it to be usable by as many people as possible. This is especially relevant for providing closed source libraries but the same applies to open source libs as well. You especially do not want to rewrite and maintain multiple implementations of the code in different languages. So what do you do?

Let's start by going through a list of programming languages and seeing what sort of dependencies they can use natively (i.e. the toolchain or stdlib provides this support out of the box rather than requiring an addon, code generator, IDL tool or the like)

• C : C
• Perl : Perl and C
• Python : Python and C
• C++ : C++ and C
• Rust : Rust and C
• Java : Java and C
• Lua : Lua and C
• D : D, subset of C++ and C
• Swift : Swift, Objective C, C++ (eventually?) and C
• PrettyMuchAnyNewLanguage : itself and C

So even though C as a language is (most likely) going away, C APIs are not. In fact, designing C APIs is a skill that might even see a resurgence as the language ecosystem fractures even further. Note that providing a library with a C API does not mean having to implement it in C. All languages have ways of providing libraries whose external API is compatible with C. As an extreme example, Visual Studio's C runtime libraries are nowadays written in C++.

CapyPDF's design and things picked up along the way

One of the main design goals of CapyPDF was that it should provide a C API and be usable from any language. It should also (eventually) provide a stable API and ABI. This means that the ground truth of the library's functionality is the C header. This turns out to have design implications to the library's internals that might be difficult to add in after the fact.

Hide everything

Perhaps the most important declaration in widely usable C headers is this.

In C parlance this means "there is a struct type _someObject somewhere, create an alias to it called SomeObjectType ". This means that the caller can create pointers to structs of type SomeObject but do nothing else with them. This leads to the common "opaque structs" C API way of doing things:

This permits you to change the internal representation of the object while still maintaining stable public API and ABI. Avoid exposing the internals of structs whenever possible, because once made public they can never be changed.

Objects exposed via pointers must never move in memory

This one is fairly obvious when you think about it. Unfortunately it means that if you want to give users access to objects that are stored in an std::vector , you can't do it with pointers, which is the natural way of doing things in C. Pushing more entries in the vector will eventually cause the capacity to be exceeded so the storage will be reallocated and entries moved to the new backing store. This invalidates all pointers.

There are several solutions to this, but the simplest one is to access those objects via type safe indices instead. They are defined like this:

This struct behaves "like an integer" in that you can pass it around as an int but it does not implicitly convert to any other "integer" type.

Objects must be destructable in any order

It is easy to write into documentation that "objects of type X must be destroyed before any object Y that they use". Unfortunately garbage collected languages do not read your docs and thus provide no guarantee whatsoever on object destruction order. When used in this way any object must be destructable at any time regardless of the state of any other object.

This is the opposite of how modern languages want to work. For the case of CapyPDF especially page draw contexts were done in an RAII style where they would submit their changes upon destruction. For an internal API this is nice and usable but for a public C API it is not. The implicit action had to be replaced with an explicit function to add the page that takes both object pointers (the draw context and document) as arguments. This ensures that they both must exist and be valid at the point of call.

Use transactionality whenever possible

It would be nice if all objects were immutable but sadly that would mean that you can't actually do anything. A library must provide ways for end users to create, mutate and destroy objects. When possible try to do this with a builder object. That is, the user creates a "transactional change" that they want to do. They can call setters and such as much as they want, but they don't affect the "actual document". All of this new state is isolated in the builder object. Once the user is finished they submit the change to the main object which is then validated and either rejected or accepted as a whole. The builder object then becomes an empty shell that can be either reused or discarded.

CapyPDF is an append only library. Once something has been "committed" it can never be taken out again. This is also something to strive towards, because removing things is a lot harder than adding them.

Prefer copying to sharing

When the library is given some piece of data, it makes a private copy of it. Otherwise it would need to coordinate the life cycle of the shared piece of data with the caller. This is where bugs lie. Copying does cost some performance but makes a whole class of difficult bugs just go away. In the case of CapyPDF the performance hit turned out not to be an issue since most of the runtime is spent compressing the output with zlib.

Every function call can fail, even those that can't

Every function in the library returns an error code. Even those that have no way of failing, because circumstances can change in the future. Maybe some input that could be anything somehow needs to be validated now and you can't change the function definition as it would break API. Thus every function returns an error code (except the function that converts an error code into an error string). Sadly this means that all "return values" must be handled via out parameters.

This is not great, but such is life.

Think of C APIs as "in-process RPC"

• Wedding Anniversary; 22 years of happiness.
• Off to see R&A's new daughter; exciting. Caught up with blog in the car, up-loaded some slides on background save from COOL days.

Sometimes I build stuff other than software and this is a post about that.

My wife and I had to postpone our honeymoon for a couple years due to COVID. Last spring we were able to take a trip to Thailand and really enjoyed it. So much so that when we got back we had a desire recreate that sort of relaxed urban yet tropical feel we enjoyed so much in various cities.

We don’t have a lot of extra space at our house but we did have some in the back which was vacated by recently fell Elm which were diseased.

I’m not one to shy away from projects I’ve never done before so why not build a deck and pergola to fill out that space for entertaining and hacking?

The first step was to even the grade. When we bought the house it was already sloped but the grinding of Elm tree trunks added another layer on top of that.

That is a lot of work but to make something 8’×20′ we need to clear a lot more than this. Only to be made worse by root after root of dead Elm tree remnants. Pick axe and sweat.

That’s more like it.

We don’t have much of a frost line here so you don’t need to do the whole concrete-pylon thing for the deck and integrated pergola. But what we do need to do is ensure that we have a retaining wall to keep wet ground from touching the cedar for the deck.

The 2×4 cedar topper is mostly just there to match what will be the cedar floor of the deck.

A bunch of gravel of various grades under the cinders create something stable that drains water beneath the cedar 4×4 which will be our columns.

This is built as two 8’×10′ sections which are connected. That makes the math much easier in my head but also easier to source good wood. Above you see the rim and floor joists for the first side with the blocking yet to be done.

I lucked out that the cedar supplier gave me 12′ 4×4s instead of the requested 10′. That gave me 2′ extra to use when framing without having to first get the whole 10′ columns leveled. Later one we’ll take them out one-by-one and replace them with the columns.

Now we have the columns in place and the whole thing leveled. Blocking has started and will get completed before adding the floor boards. I ended up using screws at an angle from the rounded corners of the ceder floor boards using “camo hidden fasteners”. After seeing how they handled the winter that was definitely the right decision.

Thankfully I had help from my wife and emeritus GNOMie Cosimo with all this.

Now we have all the floor boards on. The cedar appearance board is tacked on the edge of the rim joist to hide the pressure treated lumber. The beams have been connected at the top of the columns. The start of a privacy wall is in place to see how it will fit together. Next up, all those rafters which I used some simple angle cuts and a multi-tool to extract pockets to interference-fit over the beams.

Lastly my wife did her design magic after gathering all the outdoor furniture together.

This year I have jasmine and honeysuckle doing it’s thing to hopefully start getting some of this pergola covered with leaves and flowers by summer. I also extended the privacy wall a bit more since this picture to have a lattice for the vines to climb.

That’s it!

For the last few months, Benjamin Tissoires and I have been working on and polishing a little tool called udev-hid-bpf [1]. This is the scaffolding required quickly and easily write, test and eventually fix your HID input devices (mouse, keyboard, etc.) via a BPF program instead of a full-blown custom kernel driver or a semi-full-blown kernel patch. To understand how it works, you need to know two things: HID and BPF [2].

Why BPF for HID?

HID is the Human Interface Device standard and the most common way input devices communicate with the host (HID over USB, HID over Bluetooth, etc.). It has two core components: the "report descriptor" and "reports", both of which are byte arrays. The report descriptor is a fixed burnt-in-ROM byte array that (in rather convoluted terms) tells us what we'll find in the reports. Things like "bits 16 through to 24 is the delta x coordinate" or "bit 5 is the binary button state for button 3 in degrees celcius". The reports themselves are sent at (usually) regular intervals and contain the data in the described format, as the devices perceives reality. If you're interested in more details, see Understanding HID report descriptors .

BPF or more correctly eBPF is a Linux kernel technology to write programs in a subset of C, compile it and load it into the kernel. The magic thing here is that the kernel will verify it , so once loaded, the program is "safe". And because it's safe it can be run in kernel space which means it's fast. eBPF was originally written for network packet filters but as of kernel v6.3 and thanks to Benjamin, we have BPF in the HID subsystem. HID actually lends itself really well to BPF because, well, we have a byte array and to fix our devices we need to do complicated things like "toggle that bit to zero" or "swap those two values".

If we want to fix our devices we usually need to do one of two things: fix the report descriptor to enable/disable/change some of the values the device pretends to support. For example, we can say we support 5 buttons instead of the supposed 8. Or we need to fix the report by e.g. inverting the y value for the device. This can be done in a custom kernel driver but a HID BPF program is quite a lot more convenient.

HID-BPF programs

For illustration purposes, here's the example program to flip the y coordinate. HID BPF programs are usually device specific, we need to know that the e.g. the y coordinate is 16 bits and sits in bytes 3 and 4 (little endian):

SEC("fmod_ret/hid_bpf_device_event")
int BPF_PROG(hid_y_event, struct hid_bpf_ctx *hctx)
{
	s16 y;
	__u8 *data = hid_bpf_get_data(hctx, 0 /* offset */, 9 /* size */);

	if (!data)
		return 0; /* EPERM check */

	y = data[3] | (data[4] << 8);
	y = -y;

	data[3] = y & 0xFF;
	data[4] = (y >> 8) & 0xFF;

	return 0;
}
  

As said above, this is device specific because where the coordinates is in the report depends on the device (the report descriptor will tell us). In this example we want to ensure the BPF program is only loaded for our device (vid/pid of 04d9/a09f), and for extra safety we also double-check that the report descriptor matches.

// The bpf.o will only be loaded for devices in this list
HID_BPF_CONFIG(
	HID_DEVICE(BUS_USB, HID_GROUP_GENERIC, 0x04D9, 0xA09F)
);

SEC("syscall")
int probe(struct hid_bpf_probe_args *ctx)
{
	/*
	* The device exports 3 interfaces.
	* The mouse interface has a report descriptor of length 71.
	* So if report descriptor size is not 71, mark as -EINVAL
	*/
	ctx->retval = ctx->rdesc_size != 71;
	if (ctx->retval)
		ctx->retval = -EINVAL;

	return 0;
}

This is pretty much it, the full working program only has a few extra includes and boilerplate. So it mostly comes down to compiling and running it, and this is where udev-hid-bpf comes in.

udev-hid-bpf as loader

udev-hid-bpf is a tool to make the development and testing of HID BPF programs simple, and collect HID BPF programs. You basically run meson compile and meson install and voila, whatever BPF program applies to your devices will be auto-loaded next time you plug those in. If you just want to test a single bpf.o file you can udev-hid-bpf install /path/to/foo.bpf.o and it will install the required udev rule for it to get loaded whenever the device is plugged in. If you don't know how to compile, you can grab a tarball from our CI and test the pre-compiled bpf.o. Hooray, even simpler.

udev-hid-bpf is written in Rust but you don't need to know Rust, it's just the scaffolding. The BPF programs are all in C. Rust just gives us a relatively easy way to provide a static binary that will work on most tester's machines.

The documentation for udev-hid-bpf is here . So if you have a device that needs a hardware quirk or just has an annoying behaviour that you always wanted to fix, well, now's the time. Fixing your device has never been easier! [3].

[1] Yes, the name is meh but you're welcome to come up with a better one and go back in time to suggest it a few months ago.
[2] Because I'm lazy the terms eBPF and BPF will be used interchangeably in this article. Because the difference doesn't really matter in this context, it's all eBPF anyway but nobody has the time to type that extra "e".
[3] Citation needed

We first introduced support for dmabufs and graphics offload last fall, and it is included in GTK 4.14. Since then, some improvements have happened, so it is time for an update.

Improvements down the stack

The GStreamer 1.24 release has improved support for explicit modifiers, and the GStreamer media backend in GTK has been updated to request dmabufs from GStreamer.

Another thing that happens on the GStreamer side is that dmabufs sometimes come with padding: in that case GStreamer will give us a buffer with a viewport and expect us to only show that part of the buffer. This is sometimes necessary to accommodate stride and size requirements of hardware decoders.

GTK 4.14 supports this when offloading, and only shows the part of the dmabuf indicated by the viewport.

Improvements inside GTK

We’ve merged new GSK renderers for GTK 4.14. The new renderers support dmabufs in the same way as the old gl renderer. In addition, the new Vulkan renderer produces dmabufs when rendering to a texture.

In GTK 4.16, the GtkGLArea widget will also provide dmabuf textures if it can, so you can put it in a GtkGraphicsOffload widget to send its output directly to the compositor.

You can see this in action in the shadertoy demo in gtk4-demo in git main.

Improved compositor interaction

One nice thing about graphics offload is that the compositor may be able to pass the dmabuf to the KMS apis of the kernel without any extra copies or compositing. This is known as direct scanout and it helps reduce power consumption since large parts of the GPU aren’t used.

The compositor can only do this if the dmabuf is attached to a fullscreen surface and has the right dimensions to cover it fully. If it does not cover it fully, the compositor needs some assurance that it is ok to leave the outside parts black.

One way for clients to provide that assurance is to attach a specially constructed black buffer to a surface below the one that has the dmabuf attached. GSK will do this now if it finds black color node in the rendernode tree, and the GtkGraphicsOffload widget will put that color there if you set the “black-background” property. This should greatly increase the chances that you can enjoy the benefits of direct scanout when playing fullscreen video.

In implementing this for GTK 4.16, we found some issues with mutter’s support for single-pixel buffers, but these have been fixed quickly.

To see graphics offload and direct scanout in action in a GTK4 video player, you can try the Light Video Player .

If you want to find out if graphics offload works on your system or debug why it doesn’t, this recent post by Benjamin is very helpful.

Summary

GTK 4 continues to improve for efficient video playback and drives improvements in this area up and down the stack.

A big thank you for pushing all of this forward goes to Robert Mader.

Perhaps the most interesting feature is that this new version reduces the number of external dependencies by almost 15%. More specifically the number of deps went from 7 to 6. This is due to Apple Clang finally shipping with std::format support so fmt::format could be removed. The actual change was pretty much a search & replace from fmt::format to std::format . Nice.

Other features include:

• L*a*b* color support in paint operations
• Reworked raster image APIs
• Kerned text support
• Support for all PDF/X versions, not just 3
• Better outline support

But, most importantly, there are now stickers:

Sadly you can't actually buy them anywhere, they can only be obtained by meeting me in person and asking for one.

In which I meet QA testers, bang my head against the GNOME OS initial setup process, and travel overland from Scotland to Spain in 48 hours.

Linux QA meetup

Several companies and communities work on QA testing for Linux distros, and we mostly don’t talk to each other. GUADEC 2023 was a rare occasion where several of us were physically collocated for a short time, and a few folk proposed a “GNOME + openQA hackfest” to try and consolidate what we’re all working on.

Over time, we realized ongoing lines of communication are more useful than an expensive one-off meetup, and the idea turned into a recurring monthly call. This month we finally held the first call. In terms of connecting different teams it was a success – we had folk from Canonical/Ubuntu, Codethink, Debian, GNOME, Red Hat/Fedora and SUSE, and there are some additional people already interested in the next one. Everyone who attended this round is using openQA and we will to use the openqa:opensuse.org chat to organise future events – but the call is not specific to openQA, nor to GNOME: anything Linux-related and QA-related is in scope.

If you want to be involved in the next one, make sure you’re in the openQA chat room, or follow this thread on GNOME Discourse . The schedule is documented here and the next call should be 08:00UTC on Thursday 2nd May.

GNOME OS tests

On the topic of QA, the testsuite for GNOME OS is feeling pretty unloved at the moment. Tests still don’t pass reliably and haven’t done for months. Besides the existing issue with initial setup where GNOME Shell doesn’t start , there is a new regression that breaks the systemd user session and causes missing sound devices . Investigating these issues is a slow and boring process which you can read about in great detail on the linked issues.

Fun fact: most of GNOME OS works fine without a systemd user session – there is still a D-Bus session bus after all; systemd user sessions are quite new and we still (mostly) support non-systemd setups.

One thing is clear, we still need a lot of work on tooling and docs around GNOME OS and the tests, if we hope to get more people involved. I’m trying my best in the odd hours I have available, greatly helped by Valentin David and other folk in the #GNOME OS channel, but it still feels like wading through treacle.

We particularly could do with documentation on how the early boot and initial setup process is intended to work – its very hard to visualize just from looking at systemd unit files. Or maybe systemd itself can generate a graph of what should be happening.

Magic in the ssam_openqa tool

Debugging OS boot failures isn’t my favourite thing. I just want reliable tests. Writing support tooling in Rust is fun though, and it feels like magic to be able to control and debug VMs from a simple CLI tool, and play with them over VNC while the test suite runs.

Using a VNC connection to run shell commands is annoying at times: it’s a terminal in a desktop in a VNC viewer, with plenty of rendering glitches, and no copy/paste integration with the host. I recently noticed that while openQA tests are running, a virtio terminal is exposed on the host as a pair of in/out FIFOs, and you can control this terminal using cat and echo . This feels like actual magic.

I added a new option to ssam_openqa , available whenever the test runner is paused, to open a terminal connection to this virtio console, and now I can debug directly from the terminal on my host. I learned a few things about line buffering and terminal control codes along the way. (I didn’t get ANSI control codes like cursor movement to work, yet – not sure if my code is sending them wrong, or some TERM config is needed on the VM side. – but with backspace, tab and enter working it’s already fairly usable).

Here’s a quick demo of the feature:

Available in the 1.2.0-rc1 release. Happy debugging!

Cross country travel

Most of this month I was on holiday. If you’re a fan of overland travel, how about this: Aberdeen to Santiago in 48 hours; via night train to Crewe, camper van to Plymouth, ferry to Santander and then more driving across to Galicia. A fun trip, although I got pretty seasick on the boat until I’d had a good nights sleep. (This wasn’t a particularly low-carbon trip though despite going overland, as the train, ferry and van are all powered by big diesel engines.)

And now back to regular work – “Moving files from one machine to another using Python and YAML”, etc.

(I need to put that somewhere because people ask about it and having a little post to explain it is nice.)

What’s it about?
GTK recently introduced the ability to offload graphics rendering , but it needs rather recent everything to work well for offloading video decoding.

So, what do you need to make sure this works?

First, you of course need a video to test. On a modern desktop computer, you want a 4k 60fps video or better to have something that pushes your CPU to the limits so you know when it doesn’t work. Of course, the recommendation has to be Big Buck Bunny at the highest of qualities – be aware that the most excellent 4000×2250 @ 60fps encoding is 850MB. On my Intel TigerLake, that occasionally drops frames when I play that with software decoding, and I can definitely hear the fan turn on.
When selecting a video file, keep in mind that the format matters.

Second, you need hardware decoding. That is provided by libva and can be queried using the vainfo tool (which comes in the `libva-utils` package in Fedora). If that prints a long list of formats (it’s about 40 for me), you’re good. If it doesn’t, you’ll need to go hunt for the drivers – due to the patent madness surrounding video formats that may be more complicated than you wish. For example, on my Intel laptop on Fedora, I need the intel-media-driver package which is hidden in the nonfree RPMFusion repository .
If you look at the list from vainfo , the format names give some hints – usually VP9 and MPEG2 exist. H264 and HEVC aka H265 are the patent madness, and recent GPUs can sometimes do AV1. The Big Buck Bunny video from above is H264, so if you’re following along, make sure that works.

Now you need a working video player. I’ll be using gtk4-demo (which is in the gtk4-devel-tools package, but you already have that installed of course) and its video player example because I know it works there. A shoutout goes out to livi which was the first non-demo video player to have a release that supports graphics offloading. You need GTK 4.14 and GStreamer 1.24 for this to work. At the time of writing, this is only available in Fedora rawhide, but hopefully Fedora 40 will gain the packages soon.

If you installed new packages above, now is a good time to check if GStreamer picked up all the hardware decoders. gst-inspect-1.0 va will list all the elements with libva support. If it didn’t pick up decoders for all the formats it should have (there should be a vah264dec listed for H264 if you want to decode the video above), then the easiest way to get them is to delete GStreamer’s registry cache in ~/.cache/gstreamer-1.0 .

If you want to make sure GStreamer does the right thing, you can run the video player with GST_DEBUG=GST_ELEMENT_FACTORY:4 . It will print out debug messages about all the elements it is creating for playback. If that includes a line for an element from the previous list (like `vah264dec` in our example) things are working. If it picks something else (like `avdec_h264` or `openh264dec`) then they are not.

Finally you need a compositor that supports YUV formats. Most compositors do – gnome-shell does since version 45 for example – but checking can’t hurt: If wayland-info (in the wayland-utils package in Fedora) lists the NV12 format, you’re good.

And now everything works.
If you have a 2nd monitor you can marvel at what goes on behind the scenes by running the video player with GDK_DEBUG=dmabuf,offload and GTK will tell you what it does for every frame, and you can see it dynamically switching between offloading or not as you fullscreen (or not), click on the controls (or not) and so on. Or you could have used it previously to see why things didn’t work.
You can also look at the top and gputop variant of your choice and you will see that the video player takes a bit of CPU to drive the video decoding engine and inform the compositor about new frames and the compositor takes a bit of CPU telling the 3D engine to composite things and send them to the monitor. With the video above it’s around 10% on my laptop for the CPU usage each and about 20% GPU usage.

And before anyone starts complaining that this is way too complicated: If you read carefully, all of this should work out of the box in the near future. This post just lists the tools to troubleshoot what went wrong while developing a fast video player.