[Feature request] Wayland compositor side extension

[nyorain]

Hey i've got a feature request for an extension similiar to a combination of EGL_WL_bind_wayland_display provided by the mesa implementation (not an official extension) and EGL_KHR_image_base.

It should make wayland compositors able to retrieve vulkan images from wayland client buffer resources that are managed by the same drivers egl or vulkan client implementation.

If a wayland client uses vulkan (and the wayland wsi) the only way for the compositor to get the contents of this buffer is the egl extension. Therefore, one cannot write an efficient vulkan compositor backend which is really a pity considering that vulkan has (at least in theory) with VK_EXT_KHR_display a really nice and convinient api for direct rendering on linux.

I wrote down some first ideas how this extension could look like in vulkan.
I am not that familiar with the khronos extension naming scheme, so i am not sure what extension suffix it would get. KHR like the other WSI extensions?

To bind a wayland display to a vulkan instance (i.e. creating a wl_drm global) or to unbind it:

VkResult vkBindWaylandDisplay<Suffix>(VkInstance instance, struct wl_display *display);
VkResult vkUnbindWaylandDisplay<Suffix>(VkInstance instance, struct wl_display *display);

To check if a wayland client resource can be transformed into a vulkan image (i.e. if the resource is managed by the wl_drm):

vkResult vkQueryWaylandBuffer<Suffix>(
    VkInstance instance, 
    struct wl_resource *buffer, 
    VkWaylandBufferAttribute<Suffix> attribute, 
    uint32_t *value);

This kind of querying seems like not used anymore in vulkan, so it could also be done with a structure holding all the queryable information about a buffer (which would be something like size and format):

VkResult vkGetWaylandBufferProperties<Suffix>(
    VkInstance instance, 
    struct wl_resource *buffer,
    VkWaylandBufferProperties *pProperties);

Then just a function to create a vulkan image from a valid client buffer.
The VkWaylandBufferImageCreateInfo structure would contain the wl_resource* and some attributes like in egl.

VkResult vkCreateWaylandBufferImage<Suffix>(
    VkDevice device,
    VkWaylandBufferImageCreateInfo<Suffix> *pCreateInfo,
    VkAllocationCallbacks *pAllocator,
    VkImage *pImage);

Mesas intel driver already has a branch with some first implementation of something like this. But IMO it would really make sense to have a driver-independent extension.

If this is not the right place for a request like this or it is totally inappropriate please just let me know.
Thank you.

[cubanismo]

I don't think trying to map the EGL constructs directly into Vulkan is the right way to handle this. Instead, there should be a way to implement a Wayland Vulkan swapchain on top of purely Vulkan objects using Vulkan system layer, and then there wouldn't need to be any special visibility into the Vulkan objects. The Wayland compositor then wouldn't need any special extensions beyond sharing Vulkan images across processes. The need for the EGL extensions is just a side effect of the limitations of EGL's design. Vulkan has a far more flexible WSI and core design.

[nyorain]

I agree that shared Vulkan images would be of great use for writing Wayland compositors but even then they must be somehow integrated with the Wayland server side. The compositor must have some way to query whether it can retrieve a Vulkan image from a clients buffer (i.e. if the client uses EGL or Vulkan) which is not possible at the moment. You are right that there might be better ways to do this in Vulkan than just converting the (non standard) EGL extensions to Vulkan extension, this was more an example to show which functionality would be needed to allow compositors to use Vulkan.

[cubanismo]

Agreed. With EGL, the type of the buffer (DRM buffer, some other accelerated buffer type) was necessarily driver specific as there was no standardized cross-process image type, so it made sense to make that driver functionality. With Vulkan, there should be a standard way to represent Vulkan images outside of the driver and across process boundaries, so it shouldn't be necessary for wayland to embed itself into the Vulkan driver just to import/export buffers and info about them. Stay tuned for more info on this, as hinted by issue #205.

To handle the case of EGL clients talking to a Vulkan server, I hope a vendor-agnostic Vulkan layer rather than an in-driver hook can be used by the wayland compositor/server. That will still require an extension for the layer to expose its API, but I think it will look a bit different than the example linked here.

Note I don't necessarily speak for the entire Khronos working group. These are my design opinions.

[fooishbar]

Instead, there should be a way to implement a Wayland Vulkan swapchain on top of purely Vulkan objects using Vulkan system layer, and then there wouldn't need to be any special visibility into the Vulkan objects.

Kind of. If you're fluent in Android, a wl_buffer is the equivalent to a GraphicBuffer/ANativeBuffer; it is an opaque handle that the server receives, which it requires the native graphics stack to turn into something the compositor can texture from: in EGL, this is EGLImage. The underlying primitive is a single frame, and the general opinion of the upstream Wayland project when EGLStreams was raised (disclaimer: I was the one most strongly arguing this point) was that a stream/swapchain model which removed the compositor's ability to address individual buffers, was inappropriate in a compositor context. The Wayland WSI swapchain model presently seems to work fine for clients, however.

Given that, and the lack of a standardised ICD or HAL model for non-Android Linux driver deployments, we chose to implement the driver support inside the EGL library, which necessarily requires knowledge of Wayland primitives (i.e. 'I would like you to provide accelerated-rendering functionality on this display'). I'm more than happy to discuss this at greater length.

With Vulkan, there should be a standard way to represent Vulkan images outside of the driver and across process boundaries, so it shouldn't be necessary for wayland to embed itself into the Vulkan driver just to import/export buffers and info about them.

Yes, a hypothetical standardised cross-process image export and exchange API would resolve the majority of these issues. It does change the Wayland model and arguably make containerisation and isolation more difficult, but that's a separate issue to address I believe.

(Similarly, I don't necessarily speak for either the Wayland project, nor Khronos.)

[tomek-brcm]

You'd probably need at least 2 hypothetical standardised cross-process primitives for that: image + semaphore in Vulkish (or buffer + fence in Androidish) .

The WSI model in Vulkan looks to me like a half of a solution. Vulkan application can sit on top of a windowing system of sorts but it can't really be used to implement a windowing system(*). You can create WSI surface as an "output" abstraction but I couldn't find an "input" counterpart. Vulkan-client can write to a "window" but Vulkan-compositor can't read from such "window".

And that's only a first part of the solution. The 2nd part is writing and reading in an orderly fashion. Android uses cross-process fences sent over binder API. Vulcan has fences semaphores and events but none of them seem to be suitable to cross a process boundary. Finally Wayland has nothing.

Currently GLES/EGL implemntation of Wayland compositor and clients is based on a silent assumption that driver has an implicit cross-process synchronisation. This is not GLES or EGL requirement, just an unwritten Wayland assumption. Vulkan is all about explicit synchronisation that is responsibility of an application but it doesn't provide any cross-process synchronisation primitives. This is, to my untrained eye, a blocker issue.

Finally, if you intend to to mix and match client APIs, for example run a Vulcan client on GLES compositor, or perhaps the other way around, then even a cross-process Vulcan image + semaphore won't cut it. For that you'd need some sort system-level cross-API primitives, hopefully without eglImage craziness.

(*) Well, technically one can memcpy between host-mapped Vulkan image and shm and than memcopy again in the compositor but I don't count that as a reasonable implementation

[nyorain]

Wayland has no concepts like fences or semaphores in its core protocol to synchronize access to resources because it is not needed. It does not assume that drivers synchronize the access to the buffers internally, it has a well defined mechanism of buffer ownership and simply offers the possibility for drivers to implement the wl_drm interface on client as well as on compositor side.
EGL (with the mentioned mesa extension) implements them on both sides while Vulkan (WSI) does so only on the client side and trusts that compositors use EGL anyways to get the buffers contents.
Without EGL no compositor would be able to support Vulkan clients at the moment (besides dealing with the raw drm buffers and implementing wl_drm).

But we should simply wait and see what the next Vulkan API updates bring. Some kind of shared images would solve this request in a really nice way although we would still need some way to connect this feature with the Wayland protocol or is there any other way? How should the compositor know of the Vulkan image a client attached to a surface?

[tomek-brcm]

Wayland has no concepts like fences or semaphores in its core protocol to synchronize access to resources because it is not needed.

Of course it IS needed if GPU is involved (GLES/EGL case, see http://ppaalanen.blogspot.co.uk/2012/03/what-does-egl-do-in-wayland-stack.html). Bear in mind that CPU is always ahead of GPU. The fact that CPU has finished issuing draw commands and handed over a buffer to the other side doesn't mean that GPU has finished drawing. Also GLES/EGL only guarantees synchronisation wihin a single context. Any cross-context synchronisation must be done explicitly by an application . A cross-process synchronisation is completely out of scope as far as GLES and EGL or Vulkan are concerned.

It does not assume that drivers synchronize the access to the buffers internally

In fact it does.

it has a well defined mechanism of buffer ownership

Perhaps it's well defined but certainly it isn't documented.

and simply offers the possibility for drivers to implement the wl_drm interface on client as well as on compositor side.

Well, only if DRM is in use. Wayland doesn't mandate DRM. Wayland only mandates some platform-specific wl_buffer that is handed between client and compositor.

The platform implementation of the eglSwapBuffers() calls the "attach" to hand buffer from client to compositor. At this stage GPU is likely to be busy with client's requests. Compostitor starts texturing from that buffer without any means of knowing whether the GPU has actually finished drawing client's stuff. This is the 1st implicit synchronisation point assumed by Wayland: GPU texture reads requested by the compositor process will wait for GPU writes requested by client in a different context from a different proces without any explicit action.

The same principle applies when compositor releases buffer back to client. The client is likely to receive the release event while GPU is still busy drawing compositor's stuff. This is the 2nd implicit synchronisation point: any client-side GPU writes to such released buffer must be deferred until compositor's draw commands have finished sampling the previous contents of that buffer. Again you have 2 completely independet GL contexts from different processes that somehow synchronise their access to a shared buffer.

When you look at Vulkan, the whole API is about explicit synchronisation at every step. I doubt that Wayland client or compositor written in Vulkan can assume that it's safe to render to and texture from a shared image and Vulkan driver will automagically synchronise both process on CPU and GPU side without any Wayland involvement.

I understand that Wyland doesn't actually mandate any hardware acceleration, but in the scenarios where GPU gets involved, there are only 2 options that don't depend on the implicit synchronisation: stalling the CPU on both "attach" and "release" or sending buffer with a GPU synchronisation primitive of some sort. Stalling the CPU isn't really an option and there are no cross-process GPU synchronisation primitives as far as I can tell.

[nyorain]

The Wayland core protocol clearly specifies that after a wl_surface_commit request the compositor may read the pixels at any time and after a wl_buffer_release event the compositor must not further access the pixels. The client is not allowed to use the buffer while the compositor uses it.
I only mentioned wl_drm because it is the current way of all major drivers to implement hardware accelerated buffers in Wayland. You are right that drivers could use any (private) interface but they would still have to implement the behaviour specified by the Wayland protocol for their buffers.
But this is not the right place to discuss the Wayland protocol.

In Vulkan, one must ensure that all rendering commands have finished before a swapchain image can be presented via vkQueuePresentKHR. This is not necessarily about stalling the cpu but more about simply not committing a buffer that is still used by the client. Therefore no extra synchronization is needed between the client and compositor side (always feel free to correct me if I understood it wrong).

Although I dont see any problems in this (since one side has to wait for the rendering to finish, moving this waiting to the compositor side only introduce unnecessary complexity), I agree with you that some method for presenting/sharing Vulkan images using explicit synchronization would improve the WSI even more.

[tomek-brcm]

If that was he case, Weston, the reference compositor implementation, would be plainly wrong.

The problem is that CPU deals with Wayland calls while GPU deals with pixels. In order to fulfil such strict pixel access guarantees Weston would have to call glFinish() or glClientWaitSync() or vkWaitForFences(). But Weston doesn't do such thing for a good reason - in this scenario CPU and GPU must wait for each other and work in turns while in fact they're entirely capable to work in parallel.

[philiptaylor]

In Vulkan, one must ensure that all rendering commands have finished before a swapchain image can be presented via vkQueuePresentKHR.

I don't think that's an accurate description - you only have to ensure all rendering commands will have finished executing before the device signals the pWaitSemaphores that were passing into vkQueuePresentKHR. The vkQueuePresentKHR can be called (and can return) a long time before the rendering has completed, and the WSI implementation is responsible for synchronising the presentation with those semaphores.

[nyorain]

Yes you are right, that was an oversimplification. The point I wanted to make was that the presentation of the image (in case of Wayland the wl_surface. commit call) should not happen until the rendering onto the image is finished if I understood the specification correctly.

[tomek-brcm]

This is not how Weston does it, so it might be an oversimplification in the Wayland spec. I'd be interested to hear from Wayland guys what would be the intended use of vkSemaphore passed to the "present" command if the presentation engine happens to be Wayland compositor.

Delaying the surface commit until GPU has signalled the semaphore is a way of doing it, but it's not an efficient way of doing it. Actually in Vulkan it's also not the most straight forward way because Vulkan API doesn't have CPU-side wait on vkSemaphore.