It is true that SharpDX packages are not being actively developed anymore, but because DirectX 11 API is also not changing they are 100% correct and stable. They can also work without any problems in .Net 5 applications.
The problem with WinUI is that Ab3d.PowerToys library depends on WPF 3D objects that are not present in WinUI - there is no MeshGeometry3D, GeometryModel3D and other 3D related objects; there is even no Point3D and Vector3D structs.
It would be possible to use only Ab3d.DXEngine without any WPF objects - using only classes from Ab3d.DXEngine.dll. You would need to work with low level SceneNode objects and define your own camera controllers. This would require a much bigger effort compared to when you can also use all the helper objects from Ab3d.PowerToys library.
Then it would be possible to create a shared texture that would be shared between Ab3d.DXEngine and WinUI.
But I am not planning to provide official support for that.
Instead, a new Vulkan based rendering engine is being developed. It will be truly cross-platform and will be able to integrate with many different UI engines - including WPF, WinForms, AvaloniaUI, Glfw and others. It will be able to run on Windows, Linux, Android, iOS and MacOS.
WinUI support is also on a wish list. It is great that WinUI supports multiple ways of using DirectX shared textures. Though I must say that I am very disappointed by Microsoft because the SurfaceImageSource control does not provide the SetDevice, BeginDraw and EndDraw methods from C# and forces you to use c++ there (don't they know that there are many managed libraries there to work with DirectX ?!?!). If there is anyone that knows how to use SurfaceImageSource control from C#, please send me a sample on how to do that - this will surely speed up adding support for WinUI.
And for a simple preview of the new engine, there is a screenshot of a simple test application that is running on Linux where the rendered 3D scene is composed with the UI elements in the AvaloniaUI application:
Vulkan rendering engine running in AvaloniaUI (pre-alpha version)
EDITED:
And another screenshot with more complex scene and test environment:
Vulkan test app (pre-alpha version)