Debugging shaders

I've recently tried to make some custom shaders. Very simple unlit color and unlit texture. I've ran into some issues. Mainly when the mesh trying to use the material had mismatched layout.

For example, I had SimpleMesh<PositionNormalTexture> and material with required layout InputLayoutType.Position | InputLayoutType.TextureCoordinate. Of course, this should not work, the expected buffer is obviously wrong, but is there a way to throw an exception or log a warning? 

Also is it possible to debug the shaders on per frame/pixel basis (using NSight or VS graphics debugger)? Since ab3d is embedded in WPF, it did not seem to correctly recognize the used device (it was trying to attach itself to D3D9 from WPF).

Thank you,
Janovsky Roman
DXEngine comes with a few samples on how to write your own shaders and effects. 
In Ab3d.DXEngine.Wpf.Samples those samples are: DXEngineAdvanced/CustomFogShaderEffectSample and DXEngineAdvanced/ShadedPointCloudSample. 
There is also a Ab3d.DirectX.ShaderFactory project where you can dynamically edit the shader's HLSL code and immediately see the changed shader in action.

But to get an even better understanding of how to build your own shader and effects, it is recommended to purchase the "Effects source code" for only $299. This will give you full source code of all the shaders (HLSL) and effects in the DXEngine. This will provide you will many building blocks that you can reuse for your own shaders.

Now to your questions:

You can check if the mesh has the correct layout in the overridden ApplyMaterial method in your own Effect class. This method gets renderableGeometry as a parameter and there you can check if the InputLayoutType is correct for your shader or use different shaders for different layouts. There you can throw an exception or provide any other feedback.

The DXEngine currently does not provide any checking for that because all meshes from the engine provide vertex buffers in PositionNormalTexture layout.

The reason why the DirectX 11 is not captured by the NSight or Visual Studio Graphics debugger is probably that the DirectX 11 device that is used by DXEngine was not created with debug flag.

To use that flag, set the CreateDebugDirectXDevice static field to true:
Ab3d.DirectX.DXDiagnostics.CreateDebugDirectXDevice = true;

After that I recommend you to use DirectXOverlay as PresentationType on the DXViewportView. This will use a SwapChain in the DXEngine and NSight or Visual Studio Graphics debugger will be immediately able to "attach" to the SwapChain and you will be able to capture individual frames.

If you want to use DirectXImage, the capture cannot be done from NSight or Visual Studio Graphics debugger because they will not be able to see the SwapChain. But you can still capture the image from those two applications. First start any of those two debugging application with your application attached. Then you will need to capture the frame from your application. This can be done by calling the following method:

This method can be called from Visual Studio Immediate Window.

An easier way to call this method is to use DXEngineSnoop tool (, attach to your application and then from the menu select "Capture next frame in VS".

To debug the shaders in HLSL you will need to compile the shaders with debug information.
I use the following additional parameters in the fxc.exe: "/Zi /Od /Gfp":
/Zi   - enable debugging information

/Od  - disable optimizations
/Gfp - prefer flow control constructs

You will not be able to debug the shaders that come with DXEngine because they are not compiled with debug information. But with "Effects source code" you can compile the DXEngine shaders with debug information and then debug them.

In this case you will need to make sure that DXEngine will load the shaders with debug information instead of shaders that are included in the DXEngine.dll. To do that, you can pack the compiled shaders in another dll as EmbeddedResources and then call the RegisterShaderResourceStatic static method on EffectsManager to load the shaders:

var resourceAssembly = this.GetType().Assembly;
var assemblyShaderBytecodeProvider = new AssemblyShaderBytecodeProvider(resourceAssembly, resourceAssembly.GetName().Name + ".Resources.Shaders.");


See the following help file to see other options on how you can provide your own shaders to the DXEngine:

Based on this answer, I have also written an article about this: Debugging DirectX 11 frame and shaders with Ab3d.DXEngine
Andrej Benedik
Thank you for the extensive answer. It did not occur to me, that simple cast to RenderablePrimitive would give me access to the object input layout. During testing it I found out, that a DXMeshGeometry3D with input layout type Position | Normal | TextureCoordinate was rendering correctly (eye-balling it) with RequiredInputLayoutType = Position. 

Is there some additional collection step after applying the material? For example in the case above, is there a check if at least the position is present and if so, it will create buffer from these positions, or just binds the whole vertex buffer with a stride for positions. Is RequiredInputLayoutType strict requirement, or is there an issue only in case where you have for example required layout with TextureCoordinate, but the mesh created was SimpleMesh<PositionNormal> (it's not possible to bind to anything).
I have tried different variants for what you suggested for debugging, this is what I found:
1. VS debugger is very unreliable. Sometimes connects correctly, sometimes infinitely loops on frame capture. On DXEngine Samples works perfectly without hicup.
2. NSight does cause issues since WPF is not officially supported. But works.
3. RenderDoc proved to be the easiest to setup.
4. Debugging worked for me with both debug device and non-debug device, but DirectXOverlay is required.
5. For quick info about draw calls and context switching, DXEngine snoop tool is the best.

Forum Jump:

Users browsing this thread:
1 Guest(s)