[abenedik]

DXEngine comes with a few samples on how to write your own shaders and effects. 
In Ab3d.DXEngine.Wpf.Samples those samples are: DXEngineAdvanced/CustomFogShaderEffectSample and DXEngineAdvanced/ShadedPointCloudSample. 
There is also a Ab3d.DirectX.ShaderFactory project where you can dynamically edit the shader's HLSL code and immediately see the changed shader in action.

But to get an even better understanding of how to build your own shader and effects, it is recommended to purchase the "Effects source code" for only $299. This will give you full source code of all the shaders (HLSL) and effects in the DXEngine. This will provide you will many building blocks that you can reuse for your own shaders.


Now to your questions:

1)
You can check if the mesh has the correct layout in the overridden ApplyMaterial method in your own Effect class. This method gets renderableGeometry as a parameter and there you can check if the InputLayoutType is correct for your shader or use different shaders for different layouts. There you can throw an exception or provide any other feedback.

The DXEngine currently does not provide any checking for that because all meshes from the engine provide vertex buffers in PositionNormalTexture layout.

2)
The reason why the DirectX 11 is not captured by the NSight or Visual Studio Graphics debugger is probably that the DirectX 11 device that is used by DXEngine was not created with debug flag.

To use that flag, set the CreateDebugDirectXDevice static field to true:
Ab3d.DirectX.DXDiagnostics.CreateDebugDirectXDevice = true;

After that I recommend you to use DirectXOverlay as PresentationType on the DXViewportView. This will use a SwapChain in the DXEngine and NSight or Visual Studio Graphics debugger will be immediately able to "attach" to the SwapChain and you will be able to capture individual frames.

If you want to use DirectXImage, the capture cannot be done from NSight or Visual Studio Graphics debugger because they will not be able to see the SwapChain. But you can still capture the image from those two applications. First start any of those two debugging application with your application attached. Then you will need to capture the frame from your application. This can be done by calling the following method:
Ab3d.DirectX.DXDiagnostics.CaptureNextFrame(_dxView.DXScene);

This method can be called from Visual Studio Immediate Window.

An easier way to call this method is to use DXEngineSnoop tool (https://www.ab4d.com/DirectX/3D/Diagnostics.aspx), attach to your application and then from the menu select "Capture next frame in VS".


To debug the shaders in HLSL you will need to compile the shaders with debug information.
I use the following additional parameters in the fxc.exe: "/Zi /Od /Gfp":
/Zi   - enable debugging information

/Od  - disable optimizations
/Gfp - prefer flow control constructs


You will not be able to debug the shaders that come with DXEngine because they are not compiled with debug information. But with "Effects source code" you can compile the DXEngine shaders with debug information and then debug them.

In this case you will need to make sure that DXEngine will load the shaders with debug information instead of shaders that are included in the DXEngine.dll. To do that, you can pack the compiled shaders in another dll as EmbeddedResources and then call the RegisterShaderResourceStatic static method on EffectsManager to load the shaders:

Code:

var resourceAssembly = this.GetType().Assembly;
var assemblyShaderBytecodeProvider = new AssemblyShaderBytecodeProvider(resourceAssembly, resourceAssembly.GetName().Name + ".Resources.Shaders.");

EffectsManager.RegisterShaderResourceStatic(assemblyShaderBytecodeProvider);

See the following help file to see other options on how you can provide your own shaders to the DXEngine:
https://www.ab4d.com/help/DXEngine/html/...ovider.htm



Based on this answer, I have also written an article about this: Debugging DirectX 11 frame and shaders with Ab3d.DXEngine