[abenedik]

As it is shown in the LineSelector sample, the CalculateScreenSpacePositions method has two overloads:
one that takes BaseCamera (used in line 149) and one that takes worldToViewportMatrix (used in line 143):

Because you are using a MatrixCamera, you will need to use the second option.

You can calculate worldToViewportMatrix by using the following:

Code:

var worldToViewportMatrix = viewMatrix * projectionMatrix * GetViewportMatrix(new Size(MainViewport.ActualWidth, MainViewport.ActualHeight));

_lineSelectorData.CalculateScreenSpacePositions(ref worldToViewportMatrix, transform: null);


public static Matrix3D GetViewportMatrix(Size viewportSize)
{
    if (viewportSize.IsEmpty)
        return Ab3d.Common.Constants.ZeroMatrix;
    return new Matrix3D(viewportSize.Width / 2, 0, 0, 0,
        0, -viewportSize.Height / 2, 0, 0,
        0, 0, 1, 0,
        viewportSize.Width / 2,
        viewportSize.Height / 2, 0, 1);
}