+- AB4D Forum (https://forum.ab4d.com)
+-- Forum: Products Forums (https://forum.ab4d.com/forumdisplay.php?fid=4)
+--- Forum: Ab3d.PowerToys (https://forum.ab4d.com/forumdisplay.php?fid=9)
+--- Thread: Matrix camera (for augmented reality app) (/showthread.php?tid=4397)
Matrix camera (for augmented reality app) - SLH - 06-01-2023
Hi,
I'm trying to put together an AR demo based on examples from Opencv+opengl.
The sort of application I need to get working is https://rdmilligan.wordpress.com/2015/10/15/augmented-reality-using-opencv-opengl-and-blender/
I have the 4x4 view matrix (and also a projection matrix) but am struggling to work out how to get that information pushed to the cameras.
Thanks for any help!
Simon
RE: Matrix camera (for augmented reality app) - abenedik - 06-05-2023
Sorry, but there is currently no support for newer VR headsets. There is support for Oculus Rift (https://github.com/ab4d/Ab3d.OculusWrap), but this is probably outdated.
Also, there is no support for any Computer vision library (e.g. OpenCV).
RE: Matrix camera (for augmented reality app) - SLH - 06-05-2023
Hi, Thanks for the response.
I *think* I just need some help in porting over the code for setting the view/projection matrix - or similar.
Opencv gives us the rotation and translation vector for the pose of the camera. In Opengl, these can just be manipulated a little to build matricies for adjusting the view/projection matrix. But I can't see how to do this in AB4D.
I can however get the roll/pitch/yaw and position for the camera, but again, not sure how to get these values to work with any of the cameras available.
I've sent a request to AB4D contact page about purchasing support. Maybe we move the conversation there?
RE: Matrix camera (for augmented reality app) - abenedik - 06-05-2023
I see now that I have misunderstood the question: there is no need for VR support and you already have figured out how to get data from OpenCV.
In case you only need to set view and projection matrices to the camera, then you should not create a TargetPositionCamera (or any other camera from Ab3d.PowerToys) and you should not create the CameraMouseController.
Instead you can use the WPF's Matrix camera:
Code:
var matrixCamera = new MatrixCamera(viewMatrix, projectionMatrix);
MainViewport.Camera = matrixCamera;
// Update the camera:
matrixCamera.ViewMatrix = ...RE: Matrix camera (for augmented reality app) - SLH - 06-10-2023
Just an update: although I thought I had everything working I don't quite. Will post some code for review when I get the time.
RE: Matrix camera (for augmented reality app) - SLH - 06-12-2023
[More attachments in following post]
Here's my progress so far...
The test object should be a blue box the size of the marker, with a red box indicating the centre and then a grid of spheres just to show a bunch of positions in 3D space.
The translation tracking appears to be working (the red square is always in the centre of the marker). But the rotations are wrong. It looks like I need to add a rotation around one axis or flip an axis. But I have no idea where or how.
Here is my code for the important bits. Hopefully someone can spot an obvious bug!
View matrix code from the pose provided by openCV. cvRotation is the rodrigues function's output from the rotation. It is 3x3 matrix. cvT is the translation:
Code:
private SharpDX.Matrix OpenCVToWPFViewMatrix(Mat cvRotation, Mat cvT)
{
var srcRot = cvRotation;
CvInvoke.Invert(srcRot, srcRot, Emgu.CV.CvEnum.DecompMethod.LU);
var r = Mat3x3ToSharpDXMatrix(srcRot);
var t = Mat3x1ToSharpDXVector(cvT);
var viewMatrix = new SharpDX.Matrix(
0, 0, 0, 0,
0, 0, 0, 0,
0, 0, 0, 0,
0, 0, 0, 0);
for (int i = 0; i < 3; i++)
{
for (int j = 0; j < 3; j++)
{
viewMatrix[i, j] = r[i, j];
}
viewMatrix[i, 3] = t[i];
}
viewMatrix[3, 3] = 1;
//this bit here is to flip signs etc. 99% sure here is where the error lies
var correctionMatrix = new SharpDX.Matrix(
1, 1, 1, 1,
1, 1, 1, 1,
1, 1, 1, -1,
1, 1, 1, 1);
for (int i = 0; i < 4; i++)
{
for (int j = 0; j < 4; j++)
{
viewMatrix[i, j] = viewMatrix[i, j] * correctionMatrix[i, j];
}
}
viewMatrix.Transpose();
return viewMatrix;
}projection matrix code, again, likely something is backwards or flipped in here:
Code:
private SharpDX.Matrix OpenCVTOWPFProjectionMatrix(Size imgSize)
{
var camMtx = this.CameraCalibration.CameraMatrix;
var cameraModel = this.CameraCalibration.GetSensorInfo(imgSize);
var near = 0.00125f;
var far = 10000;
var fx = (float)camMtx.at(0, 0);
var fy = (float)camMtx.at(1, 1);
var cx = (float)camMtx.at(0, 2);
var cy = (float)camMtx.at(1, 2);
var width = (float)imgSize.Width;
var height = (float)imgSize.Height;
var f = (float)(0.5d * height * Math.Tan(fy / 2));
var m = new SharpDX.Matrix(1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1); ;
m[0, 0] = 2.0f * fx / width;
m[0, 1] = 0.0f;
m[0, 2] = 0.0f;
m[0, 3] = 0.0f;
m[1, 0] = 0.0f;
m[1, 1] = -2.0f * fy / height;
m[1, 2] = 0.0f;
m[1, 3] = 0.0f;
m[2, 0] = 1.0f - 2.0f * cx / width;
m[2, 1] = (2.0f * (cy / height)) - 1.0f;
m[2, 2] = (far + near) / (near - far);
m[2, 3] = -1.0f;
m[3, 0] = 0.0f;
m[3, 1] = 0.0f;
m[3, 2] = 2.0f * far * near / (near - far);
m[3, 3] = 0.0f;
return m;
}Code to convert to WPF matrix and back. Too many formats of Matrix in this project - maybe here I am getting something wrong?
Code:
private SharpDX.Matrix3x3 Mat3x3ToSharpDXMatrix(Mat r)
{
SharpDX.Matrix3x3 m = new SharpDX.Matrix3x3();
for (int i = 0; i < 3; i++)
{
for (int j = 0; j < 3; j++)
{
m[i, j] = (float)r.at(j, i);
}
}
return m;
}
private SharpDX.Matrix3x3 Mat3x1ToSharpDXVector(Mat r)
{
SharpDX.Matrix3x3 m = new SharpDX.Matrix3x3();
m.M11 = (float)r.at<double>(0);
m.M12 = (float)r.at<double>(1);
m.M13 = (float)r.at<double>(2);
return m;
}
private Matrix3D SharpToWPF(SharpDX.Matrix src)
{
var m = new Matrix3D();
m.M11 = src.M11;
m.M12 = src.M12;
m.M13 = src.M13;
m.M14 = src.M14;
m.M21 = src.M21;
m.M22 = src.M22;
m.M23 = src.M23;
m.M24 = src.M24;
m.M31 = src.M31;
m.M32 = src.M32;
m.M33 = src.M33;
m.M34 = src.M34;
m.OffsetX = src.M41;
m.OffsetY = src.M42;
m.OffsetZ = src.M43;
m.M44 = src.M44;
return m;
}RE: Matrix camera (for augmented reality app) - SLH - 06-12-2023
A couple more pictures of the result.
NB. The drawn axis lines are from Opencv code. So I can see the coordinates of the marker there.
As you can sort of tell, the rotation around one axis is off.
The code is largely attempted ports of opengl example code. Am I getting some weird left-right handed coodinate system error? If so, how do I fix?
RE: Matrix camera (for augmented reality app) - abenedik - 06-14-2023
If you have the matrices in left-coordinate system, then you should convert them to DirectX / WPF 3D right-handed coordinate system (x - right, y - up, z - towards the viewer).
You should probably do that by multiplying by the following matrix:
Code:
// The transformation defines the new axis - defined in matrix columns in upper left 3x3 part of the matrix:
// x axis: 1st column: 1 0 0 (in the positive x direction - same as WPF 3D)
// y axis: 2nd column: 0 0 -1 (in the negative z direction - into the screen)
// z axis: 3rd column: 0 1 0 (in the positive y direction - up)
var matrix = new Matrix3D(1, 0, 0, 0,
0, 0, -1, 0,
0, 1, 0, 0,
0, 0, 0, 1);I cannot test that so I cannot say if this will work. As you see from the values, this matrix preserves the x value, swaps the x and z value and negates the z value. The offset (the last row) is preserved.
It may be also possible that you would need to transpose the matrix (some matrices are defined as row-first, and some as column-first). But if this would be the case, then you would probably not see anything - the scene would be too distorted.
I hope that this will solve your problem.