I you have run out of memory for boxes, then you are probably creating BoxVisual3D or some other objects for each box instance.
To create many instances of the same object, you should use object instancing - InstancedMeshGeometryVisual3D.
There you define one MeshGeometry3D object and then send a list of instance data to the graphics card. This list defines the size, rotation, position (world transformation) and color of each instance. Then the graphics card can create 3D objects from that data and render the objects. This is much much faster.
See the "Instanced MeshGeometry3D" sample that comes with DXEngine.
You can easily change it to show millions of boxes:
First open the InstancedMeshGeometry3DTest.xaml and in line 65 change SelectedIndex to 0 (to initially show boxes).
<ComboBox Name="MeshTypeComboBox" SelectedIndex="2" Margin="0 0 15 0" SelectionChanged="OnSettingsChanged">
Then open InstancedMeshGeometry3DTest.xaml.cs and change the line 85 where the instanc data are prepared:
To show 1 million instances set the last three parameters to 100, 100, 100.
Instead of modelScaleFactor parameter, you can also set this value to 0.8 (so you will have smaller boxes):
InstanceData instancedData = CreateInstancesData(new Point3D(0, 200, 0), new Size3D(400, 400, 400), 0.8F, 100, 100, 100);
With that numbers I get amazing performance on my NVidia 960 GTX - render time is less then 0.2 ms (theoretically many thousands of frames per second).
Though when I change the number of instances to 10 million, the performance drops significantly - render time went to 100 ms (having only 6 frames per second). I would assume that with such number of instances graphics card had some problems with memory - therefore I think that if you would have a better graphics card with more memory (more then 4 GB), then you could still have more then 60 fps and still show 10 million boxes.
If you do not need to have a nice 3D object (if users will not zoom into one individual object), then you may use a very simple sphere instead of a box. To create sharp edges, one box requires 24 positions. But with crating a sphere mesh and setting segments count to 4, you can create a sphere with only 14 positions:
meshGeometry3D = new Ab3d.Meshes.SphereMesh3D(centerPosition: new Point3D(0, 0, 0), radius: 5, segments: 4, generateTextureCoordinates: false).Geometry;
If you have limitted camera movement and rotation so that all 3D objects are always seen from just one side, then you can optimize this even further with using PlaneMesh3D to create a simple plane with only 4 positions.
I do not know your use case but I would still recommend you that you rethink if you really need to show so many instances - maybe you can define a few groups of instances and then based on the camera position and direction show just a few of them. Maybe you can have some other way to decrease the number of instances - for example with showing one bigger box instead of many smaller boxes that are closer together. Note, that changing instance data can be slow so ideally you should prepare all instances data at the initialization time and do not change them afterwards. If you need to change individual instances data, I recommend dividing all instances into multiple instances - so that on change you do not need to change all the data but just those instances that have changes.
The ideal solution to your problem would be to create a special geometry shader that would get a list of positions and would create a simple 2x2 planes that would be always oriented towards the camera. This would require using some advanced features of the DXEngine to customize its rendering process. If the methods described before do not help, then after the new year I might find some additional time and may be able to add that to the engine - please contact me after the new year if you would really need that.
I hope that any of the proposed solution will help you.