[BillC]

I'm looking for a way to view 100 million points in a 2D view and then show a subset in 3D. I don't know if DXEngine can help with the 2D views.

I can't believe I'm asking this, but I found out late last week that what I thought would be several 100 thousand points is really 100's of millions of points.

Right now I'm trying to figure out if this stuff is even possible.

[abenedik]

Uh, 100's of millions of points is really a lot.

For example a full HD display have only 1920 * 1080 = 2.073.600 pixels.

My advice would be to implement some kind of reduction algorithm where you would group pixeles into groups and show groups with pixels or spheres where the radius would represent the quantity (for example r = 10 would mean 100.000 items).

You might also try to use WritableBitmap and get a direct access to the bitmap's memory - then for each data item calculate its position and color and write the color data to the bitmap's memory (4 bytes for one color). You can probably do that in the background thread and show some waiting animation to the user. Here using unsafe code could improve performance; c++ would help even more. Keep in mind to try to prepare your data in a memory efficient way.


Theoretically it would be possible to use DXEngine to display pixels - you would create many 3D rectangles (one rectangle for one position) and use OrthographicCamera to make positioning rectangles in 3D space easier - for example if DXViewportView's Width is 1000 pixels and you set CameraWidth on OrthographicCamera to 1000, then a 3D rectangle with SizeX = 1 would be 1 pixel width. But it would take too long to create so much rectangles that you need.

In the future I plan to add support for Sprites. They could be used to create many 2D rectangles in the 3D world. They can be also very fast because the 3D rectangles are generated on the graphics card. But still having 100's of millions of them would not work well.

[BillC]

Yep. 15 miles of 4-lane highway on a 1/2 inch grid. On the plus side I don't need to do triangles.

By the time I'm done it should be in excess of 250 million pts since I do a before and after. The concession I got was it didn't have to run on a laptop.

There is wanting, and then there is getting.

[abenedik]

This sounds like a very interesting challenge :)

I started working on adding support for rendering many individual pixels in DXEngine. I am also curious how many pixels can be shown.

So, if I understand you correctly, you have 250 million 3D positions and each probably have its own value that will be converted into appropriate pixel color. And you would like to show the positions as 2x2 (or 1x1) pixels. Please provide some additional details so I could better prepare a demo for you.


By the way, if you would like to show 3D lines, you can already render many (millions) of 3D lines. You can check that with using PolyLinesSample that comes with DXEngine samples project. To make the ultimate performance, uncheck the "Use geometry shader" and "Antialiased 3D lines" CheckBoxes. Also notice that performance is much better when you render a few Polylines with many lines then when rendering many Polylines with a few lines - each Polyline creates its own Draw call - the goal is to minimize the number of Draw calls.

My tests show that if you have a very good graphics card, you can render many millions of 3D lines. But at some point the performance drops from very good to very bad - I guess that this happens when graphics card runs out of its memory and the data needs paged to the main CPU memory. So I assume that for rendering huge amount of 3D lines, you need a graphics card with a lot of fast memory.


I will write more when I will have some results with rendering pixels.

[BillC]

You have described it well. 250M pts on about 10 layers (10 colors). 1x1 might work, if that doesn't work (can't distinguish individual pixels) then 2x2. It's not about seeing them ALL. Well it could be I guess, but that would be seeing the whole and not the pieces. Its about having them all so the user can choose what to see.

A friend described a system he used where he had a Billion pts but the system let him set the display to 100M. That could be an option, limiting how much gets displayed tho I'm not sure how one does that.

What I need is to see the data in 2D views (Plan, profile and Cross-section) then be able to push some to 3D. I had thought to attack the problem by creating a 3D model then limiting the camera to top view for Plan.

But I need to come up with something else for Profile and Cross-section. A thought was to build those as very thin 3D views with width and height but a single depth. These '2D' views would have no rotation, just zoom and pan. You have samples that do something like this. You do 3D graphs, but I'd thought it would be fairly simple to limit it to a 'thin' 3D. These graphs would need much smaller amounts of data (a few lines with 100,000 locations each)

I had also thought to limit the 3D view rotation to prevent the user getting lost. I think you do that in a sample.

[abenedik]

I have improved the DXEngine and added support for rendering pixels.

The results are very good.

On my computer with NVidia 970 GTX with 4 GB DDR5 the DXEngine can render 25 million 2x2 pixels at 40 FPS. It can also render 100 MIO pixels, but the frame rate drops to a few frames per second. Because of out of memory errors I was not able to render 250 MIO pixels at once.

It is really amazing what can modern graphics cards achieve - in this case a geometry shader is used to create 2 triangles (with for positions) from each position that is sent to graphics card. This means that in case of 100 million positions, the graphics card is actually creating and rendering 400 million triangles - incredible.

And with a newer GPU it would be possible to achieve even better results.

Here is a screenshot from a new sample:
   

I have send BillC link to a pre-release version of DXEngine. If you would also like to test this, please send me a PM or an email and I will provide you with that link too.

[BillC]

As you said: The results are very good.

Using your sample, I was able to top out at 175M pts. I ran out of memory at 200M. At 175M it is running a little under 1 FPS. So a little slow. But 175M !!!! 10M is definitely usable and 25M is pretty good. 100M was running at 2 FPS. This is all in debug so there is that slowing it down.

I'm running a Dell XPS 9550 (i7) laptop with 32GB and a NVidia GeForce GTX 960M.

In short: I like it. This gives me a place to start, which I didn't have before.

[BillC]

Do we have an idea as to when a release with support for millions of point might happen?

I'm on the clock (Beta last week of Feb, Desperation: 1st week in March), but I know that doesn't mean you are. But having a rough answer lets me pass it along.

[abenedik]

I would also like to complete and publish the next Ab3d.DXEngine because I want to continue working on the next major Ab3d.PowerToys version.

The DXEngine already got a few smaller fixes and a few interesting new features - pixles rendering, per-vertex color effect, special hidden lines rendering, etc. The next version still require a few improvements to the new samples and additional testing with a few new tests (btw., there are many unit tests and also many render tests - special test scenes are rendered and the images are compared to expected images or rendering done with WPF 3D).

If everything will go well and if there will not be no bigger support cases or requests, then the next version will be hopefully released at the beginning of the next week - I will do my best to be done until the end of the next week.

EDIT:
Rendering many points is available in Ab3d.DXEngine v2.1 that was published on 23th February 2017