[zacfromaustinpowder]

I'm aware of the existing VertexColorMaterial, but I need to be able to overlay vertex colors over a texture.

I'm writing a shader to do so, but I'm stuck figuring out how to populate the vertex colors buffer.

On my CustomVertexColorMaterial I've noticed the ProcessMeshDataChannels function which I can override, but that doesn't seem to get called. It also doesn't seem to be overriden on the existing VertexColorMaterial.

The next path I might pursue is setting renderingItem.AdditionalVertexBuffers in ApplyRenderingItemMaterial in my CustomVertexColorEffect. Though I'm not sure that's the correct way to go.

I'm wondering if you might point me in the right direction. Where do I set the vertex colors which the shader reads from?

[abenedik]

I recommend that you store the VertexColorArray to the mesh, by using:

Code:

mesh.SetDataChannel(MeshDataChannelTypes.VertexColors, vertexColors);

Then in your effect class use the following code (all taken from VertexColorEffect):

in OnInitializeDeviceResources override (prepare effect's resources):

Code:

_positionNormalTexture0Color1_vertexBufferDescription = gpuDevice.VertexBufferDescriptionsManager.GetPositionNormalTexture0Color1VertexBufferDescription();

_standardTechnique = new StandardEffectTechnique(Scene, "VertexColorEffect-StandardTechnique")  {
    VertexInputStatePtr = _positionNormalTexture0Color1_vertexBufferDescription.PipelineVertexInputStateCreateInfoPtr,
    ShaderStages = _pipelineShaderStages,
    PipelineLayout = _pipelineLayout,
    ColorBlendAttachmentState = CommonStatesManager.OpaqueAttachmentStat
};

in ApplyRenderingItemMaterial override:

Code:

if (renderingItem.ParentSceneNode is ModelNode modelNode)
{
    var sceneNodeMesh = modelNode.GetMesh();

    // Check if mesh provides PixelColors or PixelSizes data channels
    if (sceneNodeMesh != null)
    {
        vertexColorsBuffer = GetVertexColorBufferFromMesh(sceneNodeMesh, renderingContext.GpuDevice, out var hasTransparentColors);
       
        if (hasTransparentColors.HasValue)
            hasTransparency = hasTransparentColors.Value;
    }
}

if (vertexColorsBuffer != null)
{
    if (vertexColorsBuffer.ItemsCount < renderingItem.VertexCount)
        Log.Warn?.Write(LogArea, "VertexColorEffect warning: vertexColorsBuffer has less colors defined ({0}) then the number of vertices ({1}) in the RenderingItem (ParentSceneNode.Id: {2})", vertexColorsBuffer.ItemsCount, renderingItem.VertexCount, renderingItem.ParentSceneNode?.Id.ToString() ?? "<null>");
   
    renderingItem.AdditionalVertexBuffers = new Buffer[] { vertexColorsBuffer.Buffer };


    // Add Color vertex buffer to binding 1
    renderingItem.VertexBufferDescription = _positionNormalTexture0Color1_vertexBufferDescription;
}

The following method was used in the previous code block:

Code:

internal static GpuBuffer? GetVertexColorBufferFromMesh(Mesh mesh, VulkanDevice gpuDevice, out bool? hasTransparentColors)
{
    hasTransparentColors = null;

    GpuBuffer? vertexColorsBuffer = mesh.GetDataChannelGpuBuffer(MeshDataChannelTypes.VertexColors);

    if (vertexColorsBuffer == null)
    {
        mesh.GetDataChannelArray<Color4>(MeshDataChannelTypes.VertexColors)

        var pixelColorsObject = mesh.GetDataChannel(MeshDataChannelTypes.VertexColors);
        if (pixelColorsObject != null)
        {
            Color4[]? positionColorsArray;

            if (pixelColorsObject is Color4[] positionColors4)
            {
                positionColorsArray = positionColors4;
            }
            else if (pixelColorsObject is Color3[] positionColors3)
            {
                // Convert Color3 to Color4
                positionColorsArray = new Color4[positionColors3.Length];
                for (int i = 0; i < positionColors3.Length; i++)
                    positionColorsArray[i] = positionColors3[i].ToColor4();
            }
            else
            {
                Log.Warn?.Write(LogArea, $"Unknown VertexColors or PixelColors data format: {pixelColorsObject.GetType().Name}. VertexColorEffect and PixelEffect support only Color4[] and Color3[] formats.");
                positionColorsArray = null;
            }

            if (positionColorsArray != null)
            {
                Log.Trace?.Write(LogArea, "VertexColorBuffer from VertexColors mesh data");

                vertexColorsBuffer = gpuDevice.CreateBuffer(positionColorsArray, BufferUsageFlags.VertexBuffer, isDeviceLocal: true);

                // Set GpuBuffer for next VertexColors usage
                mesh.SetDataChannelGpuBuffer(MeshDataChannelTypes.PixelColors, vertexColorsBuffer);
            }
        }
    }

    if (vertexColorsBuffer != null)
    {
        // Try to get the HasTransparentColors metadata if it was set
        hasTransparentColors = mesh.GetMetadata(VertexColorEffect.HasTransparentColorsMetadataKey) as bool?;
    }

    return vertexColorsBuffer;
}

In vertex shader you define the following layouts:

Code:

layout (location = 0) in vec3 inPosition;
layout (location = 1) in vec3 inNormal;
layout (location = 2) in vec2 inUV;
layout (location = 3) in vec4 inColor;

layout (location = 0) out vec3 outNormal;
layout (location = 1) out vec2 outUV;
layout (location = 2) out vec4 outColor;

// Set the outColor = inColor;

Fragment shader is then simply:

Code:

layout (location = 0) in vec3 inNormal;
layout (location = 1) in vec2 inUV;
layout (location = 2) in vec4 inColor;

layout(location = 0) out vec4 outColor;

void main()
{
    outColor = inColor;
}

I hope you got enough information to create your own version of the VertexColorEffect.