fail to detect alpha channel heightmap in png normal map (seems to be the same with webp)

[illwieckz]

Some time ago I added some code to be able to autodetect an heightmap in a normalmap, this is very useful when loading third-party asset from games that relies on such mechanism (Xonotic maps are good example of maps that push our engine forward):

Daemon/src/engine/renderer/tr_image.cpp

Lines 1364 to 1383 in e63aa21

	// detect heightmap in normalmap alpha channel and enable it
	// if not already enabled by explicit shader keyword
	if ( image->bits & IF_NORMALMAP )
	{
	if ( ! ( image->bits & IF_DISPLACEMAP ) )
	{
	switch ( image->internalFormat )
	{
	case GL_RGBA:
	case GL_RGBA8:
	case GL_RGBA16:
	case GL_RGBA16F:
	case GL_RGBA32F:
	case GL_RGBA32UI:
	case GL_COMPRESSED_RGBA_S3TC_DXT3_EXT:
	case GL_COMPRESSED_RGBA_S3TC_DXT5_EXT:
	image->bits |= IF_DISPLACEMAP;
	}
	}
	}

I discovered that code is not able to autodetect the heightmap in png, and if I'm right, webp too. It would be useful to test against tga too.

I verified it works against crn and dds by the way (haven't experimented with ktx at all) which are some kind of “native” GPU formats.

I plan to implement an explicit normalHeightMap keyword (as others idtech3 derivated engines did) but it would be good to know why the above code fails on non-GPU textures formats.

[illwieckz]

It looks like I fixed it in one of my work-in-progress branch.

The problem is that XreaL was bad designed and expected loose heightmap to be heightmap and embedded heightmap in normalmap to be depthmap, so if normalmap had no alpha channel, loading the alpha channel would result in a weird displacement, so the code disabled normalmap alpha channel if the shader did not asked for it explicitly, making alphachannel hence heightmap detection not possible at all.

After fixing the design by using an heightmap instead of a depthmap like all the industry around, we can keep normalmap alphachannel even if does not exist: it will result in a flat heightmap with no side effect.

It even does not break the capcom z reconstruction trick (see #183 (comment)) which expects alphachannel to be fully opaque to do it's computation when dxt format does not abuse the alpha channel.

And we can still then use the detection for fully opaque alpha channel to even make the engine aware of the heightmap being embedded, even if if not knowing it would not break the render when dxt texture does not abuse alpha channel for normal z reconstruction and capcom z reconstruction trick.

Knowing the heightmap is not embedded can be used to use the capcom z reconstruction trick for normalmap abusing alpha channel for z reconstruction.