Open Sourcing DXIL Validator Hash
Today the HLSL team is announcing plans to open source the DXIL validator hashing algorithm and include it in the DirectXShaderCompiler sources on GitHub. If you’re wondering what any of that means, strap in!
DXIL Validator
To understand the DXIL Validator you first need to understand a few basics of what DXIL is. DXIL, the DirectX Intermediate Language, is the format used to provide shader binaries to DirectX drivers. DXIL is based on the LLVM-3.7 IR module format, but is constrained to not support all valid features of LLVM’s IR.
The constraints placed on a DXIL module are enforced by the DXIL Validator, which acts as a binary analysis tool running on the generated DXIL binary and verifying that it conforms to DXIL’s specified requirements.
The validator comes in two forms, the internal validator which is fully open source and built into DXC’s dxcompiler dynamic library, and the external validator which is built from a mixture of open source and closed source code and distributed in the DXIL dynamic library with DXC’s Linux and Windows releases.
The only difference between the internal and external validators is that the external validator contains the algorithm to compute the Validator Hash, and on successful validation it writes the hash into the compiled shader binary.
DXIL Validator Hash
The DirectX runtime validates the hash on each shader by computing the hash from DXIL and comparing the computed value against the value written in the shader binary. This validation is performed on all shaders unless you are using the AgilitySDK with the D3D12ExperimentalShaderModels experimental feature enabled (see: D3D12EnableExperimentalFeatures on learn.microsoft).
The DXIL Validator Hash is not cryptographically strong and was never intended as a security feature. The sole purpose of the validator hashing flow is to provide users and drivers with confidence about the correctness of the generated shader binary and detect invalid program constructs that can’t currently be caught earlier in the compiler.
Why Open Source?
Since its inception there have been several use cases where bypassing the hash verification has been essential for performance or to support critical workflows. Some of those use cases include platform mapping layers, shader debugging and profiling tools, and experimental shader features. This has forced individuals outside Microsoft to reverse engineer the algorithm.
Our users have also found that it is beneficial to be able to build shaders for DirectX applications on non-Windows build hosts. While we do ship the DXIL shared library in our Ubuntu Linux release packages, we can’t scale our release binaries to cover all of the varieties of Linux and Unix that our users find valuable.
We believe we can better serve our users by open sourcing the validator hash algorithm and providing a supported way to bypass hash validation.
Bypassing Hash Verification
In a future Agility SDK release we’ll be adding support for a new way to bypass hash verification that does not require enabling experimental features. Instead, hash verification will be skipped if a special BYPASS value is specified in the shader’s hash field.
This new method for bypassing shader verification along with options to run the DXIL validator as part of the D3D Debug Layer is intended to allow software vendors outside Microsoft to provide the best experiences possible for their users without needing to reverse engineer code.
You can read a lot more about the technical details of the plan on the HLSL-Specs proposal.
Light
Dark
1 comment
Thank you! This will be great for open source game engines like Godot.