Proposal : Specification for Named Constants and Template Elements

[ld-kerley]

Overview

This proposal aims to define the specification language that would be added to the MaterialX specification to describe the new template syntax that was introduced in this PR (#2451).

Details

The templating PR describes and implements a system intended to simplify the authoring of the MaterialX standard data library, making it less repetitive, and consequently less error-prone. The system introduces new syntax to the XML-based MaterialX grammar to provide the ability to describe families of nodes in a single "templated" definition, reducing the need to explicitly state each different node signature that exists for a given node.

The PR above explicitly intends for this new grammar to be introduced in the standard data library source files, and concretely NOT in any MaterialX data library files that are built or installed by the CMake-based build process. This approach has the advantage of not requiring that any downstream integrations of MaterialX have knowledge of this update to the grammar, as all templates can be expanded at build time. This is a deliberate design choice of the PR, as discussed in the previous proposal and associated TSC meeting discussions, as it will allow the project the freedom to iterate on the grammar without affecting downstream MaterialX integrations. As such, the original PR did NOT intend the formal MaterialX specification to be updated initially, as the generated data library files should be unchanged by this process. We are initially just introducing an optimization of the sources used to generate the data library during the build.

Once the PR has been merged, the new build-time template grammar has been given time to mature, and feedback gathered from MaterialX stakeholders, we may decide to introduce a runtime-based system. If so, then at that point, the new template grammar would need to be part of the formal specification of MaterialX. This would infer that downstream integrators of MaterialX would be required to support ingestion of a data library containing this new formalized template syntax, if they wanted to provide 100% MaterialX compliance.

Here we attempt to describe formally that syntax as it currently stands. We should note that it is possible the MaterialX project may opt to change this before it becomes formalized, or perhaps even decide that the template syntax should remain solely the domain of the build-time files, and that downstream consumers prefer to ingest the fully expanded data library files.

[jstone-lucasfilm]

Although I see the logic behind the proposed Constant: prefix, we've historically used the : separator to represent the concept of namespaces in MaterialX, with brackets used to represent the concept of substitutions.

To my mind, this new mechanism seems closer in spirit to a substitution, and I'd be curious as to whether the following syntax might be clearer and more idiomatic to MaterialX:

 <input name="in1" type="color3" value="[zero]"/> <input name="in2" type="float" value="[one]"/>

[ld-kerley]

For robustness I think its important to have an identifiable string prefix that we do not think will be used for any other part of the syntax.

In some sort of conceptual way, I think the current proposed syntax is a "namespace" of sorts - in as much as we're saying that the named constants "zero" "one" all belong to the "Constants:" namespace.

If we don't want to use the ":" token - I'm open to using something else - but I feel that having the "Constant" prefix in there makes the document clearer to read - as well as aids the robustness of implementation.

I do not think we would need the closing punctuation, ie. "]" in your example, as this construct completely replaces the value field. Including a closing punctuation implies you could write something like

<input name="in" type="vector3" value="[zero], [one], [zero]"> 

Which is not the intention of this named constants grammar.

[rafalSFX]

Personally I prefer value="Constant:zero" because of explicit clarity. But I do appreciate that to people fluent in .mtlx syntax, seeing value="[zero]" feels natural and obvious, and I think it's important to keep things consistent, if they are relating to similar concepts.

There is however a certain ambiguity in value="[zero]" because there are several "zeros" to substitute with, depending on the type context. It's probably not a big deal, though.

Perhaps, value="color3:zero" makes it crystal clear what value is being used. But then the repetition of color3 is not ideal (for copy-pasting, etc):

<input name="in1" type="color3" value="color3:zero"/> 

so generalizing it to

<input name="in1" type="color3" value="type:zero"/> 

or value="typeconst:zero", but that's not much different from value="constant:zero". And I'm back to square one.

[jstone-lucasfilm]

In the same spirit as the notes above, these new generic types seem analogous to the substitution mechanisms available in MaterialX, and the following syntax might be clearer and better harmonized with existing conventions:

<nodedef name="ND_multiply_[typeName]" node="multiply" nodegroup="math">

[ld-kerley]

I'm very open to alternate punctuation other than "@" - that was a fairly arbitrary choice made, just to be able to move on with implementation. But I do feel strongly that if we want things to be robust we need to have different punctuation, otherwise what happens if the existing substitution is used in the same context as the template expansion.

I'm not sure I'm super familiar with the substitution mechanism you're referring to here - can you perhaps provide a concrete example of a materialx document that is using it here, so we can compare the two syntaxes together.

[jstone-lucasfilm]

@ld-kerley Here is the current section on Token elements in MaterialX, which represent arbitrary key/value pairs for filename substitutions within NodeDef elements:

https://github.com/AcademySoftwareFoundation/MaterialX/blob/main/documents/Specification/MaterialX.Specification.md#nodedef-token-elements

And here's the broader section on Filename Substitutions, which might naturally extend to value and type substitutions for the named constant and templated type functionality in this proposal:

https://github.com/AcademySoftwareFoundation/MaterialX/blob/main/documents/Specification/MaterialX.Specification.md#filename-substitutions

[ld-kerley]

Looking at the example referenced from the documentation - its unclear to me how if token substitution and templates were used in the same construct how you would differentiate the two instances of [ and ].

Do you have any concrete ideas there?

[rafalSFX]

If I understand correctly, token substitution occurs only in string input values, such as file names. It is not currently performed in the name XML attribute. So I don't think there is any ambiguity currently. But I share your concern, and even though currently there is no issue, we should be careful about the future.

Also, there is a question whether we want to substitute the current type (typeName) in the input sting values, which would indeed lead to a serious ambiguity.

In the file name substitution link above, in addition to [], there are <> and {} substitutions too. So it looks like @@ is an available choice.

Another, more verbose option would be to introduce new attribute names available in the templates:

<nodedef nameprefix="ND_multiply_" namesuffix="_foo" node="multiply" nodegroup="math"> 

resulting in name="ND_multipy_vector3_foo" in the stamped-out nodedef. But it looks complicated/convoluted to me.