Proposal : Specification for Named Constants and Template Elements #2553
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| | @@ -157,6 +157,207 @@ Materials can inherit from other materials, to add or change shaders connected t | |
| Inheritance of material-type custom nodes is also allowed, so that new or changed input values can be applied on top of those specified in the inherited material. | ||
| | ||
| | ||
| ### Named Constants | ||
| | ||
| The MaterialX data library contains many nodes that have inputs with a number of common semantic default values, for instance zero or one. The MaterialX data library also uses a number of different concrete types, for example `float`, `color3` or `vector4`. These types each have different concrete values to represent these semantic defaults. <typedef> elements can be extended with child <constant> elements defining these named values. | ||
| | ||
| Attributes for <constant> elements: | ||
| | ||
| * `name` (string, required): a unique name for this <constant> element | ||
| * `value` (string, required): the concrete value for the given type that should be used during the substitution. | ||
| | ||
| ```xml | ||
| <typedef name="float"> | ||
| <constant name="zero" value="0.0"/> | ||
| <constant name="one" value="1.0"/> | ||
| </typedef> | ||
| <typedef name="color3"> | ||
| <constant name="zero" value="0.0,0.0,0.0"/> | ||
| <constant name="one" value="1.0,1.0,1.0"/> | ||
| </typedef> | ||
| <typedef name="vector4"> | ||
| <constant name="zero" value="0.0,0.0,0.0,0.0"/> | ||
| <constant name="one" value="1.0,1.0,1.0,1.0"/> | ||
| </typedef> | ||
| ``` | ||
| | ||
| These named constants can be used anywhere a concrete value is accepted, ie. in any `value` or `default` attribute. The named constant is referenced by prefixing the name of the named value with `"Constant:"`. | ||
| | ||
| ```xml | ||
| <input name="in1" type="color3" value="Constant:zero"/> | ||
| <input name="in2" type="float" value="Constant:one"/> | ||
| ``` | ||
| | ||
| This example, when combined with the example <typedef> elements above, is functionally identical to: | ||
| | ||
| ```xml | ||
| <input name="in1" type="color3" value="0.0,0.0,0.0"/> | ||
| <input name="in2" type="float" value="1.0"/> | ||
| ``` | ||
| | ||
| Not every <typedef> is required to define each named constant. Expansion of a named constant will raise an error if a specified name is not defined for the given <typedef>. | ||
| | ||
| ### Templated Definition Elements | ||
| | ||
| To reduce potential repetitive data library elements, any number of elements can be enclosed within a <template> element scope. The elements inside the <template> scope are concretely instantiated using the provided template attributes. | ||
| | ||
| Attributes for <template> elements: | ||
| | ||
| * `name` (string, required): a unique name for this <template> | ||
| * `varnames` (string, required): the names of variables to be used for substitution in the contained elements as a comma-separated list. | ||
| * `options` (string, required): the set of concrete values to be used during the substitution of the specified `varnames` in elements contained within the <template>, specified as a string of comma-separated lists of comma-separated strings surrounded by parentheses. | ||
| | ||
| When a <template> is expanded, all the child elements are instantiated exactly once for each entry in the `options` array. These instances replace the original <template> element in the document. They are placed at the same scope level as the original <template> element in the document. | ||
| | ||
| Each instance of the contents contained inside the <template> is processed as it is instantiated. This processing step inspects all attribute values in all the contained elements and replaces any occurrence of any specified string with the corresponding value from the `options` array. The specific string used for the string replacement matching is constructed by wrapping any of the values in the `varnames` list with `@` characters. For example, if `varnames` has the value `"foo"`, then the matching string would be `"@foo@"`. We surround the value of `varnames` with the `@` characters to ensure we do not have any accidental matches. | ||
| | ||
| ```xml | ||
| <template name="TP_ND_multiply" varames="typeName" options="(float, color3, vector4)"> | ||
| <nodedef name="ND_multiply_@typeName@" node="multiply" nodegroup="math"> | ||
| There was a problem hiding this comment. In the same spirit as the notes above, these new generic types seem analogous to the substitution mechanisms available in MaterialX, and the following syntax might be clearer and better harmonized with existing conventions: <nodedef name="ND_multiply_[typeName]" node="multiply" nodegroup="math">There was a problem hiding this comment. I'm very open to alternate punctuation other than "@" - that was a fairly arbitrary choice made, just to be able to move on with implementation. But I do feel strongly that if we want things to be robust we need to have different punctuation, otherwise what happens if the existing substitution is used in the same context as the template expansion. I'm not sure I'm super familiar with the substitution mechanism you're referring to here - can you perhaps provide a concrete example of a materialx document that is using it here, so we can compare the two syntaxes together. There was a problem hiding this comment. @ld-kerley Here is the current section on Token elements in MaterialX, which represent arbitrary key/value pairs for filename substitutions within NodeDef elements: And here's the broader section on Filename Substitutions, which might naturally extend to There was a problem hiding this comment. Looking at the example referenced from the documentation - its unclear to me how if token substitution and templates were used in the same construct how you would differentiate the two instances of Do you have any concrete ideas there? There was a problem hiding this comment. If I understand correctly, token substitution occurs only in string input values, such as file names. It is not currently performed in the Also, there is a question whether we want to substitute the current type ( In the file name substitution link above, in addition to Another, more verbose option would be to introduce new attribute names available in the templates: resulting in There was a problem hiding this comment. Hey @rafalSFX - thanks for the input - the name prefix/suffix idea is one worth considering. I'd need to look through all the templates I've made so far and see if the would be sufficient. If I'm understanding your suggestion here, you're only proposing using this for the I'll also note that currently we are applying the template variable substitution in a few different attributes.
There was a problem hiding this comment. Indeed, the prefix/suffix solution would target just the | ||
| <input name="in1" type="@typeName@" value="Value:zero" /> | ||
| <input name="in2" type="@typeName@" value="Value:one" /> | ||
| <output name="out" type="@typeName@" defaultinput="in1" /> | ||
| </nodedef> | ||
| </template> | ||
| ``` | ||
| | ||
| In this example, the provided `varnames` attribute is `"typeName"`, and so the matching string is `"@typeName@"`. Each instance of this string is then replaced by each element in the `options` string array, `"float"`, `"color3“`, and `"vector4"`. | ||
| | ||
| ```xml | ||
| <nodedef name="ND_multiply_float" node="multiply" nodegroup="math"> | ||
| <input name="in1" type="float" value="Value:zero" /> | ||
| <input name="in2" type="float" value="Value:one" /> | ||
| <output name="out" type="float" defaultinput="in1" /> | ||
| </nodedef> | ||
| <nodedef name="ND_multiply_color3" node="multiply" nodegroup="math"> | ||
| <input name="in1" type="color3" value="Value:zero" /> | ||
| <input name="in2" type="color3" value="Value:one" /> | ||
| <output name="out" type="color3" defaultinput="in1" /> | ||
| </nodedef> | ||
| <nodedef name="ND_multiply_vector4" node="multiply" nodegroup="math"> | ||
| <input name="in1" type="vector4" value="Value:zero" /> | ||
| <input name="in2" type="vector4" value="Value:one" /> | ||
| <output name="out" type="vector4" defaultinput="in1" /> | ||
| </nodedef> | ||
| ``` | ||
| | ||
| When used in combination with the [Named Constants](#named-constants) expansion, this example would become the following concrete set of fully expanded elements. | ||
| | ||
| ```xml | ||
| <nodedef name="ND_multiply_float" node="multiply" nodegroup="math"> | ||
| <input name="in1" type="float" value="0.0" /> | ||
| <input name="in2" type="float" value="1.0" /> | ||
| <output name="out" type="float" defaultinput="in1" /> | ||
| </nodedef> | ||
| <nodedef name="ND_multiply_color3" node="multiply" nodegroup="math"> | ||
| <input name="in1" type="color3" value="0.0,0.0,0.0" /> | ||
| <input name="in2" type="color3" value="1.0,1.0,1.0" /> | ||
| <output name="out" type="color3" defaultinput="in1" /> | ||
| </nodedef> | ||
| <nodedef name="ND_multiply_vector4" node="multiply" nodegroup="math"> | ||
| <input name="in1" type="vector4" value="0.0,0.0,0.0,0.0" /> | ||
| <input name="in2" type="vector4" value="1.0,1.0,1.0,1.0" /> | ||
| <output name="out" type="vector4" defaultinput="in1" /> | ||
| </nodedef> | ||
| ``` | ||
| | ||
| When multiple variable names are specified, each of those variables will be substituted simultaneously. This means that the list of options corresponding to each variable is required to be the same length. | ||
| | ||
| ```xml | ||
| <template name="TP_multivariable_example" varnames="numberName, numberValue" options="(one, two, three), (1.0, 2.0, 3.0)"> | ||
| <nodedef name="ND_constantFloat_@numberName@" node="example" nodegroup="math"> | ||
| <input name="in1" type="float" value="@numberValue@" /> | ||
| <output name="out" type="float" defaultinput="in1" /> | ||
| </nodedef> | ||
| </template> | ||
| ``` | ||
| | ||
| If there is a mismatch in the lengths of the options lists, then an error will be raised during expansion. The example below demonstrates different lengths of options lists; this is invalid syntax. | ||
| | ||
| ```xml | ||
| <!-- INVALID EXAMPLE - DO NOT COPY --> | ||
| <template name="TP_invalid_example" varnames="numberName, numberValue" options="(one, two, three, four), (1.0, 2.0)"> | ||
| <nodedef name="ND_constantFloat_@numberName@" node="example" nodegroup="math"> | ||
| <input name="in1" type="float" value="@numberValue@" /> | ||
| <output name="out" type="float" defaultinput="in1" /> | ||
| </nodedef> | ||
| </template> | ||
| ``` | ||
| | ||
| The `TP_multivariable_example` would be expanded to three concrete instances of the elements it contains, and both `@numberName@` and `@numberValue@` strings would be replaced. | ||
| | ||
| ```xml | ||
| <nodedef name="ND_constantFloat_one" node="example" nodegroup="math"> | ||
| <input name="in1" type="float" value="1.0" /> | ||
| <output name="out" type="float" defaultinput="in1" /> | ||
| </nodedef> | ||
| <nodedef name="ND_constantFloat_two" node="example" nodegroup="math"> | ||
| <input name="in1" type="float" value="2.0" /> | ||
| <output name="out" type="float" defaultinput="in1" /> | ||
| </nodedef> | ||
| <nodedef name="ND_constantFloat_three" node="example" nodegroup="math"> | ||
| <input name="in1" type="float" value="3.0" /> | ||
| <output name="out" type="float" defaultinput="in1" /> | ||
| </nodedef> | ||
| ``` | ||
| | ||
| <template> elements can be nested, and then are expanded sequentially, from the outer scope to the inner scope. This allows <template> element variable substitution to be used to define other template attributes. | ||
| | ||
| ```xml | ||
| <template name="TP_ND_add_matrix" varnames="typeName" options="(matrix33, matrix44)"> | ||
| <template name="TP_ND_add_@typeName@FA" varnames="nodeDefExt,floatTypeName" options="(@typeName@,@typeName@FA), (@typeName@,float)"> | ||
| <nodedef name="ND_add_@nodeDefExt@" node="add" nodegroup="math"> | ||
| <input name="in1" type="@typeName@" value="Constant:one" /> | ||
| <input name="in2" type="@floatTypeName@" value="Constant:zero" /> | ||
| <output name="out" type="@typeName@" defaultinput="in1" /> | ||
| </nodedef> | ||
| </template> | ||
| </template> | ||
| ``` | ||
| | ||
| Here we have two nested <template> elements. `TP_ND_add_matrix` will be expanded first. This is important as here the inner <template> element uses the `@typeName@` template variable reference in its definition. This first <template> would be expanded resulting in two instances of the contained elements, since the options list `(matrix33, matrix44)` has length two. | ||
| | ||
| ```xml | ||
| <template name="TP_ND_add_matrix33FA" varnames="nodeDefExt,floatTypeName" options="(matrix33,matrix33FA), (matrix33,float)"> | ||
| <nodedef name="ND_add_@nodeDefExt@" node="add" nodegroup="math"> | ||
| <input name="in1" type="matrix33" value="Constant:one" /> | ||
| <input name="in2" type="@floatTypeName@" value="Constant:zero" /> | ||
| <output name="out" type="matrix33" defaultinput="in1" /> | ||
| </nodedef> | ||
| </template> | ||
| <template name="TP_ND_add_matrix44FA" varnames="nodeDefExt,floatTypeName" options="(matrix44,matrix44FA), (matrix44,float)"> | ||
| <nodedef name="ND_add_@nodeDefExt@" node="add" nodegroup="math"> | ||
| <input name="in1" type="matrix44" value="Constant:one" /> | ||
| <input name="in2" type="@floatTypeName@" value="Constant:zero" /> | ||
| <output name="out" type="matrix44" defaultinput="in1" /> | ||
| </nodedef> | ||
| </template> | ||
| ``` | ||
| | ||
| This results in two "inner" scoped <template> elements, `TP_ND_add_matrix33FA` and `TP_ND_add_matrix44FA`. These then both each get expanded and both result in a further two instances of each set of contained elements, because all the options lists for each of the variables are also of length two, `(matrix33,matrix33FA)` and `(matrix33,float)`. | ||
| | ||
| ```xml | ||
| <nodedef name="ND_add_matrix33" node="add" nodegroup="math"> | ||
| <input name="in1" type="matrix33" value="Constant:one" /> | ||
| <input name="in2" type="matrix33" value="Constant:zero" /> | ||
| <output name="out" type="matrix33" defaultinput="in1" /> | ||
| </nodedef> | ||
| <nodedef name="ND_add_matrix33FA" node="add" nodegroup="math"> | ||
| <input name="in1" type="matrix33" value="Constant:one" /> | ||
| <input name="in2" type="float" value="Constant:zero" /> | ||
| <output name="out" type="matrix33" defaultinput="in1" /> | ||
| </nodedef> | ||
| <nodedef name="ND_add_matrix44FA" node="add" nodegroup="math"> | ||
| <input name="in1" type="matrix44" value="Constant:one" /> | ||
| <input name="in2" type="matrix44" value="Constant:zero" /> | ||
| <output name="out" type="matrix44" defaultinput="in1" /> | ||
| </nodedef> | ||
| <nodedef name="ND_add_matrix44" node="add" nodegroup="math"> | ||
| <input name="in1" type="matrix44" value="Constant:one" /> | ||
| <input name="in2" type="float" value="Constant:zero" /> | ||
| <output name="out" type="matrix44" defaultinput="in1" /> | ||
| </nodedef> | ||
| ``` | ||
| <p> <p><hr><p> | ||
| | ||
| # Proposals: Stdlib Nodes<a id="propose-stdlib-nodes"></a> | ||
| | ||
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Although I see the logic behind the proposed
Constant:prefix, we've historically used the:separator to represent the concept of namespaces in MaterialX, with brackets used to represent the concept of substitutions.To my mind, this new mechanism seems closer in spirit to a substitution, and I'd be curious as to whether the following syntax might be clearer and more idiomatic to MaterialX:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For robustness I think its important to have an identifiable string prefix that we do not think will be used for any other part of the syntax.
In some sort of conceptual way, I think the current proposed syntax is a "namespace" of sorts - in as much as we're saying that the named constants "zero" "one" all belong to the "Constants:" namespace.
If we don't want to use the ":" token - I'm open to using something else - but I feel that having the "Constant" prefix in there makes the document clearer to read - as well as aids the robustness of implementation.
I do not think we would need the closing punctuation, ie. "]" in your example, as this construct completely replaces the
valuefield. Including a closing punctuation implies you could write something likeWhich is not the intention of this named constants grammar.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Personally I prefer
value="Constant:zero"because of explicit clarity. But I do appreciate that to people fluent in .mtlx syntax, seeingvalue="[zero]"feels natural and obvious, and I think it's important to keep things consistent, if they are relating to similar concepts.There is however a certain ambiguity in
value="[zero]"because there are several "zeros" to substitute with, depending on the type context. It's probably not a big deal, though.Perhaps,
value="color3:zero"makes it crystal clear what value is being used. But then the repetition ofcolor3is not ideal (for copy-pasting, etc):so generalizing it to
or
value="typeconst:zero", but that's not much different fromvalue="constant:zero". And I'm back to square one.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
haha - @rafalSFX - I'm pretty sure I went around a similar circle more than once while putting this together, and always ended up coming back to something of the form
<prefix>:<constantName>. I think the<prefix>part is super important for a few different reasons. It was the cleanest way I could find to make the document readable and easy to understand, and also I think that structure gives us the easiest and most robust thing to parse and evaluate, I tried others and the code was more complex or fragile.I'm very happy to bike shed what the
<prefix>string should be. I likeConstantwhich was actually suggested by @bernardkwok , but I do also like yourtypeconstand I'm sure there are other very valid suggestions. I don't feel too strongly about which specific horse we pick in this race - but I do think this is the right race to pick a horse from.I think it would be less helpful to have
color3:zero, because in the template case that would end up being templated to something like@varName@:zerobecause we would need to support different type names in the template - this feels less clear to read to me.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we needed to implement this as a namespace prefix rather than a substitution, I'd be open to @rafalSFX's suggestion of
type:zero, which clarifies that we're declaring a new string in thetypenamespace.Taking a step back, though, I have a suggestion for an even more harmonized approach, which might give us the best of both worlds:
What if we were to declare these type substitutions using actual
tokenelements, rather than adding a newconstantelement category to MaterialX?The proposed syntax would look as follows:
With this approach, we would be directly leveraging the existing mechanism for token substitutions in MaterialX, and merely extending the declaration of token elements from
nodedefparents totypedefparents.The rules for handling overlapping tokens would remain exactly as before, and any future run-time implementation could be implemented by simply extending our existing codebase for string substitutions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think my biggest worry with re-using the existing token system would be the clashing of token names as resolved up through the hierarchy. The intention and goal with the named constants is to provide a robust indirection to the concrete values. Working through the engineering for this, we've already discovered cases in the data library where mistakes were made. Overloading the token substitution system like this I feel we would be reducing the potential robustness of the system.
The current proposal is explicitly constrained to the
valueattribute of input/output elements, and by definition uses the correspondingtypeattribute to perform the resolution.The current token substitution system is not currently tied to the type of the input, other than (I think) because it only operates on
filenametypes values? So we may risk introducing complexity to the token substitution system if we overload it with multiple uses.In general I think it's a good idea to not re-use syntax for different purposes, and instead am in favor of each different piece of grammar having a single specific purpose.
This is an example that I feel could be pretty confusing to the user.
The final complication I see with re-using the token substitution is that it becomes pretty unclear how to provide a build-time expansion of the named values. One of the concrete goals of this work is to be able to deliver a data library that is unmodified to downstream consumers. If we're using token substitution, how would we indicate which of these tokens should be substituted at build time? I don't believe we would want to expand everything, because that would defeat the purpose of the current token substitution system.