transform Code in Rewrite

Sometimes, we may want to apply some transformations to the meta variables in the fix part of a YAML rule. For example, we may want to change the case, add or remove prefixes or suffixes. ast-grep provides a transform key that allows us to specify such transformations.

Use transform in Rewrite

transform accepts a dictionary of which:

• the key is the new variable name to be introduced and
• the value is a transformation object that specifies which meta-variable is transformed and how.

A transformation object has a key indicating which string operation will be performed on the meta variable, and the value of that key is another object (usually with the source key). Different string operation keys expect different object values.

The following is an example illustrating the syntax of a transformation object:

transform:  NEW_VAR:  replace:  source: $VAR_NAME  replace: regex  by: replacement  ANOTHER_NEW_VAR:  substring:  source: $NEW_VAR  startChar: 1  endChar: -1

ast-grep 0.38.3+ supports string style transformations to simplify rule writing. The above example can be simplified to one-line style like:

transfrom:  NEW_VAR: replace($VAR_NAME, replace=regex, by=replacement)  ANOTHER_NEW_VAR: substring($NEW_VAR, startChar=1, endChar=-1)

Example of Converting Generator in Python

Converting generator expression to list comprehension in Python is a good example to illustrate transform.

More concretely, we want to achieve diffs like below:

"".join(i for i in iterable)  "".join([i for i in iterable]) 

This rule will convert the generator inside join to a list.

id: convert_generator rule:  kind: generator_expression  pattern: $GEN transform: # 1. the transform option  LIST: # 2. New variable name  substring: # 3. the transform operation name  source: $GEN # 4.1 transformation source  startChar: 1 # 4.2 transformation argument  endChar: -1 fix: '([$LIST])' # 5. use the new variable in fix

Let's discuss the API step by step:

1. The transform key is used to define one or more transformations that we want to apply to the meta variables in the pattern part of the rule.
2. The LIST key is the new variable name that we can use in fix or later transformation. We can choose any name as long as it does not conflict with any existing meta variable names. Note, the new variable name does not start with $.
3. The substring key is the transform operation name that we want to use. This operation will extract a substring from the source string based on the given start and end characters.
4. substring accepts an object
   1. The source key specifies which meta variable we want to transform. It should have $ prefix. In this case, it is $GEN that which matches the generator expression in the code.
   2. The startChar and endChar keys specify the indices of the start and end characters of the substring that we want to extract. In this case, we want to extract everything except the wrapping parentheses, which are the first and last characters: ( and ).
5. The fix key specifies the new code that we want to replace the matched pattern with. We use the new variable $LIST in the fix part, and wrap it with [ and ] to make it a list comprehension.

Pro Tips

Later transformations can use the variables that were transformed before. This allows you to stack string operations and achieve complex transformations.

Supported transformation

We have several different transformations available now. Please check out transformation reference for more details.

• replace: Use a regular expression to replace the text in a meta-variable with a new text.
• substring: Create a new string by cutting off leading and trailing characters.
• convert: Change the string case of a meta-variable, such as from camelCase to underscore_case.
• rewrite: Apply rewriter rules to a meta-variable AST and generate a new string. It is like rewriting a sub node recursively.

Rewrite with Regex Capture Groups

The replace transformation allows us to use Rust regex capture groups like (?<NAME>.*) to capture meta-variables and reference them in the by field. For example, to replace debug with release in a function name, we can use the following transformation:

id: debug-to-release language: js rule: {pattern: $OLD_FN($$$ARGS)} # Capture OLD_FN constraints: {OLD_FN: {regex: ^debug}} # Only match if it starts with 'debug' transform:  NEW_FN:  replace:  source: $OLD_FN  replace: debug(?<REG>.*) # Capture everything following 'debug' as REG  by: release$REG # Refer to REG just like a meta-variable fix: $NEW_FN($$$ARGS)

which will result in the following change:

debugFoo(arg1, arg2)  releaseFoo(arg1, arg2) 

Alternatively, replacing fooDebug with fooRelease, is difficult because you can't concatenate a meta-variable with a capitalized string literal. release$REG is fine, but $REGRelease will be interpreted as a single meta-variable and not a concatenation. One workaround is to use multiple sequential transformations, as shown below.

Limitation

You can only extract regex capture groups in the replace field of the replace transformation and you can only reference them in the by field of the same transformation. The regular regex rule does not support capture groups.

Multiple Sequential Transformations

Each transformation outputs a meta-variable that can be used as the input to later transformations. Chaining transformations like this allows us to build up complex behaviors.

Here we can see an example that transforms fooDebug into fooRelease by using convert, replace, and convert transformations.

rule: {pattern: $OLD_FN($$$ARGS)} # Capture OLD_FN constraints: {OLD_FN: {regex: Debug$}} # Only match if it ends with 'Debug' transform:  KEBABED: # 1. Convert to 'foo-debug'  convert:  source: $OLD_FN  toCase: kebabCase  RELEASED: # 2. Replace with 'foo-release'  replace:  source: $KEBABED  replace: (?<ROOT>)-debug  by: $ROOT-release  UNKEBABED: # 3. Convert to 'fooRelease'  convert:  source: $RELEASED  toCase: camelCase fix: $UNKEBABED($$$ARGS)

Add conditional text

Occasionally we may want to add extra text, such as punctuations and newlines, to our fixer string. But whether we should add the new text depends on the presence of absence of other syntax nodes.

A typical scenario is adding a comma between two arguments or list items. We only want to add a comma when the item we are adding is not the last one in the argument list.

We can use replace transformation to create a new meta-variable that only contains text when another meta-variable matches something.

For example, suppose we want to add a new argument to existing function call. We need to add a comma , after the new argument only when the existing call already has some arguments.

id: add-leading-argument language: python rule:  pattern: $FUNC($$$ARGS) transform:  MAYBE_COMMA:  replace:  source: $$$ARGS  replace: '^.+'  by: ', ' fix:  $FUNC(new_argument$MAYBE_COMMA$$$ARGS)

In the above example, if $$$ARGS matches nothing, it will be an empty string and the replace transformation will take no effect. The final fix string will be instantiated to $FUNC(new_argument).

If $$$ARGS does match nodes, then the replacement regular expression will replace the text with ,, so the final fix string will be $FUNC(new_argument, $$$ARGS)

DasSurma Trick

This method is invented by Surma in a tweet, so the useful trick is named after him.

String Style Transformations

To simplify the syntax of transformations, ast-grep 0.38.3+ supports a new string style transformation syntax. This allows us to write transformations in a more concise and readable way.

The string style transformation syntax is similar to the CSS function call syntax

# illustration of string style transformation syntax NEW_VAR: transform($SOURCE_VAR, option1=value1, option2=value2)

The transformation name is followed by parentheses containing the arguments. The first argument is always the source meta-variable, and the rest are the transformation options in the form of key-value pairs.

For example, the transformation examples above can be written as:

transform:  LIST: substring($GEN, startChar=1, endChar=-1)  KEBABED: convert($OLD_FN, toCase=kebabCase)  MAYBE_COMMA: replace($$$ARGS, replace='^.+', by=', ')

WARNING

The string style transformation syntax is only available in ast-grep 0.38.3 and later versions. If you are using an older version, please use the original object style syntax.

Even More Advanced Transformations

We can use rewriters in the rewrite transformation to apply dynamic transformations to the AST. We will cover it in next section.