Debugging Hiera

When debugging Hiera, puppet lookup can help identify exactly what Hiera was doing when it raised an error, or how it decided to look up a key and where it got its value.

Use these examples to guide you in debugging Hiera with the puppet lookup command.

Syntax errors

Consider the following error message.

SERVER: Evaluation Error: Error while evaluating a Resource Statement, Lookup of key 'pe_console_prune::ensure_prune_cron' failed: DataBinding 'hiera': (<unknown>): mapping values are not allowed here at line 2 column 68 on node master.inf.puppetlabs.demo
  1. To debug this, run the following command on the primary node: 
    puppet lookup pe_console_prune::ensure_prune_cron

    Note that all the rest of the errors and logging around the error are cleared.

  2. To get more information, add the --debug flag: 
    puppet lookup --debug pe_console_prune::ensure_prune_cron
  3. Look at the last few lines of output from the puppet lookup command: 
    ... Debug: hiera(): [eyaml_backend]: Hiera eYAML backend starting ... Debug: hiera(): Hiera YAML backend starting Debug: hiera(): Looking up lookup_options in YAML backend Debug: hiera(): Looking for data source nodes/master.inf.puppetlabs.demo Debug: hiera(): Cannot find datafile /etc/puppetlabs/code/environments/production/hieradata/nodes/master.inf.puppetlabs.demo.yaml, skipping Debug: hiera(): Looking for data source environment/production Debug: hiera(): Looking for data source datacenter/infrastructure Debug: hiera(): Looking for data source virtual/virtualbox Debug: hiera(): Looking for data source common Error: Could not run: Evaluation Error: Error while evaluating a Resource Statement, Lookup of key 'pe_console_prune::ensure_prune_cron' failed: DataBinding 'hiera': (<unknown>): mapping values are not allowed in this context at line 2 column 68
Results

In this example, the last thing Hiera was doing when it threw the error was looking for the common datasource in the YAML backend. That is, it was reading common.yaml. Therefore, the syntax error must be in common.yaml.

Unexpected values

Sometimes Hiera does not throw an error, but still fails to return the value you expect.

For example, if you think your node is configured to use au.pool.ntp.org but it is actually configured with us.pool.ntp.org, there is no error message but something is wrong.

The lookup command accepts a --node flag to set the node context for performing the lookup. If --node isn’t passed, the default is the context of the node on which the command runs.

  1. To determine why one of your secondary nodes is getting a particular value for the ntp::servers parameter, use lookup and the --node flag to inspect the process: 
    [root@master ~]# puppet lookup --node centos7a.pdx.puppetlabs.local --debug ntp::servers ... Debug: hiera(): Looking up ntp::servers in YAML backend Debug: hiera(): Looking for data source nodes/centos7a.pdx.puppetlabs.demo Debug: hiera(): Cannot find datafile /etc/puppetlabs/code/environments/production/hieradata/nodes/centos7a.pdx.puppetlabs.demo.yaml, skipping Debug: hiera(): Looking for data source environment/production Debug: hiera(): Looking for data source datacenter/portland Debug: hiera(): Found ntp::servers in datacenter/portland --- - 0.us.pool.ntp.org - 1.us.pool.ntp.org - 2.us.pool.ntp.org - 3.us.pool.ntp.org
    Here you can see how Hiera walks through the fully evaluated version of the hierarchy for the target node. The original hierarchy is full of variables for interpolation: 
    :hierarchy: - nodes/%{clientcert} - environment/%{environment} - datacenter/%{datacenter} - virtual/%{virtual} - common
  2. In the lookup debug output, you can see that for this node, environment/%{environment} evaluates to environment/production, but Hiera does not find any data there for ntp:servers. And so on, down the hierarchy. This visibility gives you what you need to think through the lookup process step by step, and confirm with each iteration whether or not everything is working correctly.

Common errors

In addition to knowing how to use the puppet lookup command, knowing some common errors can be useful in debugging Hiera.

  1. Error: 
    Error: Could not run: (<unknown>): mapping values are not allowed in this context at line 2 column 8 Error: Could not run: (<unknown>): did not find expected '-' indicator while parsing a block collection at line 1 column 1

    Cause:

    The opening --- could be malformed. If it gets converted into a unicode character, such as —, or if there is a space at the start of the line ---, or in between the three dashes - --, you might get an error like this.

  2. Error: 
    Error: Could not run: (<unknown>): mapping values are not allowed in this context at line 3 column 10

    Cause:

    This can be caused by using tabs for indentation instead of spaces. In general, avoid tab characters in yaml files.

  3. Error: 
    Error: Could not run: Hiera type mismatch: expected Hash and got String Error: Could not run: Hiera type mismatch: expected Hash and got Array Error: Could not run: Hiera type mismatch: expected Array and got Hash

    These types of errors often happen when you use hiera_array() or hiera_hash(), but one or more of the found values are of a data type incompatible with that lookup.