[#7] Add missing robots.txt file to the asset file types

[brambaud]

There is no symlink to the robots.txt in the web directory.

Issue links

#7

Description

As suggested in #7 (comment)

• Provide the optional composer.json assets types config: drupal-asset-files
• Remove the default value from $assetFileTypes property
• Create a new method setAssetFileTypes() that will:
  • Set a default list of files copied from $assetFileTypes (include robots.txt)
  • Merge drupal-asset-files list to the list (if defined)
  • Trigger an event post-drupal-set-asset-file-types to allow other plugins alter the list of files
  • Set $assetFileTypes property value
• Call setAssetFileTypes() at __construct()
• Remove .php values from the list on createAssetSymlinks() method.
• Update README with the new optional config drupal-asset-files
• Update travis tests script test-script.sh.

I choose to remove .php values from the list on createAssetSymlinks() method and not setAssetFileTypes() because if in the last one we just use a simple preg_match on the list to remove them people could still trick it with something like mycustomfile.* and a file named mycustomfile.php.

[jkribeiro]

@brambaud Thanks for the contribution.

[jkribeiro]

FYI I've extended the list to:

+ $finder->notName('*.install'); + $finder->notName('*.module'); + $finder->notName('*.php'); + $finder->notName('*.profile'); + $finder->notName('*.theme');``` https://github.com/drupal-composer/drupal-paranoia/commit/160a9337b91224eecf3e9671480a3cb1381e457c#diff-c984225cadcacb6a40cb298c78b32434R260