Add support for external licenses in scans

[KevinJi22]

This is for the 2022 Google Summer of Code. This PR contains all the work that needs to be done for the project.

This adds --additional-license-directory as a command line option in license reindexing. This allows users to specify paths to directories of licenses and rules they'd like to use during license detection, but would not like to add to the ScanCode database of licenses. First, the user must run Scancode with the --reindex-licenses option and they can use --additional-license-directory to add directories of licenses or rules to the index.

This involves adding a new option in licensedcode/plugin_license.py, and this option is used as a parameter in scancode/api.py. In this approach, the licenses and rules contained in these additional directories are combined with the existing licenses and rules in the ScanCode database to produce a single index. The code for this is found in licensedcode/cache.py and the helper methods for loading these licenses and rules are found in licensedcode/models.py.

This commit also includes unit tests to verify that license detection succeeds with additional directories and installed licenses/rules. Part of the setup for the unit test and future tests involves creating a new directory in tests/licensedcode/data that contains sample external licenses used in the unit tests.

Fixes #480

Tasks

• Reviewed contribution guidelines
• PR is descriptively titled 📑 and links the original issue above 🔗
• Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR
  Run tests locally to check for errors.
• Commits are in uniquely-named feature branch and has no merge conflicts 📁

[JonoYang]

@KevinJi22 I think this is looking good so far! I left some notes about docstrings and style.

[pombredanne]

There is something in your changes that make the tests fail...
I do not think you can just use multiple cached directories. You should use a single cache and then validate, then combine multiple input a single cached index.

[KevinJi22]

@pombredanne I'm actually combining all the licenses and rules in the input directories into a single index here. In load_licenses_from_multiple_dirs, all the licenses (including the ones in the ScanCode database) are combined into a single license database, and the same is done with all the rules. The _CACHED_DIRECTORIES variable is used to determine whether a new index needs to be built. For example, if we've created an index with additional directories A and B, but now we run a scan with additional directories B and C, we'd need to recreate the combined index.

I'll update this code with comments so that this is more clear.

https://github.com/nexB/scancode-toolkit/blob/b1cf4aa16386e447b024b2fd3f5386251560628d/src/licensedcode/cache.py#L182-L195

[JonoYang]

I've added a few nitpicks about formatting. I have an issue trying out the new -dir option. I am running scancode -l -dir tests/licensedcode/data/example_external_licenses/example1 --json-pp ~/Desktop/out.json /home/jono/src/scancode-toolkit-kevinji22/tests/licensedcode/data/plugin_license/external_licenses/scan/, however I am not seeing the custom license being detected in the scan results.

[KevinJi22]

Thanks for the feedback! This bug was actually due to how os.path.join() works, since the path that you passed in wasn't an absolute path, so the paths were just concatenated in the function build_rule_from_license() in models.py. This meant that the external license wasn't being loaded in as a rule. I added some code in get_license_dirs() that converts the input path to an absolute path, which will hopefully prevent this from happening in the future.

This means that running the scan with the same parameters should work this time.

[JonoYang]

Looks good! After talking to @pombredanne, the next thing we want on top is to be able to install custom licenses using wheels. Something similar to the way we handle providing binaries to scancode via plugins: https://github.com/nexB/scancode-plugins/tree/main/builtins

We also merged the fix for the PyPI test in develop, so be sure to rebase your branch.

[JonoYang]

@KevinJi22 Don't feel compelled to amend your commit and force push it every time you make a change. You can create as many commits as you want on this PR. 🙂

[AyanSinhaMahapatra]

@KevinJi22 we (@pombredanne and me) added the following changes:

CLI Options:

• various reindexing operations are no longer CLI options
• we have a seperate script scancode-reindex-licenses which does that
• only one additional directory of licenses/rules can be added, i.e. this is not a multiple option

Tests:

• Organize tests data under one single folder.
• Organize test functions under one single file.
• Keep three tests, one for additional directory, one for
  additional plugin, and one for combined testing.
• Edit the CI file accordingly.

Misc:

• code refactoring at get_rules_from_multiple_dirs, get_rules_from_multiple_dirs etc
• external license directories/plugin info is returned in scan header

Remaining work:

• update how license data (links and other) is returned in api.py
• also report new attribute is_builtin
  @pombredanne ^ this will create a lot of test file updates, I was thinking I'll do this update in the main LicenseDetection PR as there's lot's of change in license data reporting already.

Otherwise tests are all green and this is ready AFAIK.
Please review. 🙇

[AyanSinhaMahapatra]

Remaining work:

• update how license data (links and other) is returned in api.py
• also report new attribute is_builtin
  @pombredanne ^ this will create a lot of test file updates, I was thinking I'll do this update in the main LicenseDetection PR as
  there's lot's of change in license data reporting already.

Modified the code to add is_builtin flags to license match data when there is an additional license directory or additional license plugins installed. So this work is not remaining anymore.

Ready to review @pombredanne!

Was also wondering if there could be a --only-builtin CLI option in scancode-reindex-licenses to reindex the license freshly without having to delete and reconfigure the environment, but not sure how to uninstall plugins from the python script. @pombredanne is this doable?

[pombredanne]

Was also wondering if there could be a --only-builtin CLI option in scancode-reindex-licenses to reindex the license freshly without having to delete and reconfigure the environment, but not sure how to uninstall plugins from the python script. @pombredanne is this doable?

Sure, but remember that each new added option may be a sign of our failure to automatically do the right thing without options :)

[pombredanne]

Here some extra feedback. I think we should refactor this a bit more such that indexing licenses and rules just takes a list of directories. Everything else should IMHO just use the things already loaded. And we can simplify the detection of builtins based on a list of builtin licenses... This would help keep the code tidier IMHO. @AyanSinhaMahapatra @KevinJi22 let's chat at your convenience

[pombredanne]

Here some extra feedback. I think we should refactor this a bit more such that indexing licenses and rules just takes a list of directories. Everything else should IMHO just use the things already loaded. And we can simplify the detection of builtins based on a list of builtin licenses... This would help keep the code tidier IMHO. @AyanSinhaMahapatra @KevinJi22 let's chat at your convenience

I created #3137 as a follow up!

[pombredanne]

All good to go. Merging