Skip to content

Conversation

@AyanSinhaMahapatra
Copy link
Member

@AyanSinhaMahapatra AyanSinhaMahapatra commented Sep 12, 2022

  1. Modifies the License and Rule files to have their data as YAML frontmatter.
  • .RULE files and .LICENSE files now also contain their data.
  • git history is preserved
  • loading/dumping methods are modified to support YAML frontmatter
  • License/Rule now has different methods to point to their file.
  1. Adds a new command line option: --get-license-data to:
  • Dump license data in JSON, YAML and HTML formats.
  • Also dumps the .LICENSE file with text and data as YAML frontmatter.
  • Generates an index and a static website to view the data.

This is reusing code originally located at:
https://github.com/nexB/scancode-licensedb

Reference: #2738
Reference: #3049

This also contains PR: #3063 btw.

Tasks

  • Reviewed contribution guidelines
  • PR is descriptively titled 📑 and links the original issue above 🔗
  • Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR
    Run tests locally to check for errors.
  • Commits are in uniquely-named feature branch and has no merge conflicts 📁
tdruez and others added 30 commits December 4, 2020 16:30
Signed-off-by: Thomas Druez <tdruez@nexb.com>
Signed-off-by: Thomas Druez <tdruez@nexb.com>
Signed-off-by: Thomas Druez <tdruez@nexb.com>
Signed-off-by: Thomas Druez <tdruez@nexb.com>
Signed-off-by: Thomas Druez <tdruez@nexb.com>
Signed-off-by: Thomas Druez <tdruez@nexb.com>
Signed-off-by: Thomas Druez <tdruez@nexb.com>
 * Expose spdx_license_key, other_spdx_license_keys, and is_exception Signed-off-by: Jono Yang <jyang@nexb.com>
Expose SPDX license info in index.json #7
Signed-off-by: tdruez <tdruez@nexb.com>
Signed-off-by: tdruez <tdruez@nexb.com>
Signed-off-by: Jono Yang <jyang@nexb.com>
Expose is_deprecated in index.json
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
This commit ends the process of merging .RULE and .yml files into a single .RULE file which has YAML frontmatter storing the rule metadata present in the .yml file previously. This renaming and merging has been done to preserve line-history for both the files. Reference: #3049 Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
* Import and use functions/classes from frontmatter * Copy and modify class to use saneyaml * Copy and modify load/parse functions * Add .ABOUT and LICENSE files. Signed-off-by: AyanSinhaMahapatra <ayansmahapatra@gmail.com>
Reference: aboutcode-org/scancode-licensedb#25 Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
Regen test expectations for this and also fix another test. Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
Copy link
Member

@pombredanne pombredanne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM... Please do one last check wrt. spaces:

from licensedcode.models import * gpl=db["gpl-2.0"] bottom = gpl.text.strip("\n") + "\n" top = saneyaml.dump(gpl.to_dict()) fm=f"---\n{top}\n---\n{bottom}"

IMHO redumping all licenses as frontmatter this way should be to help that we are not damaging space-based formatting. In particualr watch out for the GPL -2.0.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra AyanSinhaMahapatra force-pushed the add-license-dump-option branch 2 times, most recently from 8b1d58e to 093b8b4 Compare November 1, 2022 11:09
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Copy link
Member Author

@pombredanne post your approval I've pushed the following changes:

  1. Fix whitespace in License texts
  2. Merge the latest changes in develop including external licenses work and make tests pass.
  3. Update changelog

Should I go ahead and merge?

@AyanSinhaMahapatra
Copy link
Member Author

AyanSinhaMahapatra commented Nov 1, 2022

Some scripts need update but I opened an issue: #3138 to track that separately to not block this anymore.

@AyanSinhaMahapatra AyanSinhaMahapatra merged commit 9f91bf5 into develop Nov 2, 2022
@pombredanne pombredanne deleted the add-license-dump-option branch November 14, 2022 16:53
@AyanSinhaMahapatra
Copy link
Member Author

Note that the command line option was later renamed from --get-license-data to --dump-license-data later.

sschuberth added a commit to oss-review-toolkit/ort that referenced this pull request Nov 16, 2023
ScanCode 32.0.0 started to prepend its `*.LICENSE` files with YAML-encoded metadata, see [1]. This is a hot fix to remove this header, if present, from the license files. A better solution will be implemented later as part of a larger refactoring of license providers. Different ScanCode versions also differ in whether license files come with a final newline or not. Align on not having a final newline to make tests pass either way. [1]: aboutcode-org/scancode-toolkit#3100 Signed-off-by: Sebastian Schuberth <sebastian@doubleopen.org>
sschuberth added a commit to oss-review-toolkit/ort that referenced this pull request Nov 16, 2023
ScanCode 32.0.0 started to prepend its `*.LICENSE` files with YAML-encoded metadata, see [1]. This is a hot fix to remove this header, if present, from the license files. A better solution will be implemented later as part of a larger refactoring of license providers. Different ScanCode versions also differ in whether license files come with a final newline or not. Align on not having a final newline to make tests pass either way. [1]: aboutcode-org/scancode-toolkit#3100 Signed-off-by: Sebastian Schuberth <sebastian@doubleopen.org>
sschuberth added a commit to oss-review-toolkit/ort that referenced this pull request Nov 17, 2023
ScanCode 32.0.0 started to prepend its `*.LICENSE` files with YAML-encoded metadata, see [1]. This is a hot fix to remove this header, if present, from the license files. A better solution will be implemented later as part of a larger refactoring of license providers. Different ScanCode versions also differ in whether license files come with a final newline or not. Align on not having a final newline to make tests pass either way. [1]: aboutcode-org/scancode-toolkit#3100 Signed-off-by: Sebastian Schuberth <sebastian@doubleopen.org>
sschuberth added a commit to oss-review-toolkit/ort that referenced this pull request Nov 17, 2023
ScanCode 32.0.0 started to prepend its `*.LICENSE` files with YAML-encoded metadata, see [1]. This is a hot fix to remove this header, if present, from the license files. A better solution will be implemented later as part of a larger refactoring of license providers. Different ScanCode versions also differ in whether license files come with a final newline or not. Align on not having a final newline to make tests pass either way. [1]: aboutcode-org/scancode-toolkit#3100 Signed-off-by: Sebastian Schuberth <sebastian@doubleopen.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

4 participants