Sparknlp-1174 Adding Partitioning Documents Feature

[danilojsl]

Description

This PR introduces the new Partition component and PartitionTransformer annotator to spark-nlp, providing a streamlined and user-friendly interface for interacting with Spark NLP readers. It also extends the configurability of these readers by adding support for additional options.

Key Improvements:

• Simplifies integration with Spark NLP readers through a unified interface
• Adds flexibility by enabling more reader-specific configurations
• Enhances the maintainability and scalability of data loading workflows

Motivation and Context

The current approach to interfacing with Spark NLP readers is fragmented and lacks flexibility, often requiring custom code for handling various input sources and options. This makes onboarding harder for new users and hinders reuse across pipelines.

The Partitioning component abstracts these complexities by:

• Unifying access patterns for multiple readers
• Reducing boilerplate code in reader configuration
• Making it easier to scale and switch between different data sources

How Has This Been Tested?

Screenshots (if appropriate):

• Local Tests
• Google Colab Notebooks
• Databricks Notebooks

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• Code improvements with no or little impact
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING page.
• I have added tests to cover my changes.
• All new and existing tests passed.

[coveralls]

Pull Request Test Coverage Report for Build 15257196653

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 550 of 666 (82.58%) changed or added relevant lines in 25 files are covered.
• 67 unchanged lines in 49 files lost coverage.
• Overall coverage increased (+0.8%) to 53.543%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
src/main/scala/com/johnsnowlabs/partition/HasEmailReaderProperties.scala	4	5	80.0%
src/main/scala/com/johnsnowlabs/partition/HasPowerPointProperties.scala	4	5	80.0%
src/main/scala/com/johnsnowlabs/reader/util/HasPdfProperties.scala	28	29	96.55%
src/main/scala/com/johnsnowlabs/partition/HasExcelReaderProperties.scala	7	9	77.78%
src/main/scala/com/johnsnowlabs/reader/EmailReader.scala	7	9	77.78%
src/main/scala/com/johnsnowlabs/partition/HasHTMLReaderProperties.scala	5	8	62.5%
src/main/scala/com/johnsnowlabs/reader/TextReader.scala	20	23	86.96%
src/main/scala/com/johnsnowlabs/reader/WordReader.scala	22	25	88.0%
src/main/scala/com/johnsnowlabs/reader/util/TextParser.scala	24	27	88.89%
src/main/scala/com/johnsnowlabs/partition/HasReaderProperties.scala	25	29	86.21%

Files with Coverage Reduction	New Missed Lines	%
src/main/scala/com/johnsnowlabs/nlp/annotators/cleaners/Extractor.scala	1	93.66%
src/main/scala/com/johnsnowlabs/nlp/annotators/common/TableData.scala	1	85.19%
src/main/scala/com/johnsnowlabs/nlp/annotators/common/Tagged.scala	1	67.82%
src/main/scala/com/johnsnowlabs/nlp/annotators/cv/util/io/ImageIOUtils.scala	1	54.22%
src/main/scala/com/johnsnowlabs/nlp/annotators/DateMatcher.scala	1	97.25%
src/main/scala/com/johnsnowlabs/nlp/annotators/DateMatcherTranslator.scala	1	89.35%
src/main/scala/com/johnsnowlabs/nlp/annotators/DateMatcherUtils.scala	1	88.29%
src/main/scala/com/johnsnowlabs/nlp/annotators/DocumentCharacterTextSplitter.scala	1	96.72%
src/main/scala/com/johnsnowlabs/nlp/annotators/keyword/yake/YakeKeywordExtraction.scala	1	96.79%
src/main/scala/com/johnsnowlabs/nlp/annotators/MultiDateMatcher.scala	1	99.07%

Totals
Change from base Build 15207655495:	0.8%
Covered Lines:	10344
Relevant Lines:	19319

---

💛 - Coveralls