Skip to content

feat: new fuse table option enable_parquet_encoding #17675

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 6 commits into
base: main
Choose a base branch
from
Draft

feat: new fuse table option enable_parquet_encoding #17675

wants to merge 6 commits into from

Conversation

@dantengsky
Copy link
Member

@dantengsky dantengsky commented Mar 31, 2025 •
edited
Loading

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

New fuse table option enable_parquet_encoding, currently for evaluation only.

When this option is set to true(by default false), fuse table with parquet storage format will enable encoding during serialization, and the default encoding policy of parquet-rs will be used.

e.g.

CREATE TABLE T ... enable_parquet_encoding = 'true'

or

ALTER TABLE T SET OPTIONS (enable_parquet_encoding = 'true')

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable
@github-actions github-actions bot added the pr-feature this PR introduces a new feature to the codebase label Mar 31, 2025
@dantengsky dantengsky force-pushed the feat-parquet-encoding branch from 88aada7 to 1e1b52e Compare March 31, 2025 12:19
dantengsky added 5 commits May 21, 2025 15:47
@dantengsky
feat: new table option FUSE_OPT_KEY_ENABLE_PARQUET_ENCODING
29ccc92 
When this option is set to true, fuse table with parquet storage format will enable encoding and dictionary during serialization. The defult encoding policy of paquet-rs will be used.
@dantengsky
tweak logic test of parquet encoding
52817b3
@dantengsky
fix logic test
f363a88
@dantengsky
tweak logic test
f664f8b
@dantengsky
tweak logic case
fe8689f
@dantengsky dantengsky force-pushed the feat-parquet-encoding branch from 3a47a4a to fe8689f Compare May 21, 2025 09:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-feature this PR introduces a new feature to the codebase

1 participant

@dantengsky