add support for setting mutipart upload threshold (logstash-plugins#202)

* add support for setting mutipar upload threshold * Add upload_multipart_threshold to docs * Apply sorting in docs * CHANGELOG and version bump

Author: shamil

CHANGELOG.md

@@ -1,3 +1,6 @@
+## 4.1.8
+ - Add support for setting mutipart upload threshold [#202](https://github.com/logstash-plugins/logstash-output-s3/pull/202)
+
 ## 4.1.7
  - Fixed issue where on restart, 0 byte files could erroneously be uploaded to s3 [#195](https://github.com/logstash-plugins/logstash-output-s3/issues/195)
 

docs/index.asciidoc

@@ -101,6 +101,7 @@ This plugin supports the following configuration options plus the <<plugins-{typ
 | <<plugins-{type}s-{plugin}-storage_class>> |<<string,string>>, one of `["STANDARD", "REDUCED_REDUNDANCY", "STANDARD_IA"]`|No
 | <<plugins-{type}s-{plugin}-temporary_directory>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-time_file>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-upload_multipart_threshold>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-upload_queue_size>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-upload_workers_count>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-validate_credentials_on_root_bucket>> |<<boolean,boolean>>|No
@@ -352,6 +353,14 @@ If you define file_size you have a number of files in consideration of the secti
 0 stay all time on listerner, beware if you specific 0 and size_file 0, because you will not put the file on bucket,
 for now the only thing this plugin can do is to put the file when logstash restart.
 
+[id="plugins-{type}s-{plugin}-upload_multipart_threshold"]
+===== `upload_multipart_threshold`
+
+ * Value type is <<number,number>>
+ * Default value is `15728640`
+
+Files larger than this number are uploaded using the S3 multipart APIs
+
 [id="plugins-{type}s-{plugin}-upload_queue_size"]
 ===== `upload_queue_size` 
 

lib/logstash/outputs/s3.rb

@@ -160,6 +160,9 @@ class LogStash::Outputs::S3 < LogStash::Outputs::Base
  # Number of items we can keep in the local queue before uploading them
  config :upload_queue_size, :validate => :number, :default => 2 * (Concurrent.processor_count * 0.25).ceil
 
+ # Files larger than this number are uploaded using the S3 multipart APIs. Default threshold is 15MB.
+ config :upload_multipart_threshold, :validate => :number, :default => 15 * 1024 * 1024
+
  # The version of the S3 signature hash to use. Normally uses the internal client default, can be explicitly
  # specified here
  config :signature_version, :validate => ['v2', 'v4']
@@ -296,7 +299,8 @@ def upload_options
  :server_side_encryption => @server_side_encryption ? @server_side_encryption_algorithm : nil,
  :ssekms_key_id => @server_side_encryption_algorithm == "aws:kms" ? @ssekms_key_id : nil,
  :storage_class => @storage_class,
- :content_encoding => @encoding == "gzip" ? "gzip" : nil
+ :content_encoding => @encoding == "gzip" ? "gzip" : nil,
+ :multipart_threshold => @upload_multipart_threshold
  }
  end
 

logstash-output-s3.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
  s.name = 'logstash-output-s3'
- s.version = '4.1.7'
+ s.version = '4.1.8'
  s.licenses = ['Apache-2.0']
  s.summary = "Sends Logstash events to the Amazon Simple Storage Service"
  s.description = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"

spec/outputs/s3_spec.rb

@@ -60,6 +60,23 @@
  end
  end
 
+ describe "Multipart upload threshold" do
+ context "when configured" do
+ it "should use the configured threshold" do
+ threshold = 1 * 1024 * 1024
+ s3 = described_class.new(options.merge({ "upload_multipart_threshold" => threshold }))
+ expect(s3.upload_options).to include(:multipart_threshold => threshold)
+ end
+ end
+
+ context "when not configured" do
+ it "should use 15MB as the default" do
+ s3 = described_class.new(options)
+ expect(s3.upload_options).to include(:multipart_threshold => 15 * 1024 * 1024)
+ end
+ end
+ end
+
  describe "Service Side Encryption" do
 
  context "when configured" do