Questions tagged [compression]
Decrease the size of data, typically to preserve space or reduce network bandwidth required while transmitting files. It can be applied to files, folders, disks, etc.
422 questions
0 votes
0 answers
72 views
Zipping an S3 folder across AWS Batch containers
I am building a data pipeline where the final step for each submitted job will be zipping all the files inside the job folder, sitting in an Amazon S3 bucket, into a single zip file. I'm evaluating ...
- 101
0 votes
2 answers
131 views
Disabling imap compression in Dovecot
My dovecot imap server use deflate compression for connection. How can I disable it for testing? I tried to add plugin { imap_compress_deflate_level = 0 } to custom config in /etc/dovecot/conf.d/, ...
- 103
3 votes
1 answer
1k views
Does "btrfs filesystem defrag -c" (compress option) force compression?
BTRFS with compression enabled uses a heuristic not to compress files that are not well compressible. Does compressing existing files with "btrfs filesystem defrag -c" also use the heuristic,...
- 1,821
0 votes
0 answers
1k views
Re-compressing XZ kernel module?
I have a kernel module produced by kernel option CONFIG_MODULE_COMPRESS_XZ. When I manually decompress the file with unxz and re-compress with xz, the new .ko.xz file differs from the original .ko.xz ...
- 71
1 vote
0 answers
69 views
zfs send to NAS with PIGZ compression slowness
The following command that I use to backup used to take 5 minutes to complete and since last week it is taking around 25 minutes. This is after restarting a switch stack the server is connected to. ...
- 11
2 votes
1 answer
405 views
Ways to make deployment ready pre-compressed brotli files on windows server
I am dynamically generating HTML files, which I use for deployment. This is what I use to make these HTML file: function writeFile() Dim objStream Set objStream = CreateObject("ADODB....
- 294
1 vote
2 answers
220 views
Nginx performance issues
I made a website using React and connected to Strapi. But I need to keep some background videos mainly on the home page. The main banner has 4 videos and the background of the page is also a video. ...
1 vote
2 answers
923 views
Will disk compression impact the performance of a MySQL database?
I've recently setup a TrueNAS storage server and hosted on it is a MySQL database. It's about 3GB; I'm not too concerned with how much space it's utilising. By default, TrueNAS enables LZ4 compression ...
- 113
1 vote
1 answer
666 views
Cloudflare uncompressing, then re-compressing response bodies
I'm using Node.js/express and if I set the headers Content-Type: application/json Content-Encoding: gzip and send gzipped compressed bodies of JSON data, this works as expected - when not behind ...
- 111
0 votes
1 answer
310 views
Debian: Simple way to compress files wanted
I often need to compress archives in Linux. is there a simpler way instead of always building such complicated commands? tar zcvf /tmp/mybackup.tar.gz /home/important
- 783
0 votes
0 answers
5k views
is there a faster way to extract a large file tar in lz4 format?
I have a 7 TB file that is tared with lz4 compression. It takes about 8 hours to extract. is there a faster way to do this? This is the command I use: lz4 -d /mnt/tmp/7TBFile.lz4 | tar -xvf -
2 votes
0 answers
735 views
Are servers allowed to add a Content-Encoding header based on filename, if there is no compression on-the-fly?
Problem Suppose we have a compressed archive on disk, e.g. file.tar.gz, which should be served as-is. The file is served with Content-Type: application/gzip, but for some reason the server also adds a ...
- 183
1 vote
1 answer
609 views
Compressing rarely queried historical data in postgres
In short: compress multiple historical DB records to bigger, compressed records and temporarily uncompress it to the original records for easier querying, and clean that up when querying is done. ...
- 171
-1 votes
1 answer
933 views
MySQL JDBC compression in transit
Good morning, I am looking to set my application connection to MySQL to compress data in transit. I read MySQL doc. but it's not cleat to me if I need to do something on the client side, on the server ...
- 858
0 votes
1 answer
2k views
Returning compressed data when client omits accept-encoding header
In the HTTP 1.1 Standard, it says that "If no Accept-Encoding field is in the request, any content-coding is considered acceptable by the user agent." Which means that a server can, for ...
- 237
1 vote
1 answer
483 views
Compress input files from rsync before writing to disk
I need to load big text files from another machine through a 10Gb link, files been creating by external/closed-source software that i don't have access to change (for example to make it to compress ...
- 111
0 votes
2 answers
4k views
scp with -C flag does not perform compression
The scp man page shows that the -C flag can be used for compressing a file on the fly while performing a remote copy - $ man scp | grep "\-C" -C Compression enable. Passes the -C ...
- 123
0 votes
1 answer
31 views
Changes in compression rate
Every month I usually compress all my database dumps into a rar file. I was very happy with the compression rate, as each compressed file was usually about 30MB, with a compression rate of 99%. Since ...
- 265
0 votes
1 answer
860 views
Stratis filesystem will not mount after a reboot because the disk is full
It concerns a rhel8 server that has two disks (sda and sdb), and only on the sdb disk a Stratis filesystem has been created. After a restart of the server, no more mount point appears on the sdb disk. ...
0 votes
0 answers
594 views
community.windows.win_unzip shows Filename contains relative paths which would extract outside the destination error message
I'm trying to unzip a zip file in a remote server This file was download first from jFrog repository The following task should handle the unzip: - name: Unzip Config.zip file community.windows....
- 135
0 votes
0 answers
404 views
Transfer Compression with Cloudfront and nginx SSI
Is there any way to enable GZIP / Brotli compression on Cloudfront if the origin is a nginx server with SSI (Server Side Includes) enabled? The issue seems that nginx SSI drops the Content-Length ...
- 151
0 votes
1 answer
259 views
ZFS Transfer extremely bursty
I am testing a new ZFS configuration with z-std for log storage and storage of highly compressible files. The Array is tested on a 5 drive raidz-1 in a virtual machine on my PC which has direct access ...
- 11
5 votes
1 answer
731 views
Unable to compress 18GB file on Ubuntu Linux 18.04
It never happened to me before, but I'm unable to do a simple task such as compressing a 18.5 GB file on Ubuntu Linux 18.04 with any of the popular compressing tools such as gzip, bzip2 and 7z. All of ...
- 181
0 votes
1 answer
3k views
How do I make a compressed disk image (qcow2 and/or squashfs) from a disk directly?
I have a disk attached to my server and I want to make the disk image of it downloadable on the same server. The disk itself is 500GB but I don't want to have a 500GB disk image for download. I'd like ...
2 votes
2 answers
2k views
Cannot turn off SSLCompression in Apache?
My Apache is 2.4.46 and is using Openssl version 1.1.1f I have set the directive SSLCompression Off. Even if i enable it, it says SSL Compression is not supported, which is good I guess. But, when I ...
user549144
0 votes
2 answers
784 views
Tool to create pre-compressed [Brotli] files
I am trying to pre-generate compressed brotli files in designated directories for static html, css and js files. I found a rust tool here – https://github.com/neosmart/static-compress Besides this, ...
- 21
0 votes
1 answer
1k views
Preserve Cloudfront brotli compression through a nginx reverse_proxy
My question is similar to Nginx reverse-proxy to a cloudfront distribution and preserve gzip compression but especially regarding the new brotli compression algorithm that is supposed to perform ...
1 vote
0 answers
735 views
btrfs compress existing data with snapshots
I'm runninga a btrfs raid1 on 5 8 TB disks, where primarily urbackup stores backups to. Urbackup uses snapshots extensively. compsize produces the following output: Processed 106161732 files, 57335496 ...
- 111
0 votes
0 answers
139 views
Opera Mini Compression Technology
I'm trying to find the technology used in Opera Mini's browser. I can't find anything related to this, so I would like to know if anyone knows whether there exists a paper on this or if there is ...
- 101
1 vote
1 answer
9k views
How to install nginx-module-brotli?
I have a server with ubuntu 20.04 But I am unable to install nginx-module-brotli for my nginx server. Here are the commands I use to install Brotli for Nginx : $ sudo apt install git gcc cmake ...
- 39
0 votes
1 answer
238 views
How to syncronize XML files and store them as .gz at destination using rsync
I'm using rsync -z to backup a large amount of XML files. The source is a huge machine with thousands of GBs available on disk, but the destination is not that powerful. What I want to do is to ...
5 votes
1 answer
4k views
Compress old emails with dovecot
How can I configure dovecot to compress/archive old emails (say 6 months old) with the users still being able to read them, to save some precious space on the server?
- 222
0 votes
0 answers
208 views
tar | pigz dies if run under cron
I run a kind of backup script using tar joint with pigz for compression. The problem is that tar -zcf /file.tar.gz /folder runs under cron. While tar -I pigz -cvf /file.tar.gz /folder nor tar cvf ...
- 151
0 votes
0 answers
325 views
Nginx serve another file when possible
I currently compile my assets with Webpack's BrotliPlugin, which creates a separate file, brotli-encoded, i.e.: style.css style.cssb Here's what I want to achieve: Consider the HTML page requests ...
- 101
1 vote
0 answers
243 views
haproxy compression type regex/wildcard
As stated in haproxy docs, we can use "compression type" with list of mime types. In my case, there is REST API backend, and the compression should be applied to multiple vendor-specific versioned ...
- 503
2 votes
2 answers
2k views
Extracting single file from large tar.gz file
I have an incredibly large tarball. I'd extract several files out of many thousands within the archive. I'm on CentOS 6.10 running GPFS 4.2.3. I've seen from this answer that pigz is useful in ...
0 votes
1 answer
111 views
How do I find out why compact command appears to hang on certain files?
After extending the hard disk sizes on a Windows 2008 server I am trying to revert the compressing of a lot of files on the hard drives. I use the command compact /u /s /i /f /q for it. But it ...
- 1
0 votes
2 answers
1k views
How to untar a very large file with missing space
I have reinstalled my server and before that i did a backed up. My backup is a tar file without special compress option (meaning that the size of the tar file is approximately the same than the size ...
3 votes
1 answer
6k views
How to install the brotli nginx module properly on debian
I'm trying to setup brotly compression on a nginx/1.10.3 server running on Debian 9.5 Stretch / Linux 4.9.0-8-amd64 #1 SMP Debian 4.9.110-3+deb9u5 (on a lightsail instance). I used the following ...
- 133
0 votes
1 answer
968 views
Script isn't served compressed from IIS
I have IIS 10 running on Windows 2016 (I see exactly same behavior on Windows 2019). I enabled static compression. I can see that the browser (Chrome) sends accept-encoding: gzip, deflate, br. I can ...
- 563
1 vote
1 answer
951 views
Apache mod_deflate MIME type settings
CPanel gives me options about what MIME content types to compress with Apache's mod_deflate, as shown below. Are there disadvantages to having a long list of MIME types (eg. slower server response)? ...
- 303
0 votes
1 answer
3k views
Can not run Brotli on my Nginx server
I installed Brotli on Ubuntu 18.04 with Nginx. Here is the procedure I followed : $ sudo apt-add-repository ppa:hda-me/nginx-stable $ sudo apt update $ sudo apt install brotli nginx nginx-module-...
- 57
4 votes
2 answers
11k views
How to install Brotli on nginx and Ubuntu 18.04?
I installed Brotli on Ubuntu 18.04 with Nginx. Here is the procedure I followed : $ sudo apt-add-repository ppa:hda-me/nginx-stable $ sudo apt update $ sudo apt install brotli nginx nginx-module-...
- 57
5 votes
1 answer
3k views
What kind of algorithm is used in LTO tape hardware compression?
LTO tape drives, from their very first generation, offer hardware compression that, theoretically, allows a maximum of 2 - 2.5x the rated data capacity of each cartridge to be stored, with only a ...
- 1,447
1 vote
1 answer
213 views
How to enable IIS to always precompress / pregzip a certain file?
I have a 4mb js file (gzipped version is 900kb). The file is like this: main.328479fdsdf.js (so it has a hash based on the content). IIS seems to compress my big js file only if is accessed 2 times ...
- 111
0 votes
1 answer
725 views
Use of gzip with multiple, sequential nginx proxies
Consider the following server setup on a network: In this example, both of the proxy servers are Nginx servers. If the aim is to reduce load time for the client, should compression be used by the ...
- 49
4 votes
2 answers
661 views
Are LTO Hardware compression and encryption standardized?
I'm about to setup a backup using an LTO-4 Drive. I know that drives are meant to be compatible to any tape within the past 3 generations (so I should be able to read an LTO-4 tape using an LTO-4, 5 ...
- 4,090
1 vote
2 answers
3k views
Modern transparent folder compression in Ubuntu 18.04
I'm looking for a way to transparently compress files in a directory, that will be shared over SMB. I have a server, running Ubuntu 18.04.1 LTS, and I use it for file storage. Mainly what I want to ...
1 vote
0 answers
674 views
Why do i get gzip css and js with full size instead of compressed size
I am doing some tests with gzip, I have manually compressed the CSS and JS with gzip. The size of the JS has dropped from 33kb to 7kb. The problem is when I look in the network tab in Chrome or ...
- 111
-2 votes
1 answer
492 views
openvpn optimizations for Low bandwidth (250mb a month)
I am using openwrt lede with a lte stick 250-300mb a month depending on the unit so preferably hitting 250 or lower would be better for the entire system these are the clients the server doesn't have ...
