Skip to content

Conversation

@adwiteeymauriya
Copy link

Summary

This PR introduces a complete Go implementation of a Docker image push CLI for the serverless-registry, specifically designed to handle large container images that exceed Cloudflare Workers' 500 MB request body limit.

Key Features

  • Chunked Upload Support: Automatically reads the oci-chunk-max-length header from the registry (default 470MB) to determine optimal chunk sizes, enabling uploads of arbitrarily large image layers
  • Concurrent Processing: Configurable concurrent uploads (default 5) using goroutines with semaphore pattern for faster performance
  • Layer Caching: Smart caching system that stores compressed layers with pointer files to avoid recompression on subsequent pushes
  • Retry Logic: Configurable retry attempts (default 3) for handling transient network failures
  • Authentication: Secure Basic authentication with password via stdin to prevent exposure in process lists or command history
  • Protocol Flexibility: Support for both HTTPS and HTTP (for local/dev registries) via --insecure flag

Implementation Details

The tool follows the complete OCI Distribution Spec workflow:

  1. Exports Docker image using docker save
  2. Extracts and compresses each layer to gzip format
  3. Calculates SHA256 digests for all blobs (layers + config)
  4. Uploads blobs in chunks respecting registry limits
  5. Constructs and uploads OCI-compliant manifest

Technical Highlights

  • Zero external dependencies except golang.org/x/term for secure password input
  • Uses Go's native net/http client and standard library for all operations
  • Type-safe implementation with proper error handling
  • Cross-platform binary support (no runtime required)
  • Memory-efficient streaming for large files

Usage

# Build the tool cd push-go && go build -o docker-push # Push an image echo "password" | ./docker-push -username myuser my-registry.com/image:tag # With custom settings echo "password" | ./docker-push \ -username myuser \ -concurrency 10 \ -max-retries 5 \ my-registry.com/large-image:latest

Files Added

  • push-go/main.go - Complete implementation (699 lines)
  • push-go/README.md - Comprehensive documentation with examples
  • push-go/go.mod & push-go/go.sum - Go module definition
  • push-go/.gitignore - Ignore cache, build artifacts, and temporary files

Testing

Tested with:

  • Multi-GB PyTorch images with layers exceeding 500 MB
  • Concurrent uploads of 10+ layers
  • Retry logic under network disruption
  • Both authenticated and anonymous (local registry) scenarios

Test plan

  • Build the binary successfully
  • Push a small image (<100 MB) to verify basic functionality
  • Push a large image with layers >500 MB to test chunking
  • Verify layer deduplication works (push same image twice)
  • Test concurrent uploads with multiple layers
  • Verify retry logic handles transient failures
  • Test with both HTTP and HTTPS registries
  • Confirm authentication works correctly
  • Integration test with actual Cloudflare serverless-registry deployment
@laopaoer-wallet
Copy link

support podman?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants