Creating Custom File Systems: Implementing the FS Interface 7/9

Design Principles for Custom File Systems

When building a custom file system that implements Go's fs.FS interface, you need to establish clear design principles that will guide your implementation decisions. These principles ensure your file system behaves predictably and integrates seamlessly with existing Go code.

Interface Compliance Requirements

The foundation of any custom file system is strict adherence to the fs.FS interface contract. Your implementation must return consistent behavior that other Go packages can rely on. The core Open(name string) (fs.File, error) method serves as the entry point for all file operations, and its behavior sets the tone for your entire implementation.

Path handling requires careful attention. Your file system should accept forward slashes as path separators, regardless of the underlying platform. Invalid paths containing elements like ".." or absolute paths starting with "/" should return fs.ErrInvalid. This consistency allows your file system to work correctly with packages like http.FileServer or template.ParseFS.

File names must be validated according to the interface specification. Empty names, names containing null bytes, or names with invalid Unicode sequences should trigger appropriate errors. Your implementation should also handle edge cases like "." for the current directory, which some callers might use to explore the file system root.

Error Handling Conventions

Error handling in file systems requires a delicate balance between providing useful information and maintaining security. The fs package defines several sentinel errors that your implementation should use consistently. fs.ErrNotExist should be returned when files or directories don't exist, while fs.ErrPermission indicates access restrictions.

Custom error types become necessary when you need to provide additional context. However, ensure these errors still satisfy the expected interfaces. A well-designed file system often implements fs.PathError for operations that fail on specific paths, providing both the operation name and the path that caused the failure.

Consider the caller's perspective when crafting error messages. Avoid exposing internal implementation details that could confuse users or create security vulnerabilities. A HTTP-based file system, for instance, shouldn't leak server error details through file system error messages.

Thread Safety Considerations

Modern applications frequently access file systems concurrently, making thread safety a critical design consideration. Your file system implementation must handle multiple goroutines accessing the same resources without corruption or deadlocks.

The fs.FS interface itself doesn't require specific thread safety guarantees, but practical usage patterns demand it. Multiple goroutines might call Open() simultaneously, or different goroutines might read from files opened from the same file system instance. Your implementation should support these patterns safely.

Consider using read-write mutexes when your file system maintains internal state that supports both read and write operations. For read-heavy workloads, this allows multiple concurrent readers while still protecting against writers. However, be cautious about lock granularity - overly broad locks can create performance bottlenecks.

State management becomes particularly important in file systems that cache data or maintain connections to external resources. Design your caching strategies with concurrency in mind, potentially using sync.Map for concurrent access or implementing per-resource locking to reduce contention.

Memory-based file systems face unique challenges since they typically store all data in memory structures. These structures must support concurrent access while maintaining consistency. Consider using atomic operations for simple state changes and proper synchronization primitives for complex operations that span multiple data structures.

Basic FS Implementation

Building your first custom file system starts with understanding how to map abstract file operations to your underlying data storage. A memory-based implementation provides an excellent foundation for learning these concepts before moving to more complex storage backends.

Memory-Based File System Example

A memory-based file system stores all files and directories in RAM using Go's built-in data structures. This approach offers fast access and simplifies the initial implementation while teaching core concepts that apply to any storage backend.

type MemFS struct { mu sync.RWMutex files map[string]*memFile dirs map[string]*memDir } type memFile struct { data []byte modTime time.Time mode fs.FileMode } type memDir struct { entries []fs.DirEntry modTime time.Time } func NewMemFS() *MemFS { return &MemFS{ files: make(map[string]*memFile), dirs: make(map[string]*memDir), } } func (mfs *MemFS) Open(name string) (fs.File, error) { if !fs.ValidPath(name) { return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrInvalid} } mfs.mu.RLock() defer mfs.mu.RUnlock() if file, exists := mfs.files[name]; exists { return &memFileReader{file: file, name: name}, nil } if dir, exists := mfs.dirs[name]; exists { return &memDirReader{dir: dir, name: name}, nil } return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrNotExist} } 

The key insight here is separating the file system structure from the file handles returned to callers. The MemFS struct maintains the storage, while separate reader types handle individual file access. This separation allows multiple callers to open the same file simultaneously without interfering with each other.

File and Directory Simulation

Simulating files and directories requires implementing the various interfaces that callers expect. Files must implement fs.File, which includes methods for reading, seeking, and obtaining file information. Directories implement additional interfaces for listing contents.

type memFileReader struct { file *memFile name string offset int64 } func (r *memFileReader) Read(p []byte) (int, error) { if r.offset >= int64(len(r.file.data)) { return 0, io.EOF } n := copy(p, r.file.data[r.offset:]) r.offset += int64(n) return n, nil } func (r *memFileReader) Stat() (fs.FileInfo, error) { return &memFileInfo{ name: filepath.Base(r.name), size: int64(len(r.file.data)), mode: r.file.mode, modTime: r.file.modTime, }, nil } func (r *memFileReader) Close() error { return nil } 

Directory simulation requires additional consideration for how you'll represent directory entries. The fs.DirEntry interface provides a lightweight way to describe directory contents without requiring full file information upfront.

type memDirReader struct { dir *memDir name string entries []fs.DirEntry offset int } func (r *memDirReader) ReadDir(n int) ([]fs.DirEntry, error) { if r.offset >= len(r.entries) { return nil, io.EOF } end := r.offset + n if n <= 0 || end > len(r.entries) { end = len(r.entries) } entries := r.entries[r.offset:end] r.offset = end return entries, nil } 

Path Validation and Security

Path validation forms a critical security boundary in your file system implementation. Beyond the basic fs.ValidPath check, you need to consider how paths map to your internal storage and prevent access to unintended resources.

Implement path normalization early in your request processing. Convert all paths to a canonical form that your internal logic can work with consistently. This typically means cleaning paths, resolving any relative components, and ensuring consistent separator usage.

func (mfs *MemFS) normalizePath(path string) (string, error) { if !fs.ValidPath(path) { return "", fs.ErrInvalid } // Clean the path to remove redundant separators and resolve . components cleaned := filepath.Clean(path) // Convert back slashes to forward slashes for consistency cleaned = filepath.ToSlash(cleaned) // Ensure we don't have any parent directory references if strings.Contains(cleaned, "..") { return "", fs.ErrInvalid } return cleaned, nil } 

Security considerations extend beyond path validation. Consider what information your error messages reveal. A file system that exposes whether a path doesn't exist versus lacking permission to access it might leak information about the underlying storage structure. Design your error responses to provide useful feedback without compromising security.

Consider implementing access controls at the file system level rather than relying solely on the underlying storage. This approach gives you fine-grained control over what callers can access and provides a consistent security model regardless of your storage backend.

Advanced Interface Implementations

Beyond the basic fs.FS interface, Go provides several optional interfaces that can significantly improve your file system's performance and functionality. Implementing these interfaces allows your file system to integrate more deeply with the Go ecosystem and provide optimized operations for common use cases.

ReadDirFS for Directory Optimization

The fs.ReadDirFS interface provides a direct way to list directory contents without requiring callers to open a directory file first. This optimization becomes particularly valuable when your underlying storage can list directories more efficiently than the default file-based approach.

func (mfs *MemFS) ReadDir(name string) ([]fs.DirEntry, error) { cleanName, err := mfs.normalizePath(name) if err != nil { return nil, &fs.PathError{Op: "readdir", Path: name, Err: err} } mfs.mu.RLock() defer mfs.mu.RUnlock() // Handle root directory case if cleanName == "." { return mfs.listRootEntries(), nil } dir, exists := mfs.dirs[cleanName] if !exists { return nil, &fs.PathError{Op: "readdir", Path: name, Err: fs.ErrNotExist} } // Return a copy to prevent external modification entries := make([]fs.DirEntry, len(dir.entries)) copy(entries, dir.entries) return entries, nil } 

The performance benefit becomes clear when you consider network-based or database-backed file systems. Instead of opening a directory file and making multiple read calls, ReadDir allows a single operation to retrieve all directory contents. This reduces both latency and resource usage.

Implementing ReadDirFS also improves compatibility with Go's standard library functions. Functions like filepath.WalkDir automatically detect and use this interface when available, providing better performance for directory traversal operations.

StatFS for Metadata Efficiency

The fs.StatFS interface enables efficient metadata retrieval without opening files. This becomes crucial for operations that need file information but don't require file contents, such as directory listings that display file sizes or modification times.

func (mfs *MemFS) Stat(name string) (fs.FileInfo, error) { cleanName, err := mfs.normalizePath(name) if err != nil { return nil, &fs.PathError{Op: "stat", Path: name, Err: err} } mfs.mu.RLock() defer mfs.mu.RUnlock() if file, exists := mfs.files[cleanName]; exists { return &memFileInfo{ name: filepath.Base(cleanName), size: int64(len(file.data)), mode: file.mode, modTime: file.modTime, }, nil } if dir, exists := mfs.dirs[cleanName]; exists { return &memDirInfo{ name: filepath.Base(cleanName), mode: fs.ModeDir | 0755, modTime: dir.modTime, }, nil } return nil, &fs.PathError{Op: "stat", Path: name, Err: fs.ErrNotExist} } 

For remote file systems, StatFS can dramatically reduce network round trips. Instead of opening a file to get its metadata, your implementation can query the remote system directly for file information, potentially batching multiple stat operations into a single request.

The interface also enables more sophisticated caching strategies. You can cache file metadata separately from file contents, allowing stat operations to return immediately from cache while still requiring network access for file reads.

GlobFS for Pattern Matching

The fs.GlobFS interface provides efficient pattern matching capabilities that many applications require. Rather than forcing callers to traverse entire directory trees manually, this interface allows your file system to optimize pattern matching based on its internal structure.

func (mfs *MemFS) Glob(pattern string) ([]string, error) { if !fs.ValidPath(pattern) { return nil, &fs.PathError{Op: "glob", Path: pattern, Err: fs.ErrInvalid} } mfs.mu.RLock() defer mfs.mu.RUnlock() var matches []string // Check all files against the pattern for path := range mfs.files { if matched, _ := filepath.Match(pattern, path); matched { matches = append(matches, path) } } // Check all directories against the pattern for path := range mfs.dirs { if matched, _ := filepath.Match(pattern, path); matched { matches = append(matches, path) } } sort.Strings(matches) return matches, nil } 

For database-backed file systems, GlobFS enables pushing pattern matching down to the database level using SQL LIKE operations or similar constructs. This approach can significantly reduce the amount of data transferred and processed in your application.

Tree-structured storage systems can optimize glob operations by pruning search paths early. If a pattern doesn't match a directory prefix, the entire subtree can be skipped, dramatically reducing search time for complex directory hierarchies.

Advanced glob implementations might also support more sophisticated patterns than the basic shell-style matching. Regular expressions, SQL-style patterns, or custom matching logic can be implemented while still maintaining the standard interface contract.

The key to effective GlobFS implementation lies in understanding your storage system's strengths. Memory-based systems can afford to check every path, while remote systems benefit from server-side filtering. Database systems might translate patterns into efficient queries, while cached systems could maintain pattern indexes for frequently used glob operations.

Error Handling Patterns

Robust error handling distinguishes professional file system implementations from simple prototypes. The fs package establishes specific error conventions that your implementation should follow to ensure consistent behavior across the Go ecosystem.

PathError Construction

The fs.PathError type serves as the standard way to report path-specific errors in file system operations. This error type wraps the underlying error while providing context about the operation and path that failed.

type PathError struct { Op string Path string Err error } func (e *PathError) Error() string { return e.Op + " " + e.Path + ": " + e.Err.Error() } func (e *PathError) Unwrap() error { return e.Err } 

Your file system should construct PathError instances consistently across all operations. The Op field should use lowercase operation names that match the interface method being called. Common operations include "open", "stat", "readdir", and "glob".

func (mfs *MemFS) Open(name string) (fs.File, error) { if !fs.ValidPath(name) { return nil, &fs.PathError{ Op: "open", Path: name, Err: fs.ErrInvalid, } } // ... rest of implementation if _, exists := mfs.files[name]; !exists { return nil, &fs.PathError{ Op: "open", Path: name, Err: fs.ErrNotExist, } } } 

The path in PathError should be the original path provided by the caller, not your internal normalized version. This preserves the context that users need to understand and debug their code. However, be cautious about exposing sensitive information through error paths in security-sensitive applications.

Consistent Error Messages

Error message consistency helps users build reliable applications on top of your file system. The fs package defines several sentinel errors that have specific meanings, and your implementation should use these errors appropriately.

fs.ErrNotExist indicates that a requested file or directory doesn't exist. This error should be used consistently whether the path is completely invalid or points to a location that simply doesn't contain a file.

var ( ErrInvalid = errInvalid() // "invalid argument" ErrPermission = errPermission() // "permission denied" ErrExist = errExist() // "file already exists" ErrNotExist = errNotExist() // "file does not exist" ErrClosed = errClosed() // "file already closed" ) 

fs.ErrPermission signals access control violations. Your file system should return this error when operations fail due to insufficient permissions, not when they fail for other reasons like network connectivity issues.

fs.ErrInvalid covers malformed inputs, including invalid paths, null bytes in filenames, or other parameter validation failures. This error helps distinguish between "you asked for something that doesn't exist" and "you asked for something incorrectly."

func validateFileName(name string) error { if name == "" { return fs.ErrInvalid } if strings.Contains(name, "\x00") { return fs.ErrInvalid } if !utf8.ValidString(name) { return fs.ErrInvalid } return nil } 

Timeout and Permission Errors

Network-based and remote file systems encounter additional error conditions that require careful handling. Timeout errors should provide enough context for callers to distinguish between different types of failures while maintaining security boundaries.

type TimeoutError struct { Op string Path string Timeout time.Duration } func (e *TimeoutError) Error() string { return fmt.Sprintf("%s %s: operation timed out after %v", e.Op, e.Path, e.Timeout) } func (e *TimeoutError) Timeout() bool { return true } func (e *TimeoutError) Temporary() bool { return true } 

Implementing the Timeout() and Temporary() methods allows error-handling code to respond appropriately to different failure modes. Network clients can retry temporary errors but might handle permanent errors differently.

Permission errors require special consideration in multi-user or security-sensitive environments. Your error messages should provide enough information for legitimate debugging without exposing system internals that could aid attackers.

func (mfs *MemFS) checkPermission(path string, operation string) error { // Simplified permission checking if mfs.isRestricted(path) { return &fs.PathError{ Op: operation, Path: path, Err: fs.ErrPermission, } } return nil } 

Consider implementing custom error types for domain-specific failures. A database-backed file system might define connection errors, while a cached file system could report cache coherency issues. These custom errors should still wrap appropriate standard errors when possible.

type CacheError struct { Op string Path string Err error } func (e *CacheError) Error() string { return fmt.Sprintf("cache %s %s: %v", e.Op, e.Path, e.Err) } func (e *CacheError) Unwrap() error { return &fs.PathError{ Op: e.Op, Path: e.Path, Err: e.Err, } } 

Error handling also extends to resource cleanup. Ensure that your file system properly handles partial failures and doesn't leak resources when operations are interrupted. This becomes particularly important for file systems that manage external connections, temporary files, or other system resources.

Testing Custom Implementations

Testing custom file systems requires a systematic approach that validates both interface compliance and implementation-specific behavior. The Go standard library provides tools to help verify your implementation meets the expected contracts, while additional testing strategies ensure robust behavior under various conditions.

Using testing/fstest.TestFS

The testing/fstest package provides TestFS, a comprehensive test suite that validates your file system against the standard interface requirements. This test suite checks hundreds of edge cases and interface compliance issues that are easy to miss in manual testing.

func TestMemFS(t *testing.T) { // Create a test file system with known content mfs := NewMemFS() // Add test files and directories mfs.WriteFile("hello.txt", []byte("hello world"), 0644) mfs.WriteFile("dir/nested.txt", []byte("nested content"), 0644) mfs.MkdirAll("empty-dir", 0755) // Run the standard compliance tests if err := fstest.TestFS(mfs, "hello.txt", "dir/nested.txt", "dir", "empty-dir"); err != nil { t.Fatal(err) } } 

TestFS expects you to provide a list of paths that should exist in your test file system. The function then performs exhaustive testing of these paths and many invalid paths, ensuring your implementation handles edge cases correctly.

The test suite validates critical behaviors like path normalization, error handling, and interface compliance. It checks that your Open method returns appropriate errors for invalid paths, that directory listing works correctly, and that file reading behaves as expected.

func setupTestFS() fs.FS { mfs := NewMemFS() // Create a realistic directory structure testFiles := map[string]string{ "README.md": "# Test File System", "src/main.go": "package main\n\nfunc main() {}", "src/util/helper.go": "package util", "docs/api.md": "# API Documentation", "configs/app.yaml": "version: 1.0", } for path, content := range testFiles { mfs.WriteFile(path, []byte(content), 0644) } return mfs } 

Running TestFS early and frequently during development helps catch interface violations before they become deeply embedded in your implementation. The test suite is thorough but not exhaustive - it focuses on interface compliance rather than performance or implementation-specific features.

Unit Testing Strategies

Beyond interface compliance, your file system needs unit tests that verify implementation-specific behavior, performance characteristics, and error conditions. These tests should focus on the unique aspects of your file system that TestFS doesn't cover.

func TestConcurrentAccess(t *testing.T) { mfs := NewMemFS() mfs.WriteFile("test.txt", []byte("initial content"), 0644) const numGoroutines = 100 var wg sync.WaitGroup wg.Add(numGoroutines) // Test concurrent reads for i := 0; i < numGoroutines; i++ { go func() { defer wg.Done() file, err := mfs.Open("test.txt") if err != nil { t.Errorf("failed to open file: %v", err) return } defer file.Close() data, err := io.ReadAll(file) if err != nil { t.Errorf("failed to read file: %v", err) return } if string(data) != "initial content" { t.Errorf("unexpected content: %s", data) } }() } wg.Wait() } 

Performance testing becomes crucial for file systems that will handle large numbers of files or frequent operations. Benchmark tests help identify performance regressions and validate optimization efforts.

func BenchmarkFileOpen(b *testing.B) { mfs := setupLargeTestFS() // Create FS with many files b.ResetTimer() for i := 0; i < b.N; i++ { file, err := mfs.Open("src/main.go") if err != nil { b.Fatal(err) } file.Close() } } func BenchmarkDirectoryListing(b *testing.B) { mfs := setupLargeTestFS() b.ResetTimer() for i := 0; i < b.N; i++ { entries, err := mfs.ReadDir("src") if err != nil { b.Fatal(err) } _ = entries // Use the result to prevent optimization } } 

Edge Case Coverage

Comprehensive testing requires systematically exploring edge cases that real applications might encounter. These tests often reveal subtle bugs that don't appear in normal usage patterns.

func TestEdgeCases(t *testing.T) { mfs := NewMemFS() tests := []struct { name string operation func() error expectError error }{ { name: "empty path", operation: func() error { _, err := mfs.Open("") return err }, expectError: fs.ErrInvalid, }, { name: "path with null byte", operation: func() error { _, err := mfs.Open("file\x00name") return err }, expectError: fs.ErrInvalid, }, { name: "very long path", operation: func() error { longPath := strings.Repeat("a", 1000) _, err := mfs.Open(longPath) return err }, expectError: fs.ErrNotExist, }, { name: "unicode path", operation: func() error { mfs.WriteFile("测试.txt", []byte("test"), 0644) _, err := mfs.Open("测试.txt") return err }, expectError: nil, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { err := tt.operation() if !errors.Is(err, tt.expectError) { t.Errorf("expected error %v, got %v", tt.expectError, err) } }) } } 

Resource exhaustion testing helps ensure your file system behaves gracefully under stress. These tests might create thousands of files, attempt to read extremely large files, or simulate network failures for remote file systems.

func TestResourceLimits(t *testing.T) { if testing.Short() { t.Skip("skipping resource limit test in short mode") } mfs := NewMemFS() // Test creating many files const numFiles = 10000 for i := 0; i < numFiles; i++ { filename := fmt.Sprintf("file_%d.txt", i) err := mfs.WriteFile(filename, []byte("content"), 0644) if err != nil { t.Fatalf("failed to create file %d: %v", i, err) } } // Verify all files are accessible for i := 0; i < numFiles; i++ { filename := fmt.Sprintf("file_%d.txt", i) _, err := mfs.Open(filename) if err != nil { t.Errorf("failed to open file %d: %v", i, err) } } } 

Integration testing with real Go packages validates that your file system works correctly in practical scenarios. Testing with http.FileServer, template.ParseFS, or embed.FS operations ensures compatibility with common use cases.

func TestHTTPIntegration(t *testing.T) { mfs := NewMemFS() mfs.WriteFile("index.html", []byte("<html><body>Hello</body></html>"), 0644) mfs.WriteFile("style.css", []byte("body { color: blue; }"), 0644) handler := http.FileServer(http.FS(mfs)) server := httptest.NewServer(handler) defer server.Close() // Test serving files through HTTP resp, err := http.Get(server.URL + "/index.html") if err != nil { t.Fatal(err) } defer resp.Body.Close() if resp.StatusCode != http.StatusOK { t.Errorf("expected status 200, got %d", resp.StatusCode) } } 

Real-World Examples

Implementing custom file systems becomes more meaningful when you understand how they solve practical problems. These real-world examples demonstrate different approaches to common challenges, each highlighting specific design decisions and implementation techniques.

HTTP-Based File System

An HTTP-based file system provides access to remote files through standard HTTP requests. This approach enables serving files from web servers, CDNs, or REST APIs while maintaining the familiar file system interface.

type HTTPFS struct { baseURL string client *http.Client cache map[string]*httpFile mu sync.RWMutex } type httpFile struct { path string size int64 modTime time.Time content []byte fetched bool } func NewHTTPFS(baseURL string) *HTTPFS { return &HTTPFS{ baseURL: strings.TrimSuffix(baseURL, "/"), client: &http.Client{ Timeout: 30 * time.Second, }, cache: make(map[string]*httpFile), } } func (hfs *HTTPFS) Open(name string) (fs.File, error) { if !fs.ValidPath(name) { return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrInvalid} } url := hfs.baseURL + "/" + name hfs.mu.RLock() if file, exists := hfs.cache[name]; exists && file.fetched { hfs.mu.RUnlock() return &httpFileReader{file: file, name: name}, nil } hfs.mu.RUnlock() // Fetch file metadata and content resp, err := hfs.client.Get(url) if err != nil { return nil, &fs.PathError{Op: "open", Path: name, Err: err} } defer resp.Body.Close() if resp.StatusCode == http.StatusNotFound { return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrNotExist} } if resp.StatusCode != http.StatusOK { return nil, &fs.PathError{Op: "open", Path: name, Err: fmt.Errorf("HTTP %d", resp.StatusCode)} } content, err := io.ReadAll(resp.Body) if err != nil { return nil, &fs.PathError{Op: "open", Path: name, Err: err} } // Parse modification time from headers modTime := time.Now() if lastModified := resp.Header.Get("Last-Modified"); lastModified != "" { if parsed, err := http.ParseTime(lastModified); err == nil { modTime = parsed } } file := &httpFile{ path: name, size: int64(len(content)), modTime: modTime, content: content, fetched: true, } hfs.mu.Lock() hfs.cache[name] = file hfs.mu.Unlock() return &httpFileReader{file: file, name: name}, nil } 

The HTTP file system demonstrates several key patterns. Caching reduces repeated network requests, while proper error mapping translates HTTP status codes into appropriate file system errors. The implementation handles network timeouts and connection failures gracefully.

For production use, consider implementing cache expiration based on HTTP cache headers, supporting authentication mechanisms, and handling partial content requests for large files. Range requests can significantly improve performance when clients only need portions of large files.

Cached File System Wrapper

A cached file system wrapper adds caching capabilities to any underlying file system. This pattern proves particularly useful for slow storage backends like network file systems or encrypted storage.

type CachedFS struct { underlying fs.FS cache map[string]*cacheEntry mu sync.RWMutex maxSize int64 currentSize int64 } type cacheEntry struct { content []byte info fs.FileInfo timestamp time.Time size int64 } func NewCachedFS(underlying fs.FS, maxSize int64) *CachedFS { return &CachedFS{ underlying: underlying, cache: make(map[string]*cacheEntry), maxSize: maxSize, } } func (cfs *CachedFS) Open(name string) (fs.File, error) { cfs.mu.RLock() if entry, exists := cfs.cache[name]; exists { if time.Since(entry.timestamp) < 5*time.Minute { // Cache TTL cfs.mu.RUnlock() return &cachedFileReader{ content: entry.content, info: entry.info, name: name, }, nil } } cfs.mu.RUnlock() // Cache miss or expired - fetch from underlying FS file, err := cfs.underlying.Open(name) if err != nil { return nil, err } defer file.Close() info, err := file.Stat() if err != nil { return nil, err } // Only cache files, not directories if info.IsDir() { return cfs.underlying.Open(name) } content, err := io.ReadAll(file) if err != nil { return nil, err } // Cache the file if there's space cfs.mu.Lock() if cfs.currentSize+int64(len(content)) <= cfs.maxSize { cfs.evictIfNeeded(int64(len(content))) cfs.cache[name] = &cacheEntry{ content: content, info: info, timestamp: time.Now(), size: int64(len(content)), } cfs.currentSize += int64(len(content)) } cfs.mu.Unlock() return &cachedFileReader{ content: content, info: info, name: name, }, nil } func (cfs *CachedFS) evictIfNeeded(newSize int64) { for cfs.currentSize+newSize > cfs.maxSize && len(cfs.cache) > 0 { // Simple LRU: find oldest entry var oldestKey string var oldestTime time.Time = time.Now() for key, entry := range cfs.cache { if entry.timestamp.Before(oldestTime) { oldestTime = entry.timestamp oldestKey = key } } if oldestKey != "" { cfs.currentSize -= cfs.cache[oldestKey].size delete(cfs.cache, oldestKey) } } } 

The cached file system wrapper demonstrates composition patterns where one file system enhances another. Cache eviction policies, TTL management, and memory usage tracking become critical concerns for production implementations.

Union File System Implementation

A union file system combines multiple underlying file systems, presenting them as a single unified interface. This pattern enables layered file systems, overlay mounts, and fallback mechanisms.

type UnionFS struct { layers []fs.FS mu sync.RWMutex } func NewUnionFS(layers ...fs.FS) *UnionFS { return &UnionFS{ layers: layers, } } func (ufs *UnionFS) Open(name string) (fs.File, error) { if !fs.ValidPath(name) { return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrInvalid} } ufs.mu.RLock() defer ufs.mu.RUnlock() // Try each layer in order for i, layer := range ufs.layers { file, err := layer.Open(name) if err == nil { return &unionFile{ File: file, layer: i, path: name, }, nil } // Continue to next layer unless it's a fatal error if !errors.Is(err, fs.ErrNotExist) { return nil, err } } return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrNotExist} } func (ufs *UnionFS) ReadDir(name string) ([]fs.DirEntry, error) { if !fs.ValidPath(name) { return nil, &fs.PathError{Op: "readdir", Path: name, Err: fs.ErrInvalid} } ufs.mu.RLock() defer ufs.mu.RUnlock() entryMap := make(map[string]fs.DirEntry) // Merge entries from all layers for _, layer := range ufs.layers { if readDirFS, ok := layer.(fs.ReadDirFS); ok { entries, err := readDirFS.ReadDir(name) if err != nil && !errors.Is(err, fs.ErrNotExist) { continue // Skip layers with errors } for _, entry := range entries { // First layer wins for duplicate names if _, exists := entryMap[entry.Name()]; !exists { entryMap[entry.Name()] = entry } } } } // Convert map to sorted slice entries := make([]fs.DirEntry, 0, len(entryMap)) for _, entry := range entryMap { entries = append(entries, entry) } sort.Slice(entries, func(i, j int) bool { return entries[i].Name() < entries[j].Name() }) return entries, nil } 

Union file systems require careful consideration of precedence rules, directory merging strategies, and conflict resolution. The implementation above uses a simple "first layer wins" approach, but production systems might need more sophisticated policies.

These real-world examples demonstrate that custom file systems solve diverse problems through different architectural approaches. HTTP file systems bridge network protocols, cached wrappers improve performance, and union systems provide flexible composition. Each pattern can be adapted and combined to meet specific application requirements while maintaining the standard Go file system interface.