Skip to content

Dynamic Indexing Feature Design

Brian Sam-Bodden edited this page Aug 9, 2025 · 1 revision

Dynamic Indexing Feature Design

Executive Summary

This document presents implementation options for dynamic indexing in Redis OM Spring, leveraging existing Spring Data Redis infrastructure and Redis OM Spring's current capabilities. All solutions guarantee 100% backward compatibility - existing applications continue working without any changes.

Problem Statement

Community Requirements by Theme

Multi-Tenancy & Security

  • Dynamic Index Names (#634, #638): Runtime index name generation for tenant isolation and ACL control
  • Custom Key Prefixes (#634): Tenant-specific prefixes for data isolation
  • ACL Integration (#634): Index naming patterns that work with Redis ACLs for fine-grained access control

DevOps & Index Lifecycle Management

  • Index Versioning (#26): Support for versioned indexes during schema evolution
  • Index Aliasing (#590, #26): Enable blue-green deployments and zero-downtime migrations
  • Migration Tools (#26): Maven/Gradle tasks for index management and migration
  • Index Maintenance (#26): Tools to export, review, and manage index definitions

Flexibility & Advanced Patterns

  • Ephemeral Indexes (#376): Temporary indexes with TTL for transient data
  • Index Decoupling (#638): Separate index definition from document definition
  • Config Reuse (#332): Use single entity class with multiple index configurations
  • CQRS Support (#590): Different indexes for read and write operations

Related issues:

  • #634 - Custom index names for ACL control
  • #638 - Decouple index and document definitions
  • #590 - Blue-green deployment support
  • #376 - Ephemeral indexes
  • #332 - Multiple configs with single class
  • #26 - Index maintenance and migrations

Current Infrastructure Analysis

Spring Data Redis Index Infrastructure

Spring Data Redis provides an index infrastructure in org.springframework.data.redis.core.index:

// Core interfaces we can leverage public interface IndexDefinition { String getKeyspace(); Collection<Condition<?>> getConditions(); IndexValueTransformer valueTransformer(); String getIndexName(); } public interface IndexDefinitionProvider { boolean hasIndexFor(Serializable keyspace); Set<IndexDefinition> getIndexDefinitionsFor(Serializable keyspace); } public interface ConfigurableIndexDefinitionProvider extends IndexDefinitionProvider, IndexDefinitionRegistry { // Allows dynamic registration of index definitions }

Redis OM Spring Current Implementation

Redis OM Spring has its own parallel indexing system:

  1. RediSearchIndexer: Manages RediSearch indexes independently
  2. IndexingOptions: Controls index creation behavior
  3. IndexCreationMode: SKIP_IF_EXIST,DROP_AND_RECREATE,SKIP_ALWAYS
  4. Minimal Integration: Uses empty IndexConfiguration() in repository setup

Implementation Options

Option 1: Bridge RediSearchIndexer with Spring Data Redis Infrastructure

Design

Create a bridge between Redis OM's RediSearchIndexer and Spring Data Redis's ConfigurableIndexDefinitionProvider:

// NEW: Bridge class implementing Spring Data Redis interface @Component public class RediSearchIndexDefinitionProvider implements ConfigurableIndexDefinitionProvider { private final RediSearchIndexer indexer; private final Map<String, Set<RediSearchIndexDefinition>> definitions; @Override public Set<IndexDefinition> getIndexDefinitionsFor(Serializable keyspace) { // Bridge to RediSearchIndexer's index definitions Class<?> entityClass = indexer.getEntityClassForKeyspace(keyspace.toString()); if (entityClass == null) return Collections.emptySet(); // Convert RediSearch schema to IndexDefinition return convertToIndexDefinitions(entityClass); } @Override public void addIndexDefinition(IndexDefinition definition) { // Allow runtime registration of new index definitions if (definition instanceof RediSearchIndexDefinition) { registerDynamicIndex((RediSearchIndexDefinition) definition); } } } // NEW: Custom IndexDefinition for RediSearch public class RediSearchIndexDefinition implements IndexDefinition { private final String indexName; private final String keyspace; private final List<SchemaField> schemaFields; private final IndexNamingStrategy namingStrategy; @Override public String getIndexName() { // Can be dynamic based on context RedisIndexContext context = RedisIndexContextHolder.getContext(); return namingStrategy != null && context != null ? namingStrategy.getIndexName(keyspace, context) : indexName; } }

Pros

  • ✅ Integrates with Spring Data Redis infrastructure
  • ✅ Allows runtime index registration
  • ✅ Maintains RediSearch-specific features
  • ✅ Zero breaking changes

Cons

  • ❌ Complexity of bridging two systems
  • ❌ Potential performance overhead

Option 2: Enhance IndexingOptions with Dynamic Resolution

Design

Extend the existing IndexingOptions annotation to support dynamic resolution:

// ENHANCE: Existing annotation with new capabilities @Target(ElementType.TYPE) @Retention(RetentionPolicy.RUNTIME) public @interface IndexingOptions { String indexName() default ""; // NEW: Support for SpEL expressions String dynamicIndexName() default ""; // NEW: Support for custom naming strategy Class<? extends IndexNamingStrategy> namingStrategy() default DefaultNamingStrategy.class; // Existing IndexCreationMode creationMode() default IndexCreationMode.SKIP_IF_EXIST; // NEW: Support for runtime index creation boolean allowRuntimeCreation() default false; } // NEW: Interface for dynamic naming public interface IndexNamingStrategy { String resolveIndexName(Class<?> entityClass, String baseIndexName); String resolveKeyPrefix(Class<?> entityClass, String basePrefix); } // ENHANCE: RediSearchIndexer to support dynamic resolution public class RediSearchIndexer { private final Map<Class<?>, IndexNamingStrategy> namingStrategies = new ConcurrentHashMap<>(); public void createIndexFor(Class<?> cl) { IndexingOptions options = cl.getAnnotation(IndexingOptions.class); // Dynamic index name resolution String indexName; if (options != null && !options.dynamicIndexName().isEmpty()) { // Evaluate SpEL expression indexName = evaluateExpression(options.dynamicIndexName(), cl); } else if (options != null && options.namingStrategy() != DefaultNamingStrategy.class) { // Use custom naming strategy IndexNamingStrategy strategy = getOrCreateStrategy(options.namingStrategy()); indexName = strategy.resolveIndexName(cl, options.indexName()); } else { // Current behavior indexName = getCurrentIndexName(cl, options); } // Rest remains the same... } }

Pros

  • ✅ Minimal changes to existing code
  • ✅ Leverages existing annotation infrastructure
  • ✅ Clear migration path
  • ✅ SpEL support for simple cases

Cons

  • ❌ Limited to compile-time configuration
  • ❌ Cannot change strategy at runtime

Option 3: Context-Aware RediSearchIndexer

Design

Enhance RediSearchIndexer to be context-aware while maintaining backward compatibility:

// NEW: Context for index resolution public class RedisIndexContext { private final Map<String, Object> attributes; private final String tenantId; private final String environment; // ThreadLocal management private static final ThreadLocal<RedisIndexContext> CONTEXT = new ThreadLocal<>(); public static void setContext(RedisIndexContext context) { CONTEXT.set(context); } public static RedisIndexContext getContext() { return CONTEXT.get(); } } // ENHANCE: RediSearchIndexer with context support @Component public class RediSearchIndexer { // NEW: Index resolver for dynamic resolution private IndexResolver indexResolver = new DefaultIndexResolver(); public String getIndexName(Class<?> entityClass) { // Check for context first RedisIndexContext context = RedisIndexContext.getContext(); if (context != null) { // Dynamic resolution based on context return indexResolver.resolveIndexName(entityClass, context); } // Fall back to current behavior return entityClassToIndexName.get(entityClass); } // NEW: Support for runtime index creation public void createIndexForContext(Class<?> entityClass, RedisIndexContext context) { String contextualIndexName = indexResolver.resolveIndexName(entityClass, context); // Check if index already exists for this context if (!indexExistsForContext(entityClass, context)) { // Create index with context-specific name and prefix createContextualIndex(entityClass, context, contextualIndexName); } } } // NEW: Interface for index resolution public interface IndexResolver { String resolveIndexName(Class<?> entityClass, RedisIndexContext context); String resolveKeyPrefix(Class<?> entityClass, RedisIndexContext context); }

Pros

  • ✅ True runtime dynamism
  • ✅ Clean separation of concerns
  • ✅ Thread-safe context handling
  • ✅ Extensible resolver pattern

Cons

  • ❌ Requires context management
  • ❌ More complex than annotation-based

Index Lifecycle Management

Index Versioning Strategy

Support versioned indexes for zero-downtime schema migrations:

@Document("users") @IndexingOptions( indexName = "users_idx", versioningStrategy = IndexVersioningStrategy.TIMESTAMP, aliasName = "users_current" // Alias always points to active version ) public class User { // When schema changes, new index created as users_idx_20240115_1420 // Alias 'users_current' switches after data migration }

Migration Workflow

@Component public class IndexMigrationService { public void migrateIndex(Class<?> entityClass, MigrationStrategy strategy) { // 1. Create new versioned index String newIndexName = createVersionedIndex(entityClass); // 2. Migrate data (configurable strategy) switch (strategy) { case DUAL_WRITE: enableDualWrite(entityClass, newIndexName); reindexInBackground(entityClass, newIndexName); break; case BLUE_GREEN: reindexToNewIndex(entityClass, newIndexName); switchAlias(entityClass, newIndexName); break; } // 3. Verify and cleanup verifyIndexIntegrity(newIndexName); scheduleOldIndexCleanup(entityClass); } }

Index Maintenance Tools

Maven Plugin

<plugin> <groupId>com.redis.om</groupId> <artifactId>redis-om-spring-maven-plugin</artifactId> <executions> <execution> <goals> <goal>export-indexes</goal> <!-- Export index definitions --> <goal>validate-indexes</goal> <!-- Validate against Redis --> <goal>migrate-indexes</goal> <!-- Run migrations --> </goals> </execution> </executions> </plugin>

Gradle Task

task exportIndexes(type: RedisIndexExportTask) { outputDir = file("$buildDir/redis-indexes") format = 'json' // or 'redis-cli' } task migrateIndexes(type: RedisIndexMigrationTask) { migrationDir = file("src/main/resources/redis/migrations") strategy = 'blue-green' }

Security & Access Control Integration

ACL-Compatible Index Naming

Support index naming patterns that work with Redis ACLs:

public class AclAwareIndexNamingStrategy implements IndexNamingStrategy { // Generate: <env>:<service>:<tenant>:<entity>:<version> // Example: prod:user-service:tenant123:users:v1 @Override public String resolveIndexName(Class<?> entityClass, RedisIndexContext context) { return String.format("%s:%s:%s:%s:v%d", context.getEnvironment(), // prod, staging, dev context.getServiceName(), // microservice name context.getTenantId(), // tenant identifier entityClass.getSimpleName().toLowerCase(), context.getSchemaVersion() ); } } // Redis ACL configuration // ACL SETUSER tenant123 ~prod:user-service:tenant123:* &prod:user-service:tenant123:* +@read +@write

Per-Tenant Index Isolation

@Configuration public class TenantIndexConfiguration { @Bean public IndexNamingStrategy tenantAwareStrategy() { return new TenantAwareIndexNamingStrategy(); } @Bean public TenantContextResolver tenantResolver() { return () -> { // Extract from JWT/OAuth2 token Authentication auth = SecurityContextHolder.getContext().getAuthentication(); return ((OAuth2Authentication) auth).getTenantId(); }; } }

Recommended Approach: Hybrid Solution

Combine the best aspects of each option:

Phase 1: Enhanced IndexingOptions (Quick Win)

@Document("users") @IndexingOptions( indexName = "users_idx", dynamicIndexName = "#{T(com.example.TenantContext).currentTenant + '_users_idx'}", creationMode = IndexCreationMode.SKIP_IF_EXIST ) public class User { // ... }

Phase 2: Context-Aware Indexer

// For programmatic control @Service public class TenantAwareService { @Autowired private RediSearchIndexer indexer; public void setupTenantIndex(String tenantId) { RedisIndexContext context = RedisIndexContext.builder() .tenantId(tenantId) .build(); // Create tenant-specific index indexer.createIndexForContext(User.class, context); } }

Phase 3: Spring Data Redis Bridge (Long-term)

Implement ConfigurableIndexDefinitionProvider to fully integrate with Spring Data Redis ecosystem.

Backward Compatibility Guarantee

Default Behavior Unchanged

// Existing code works exactly as today @Document("products") public class Product { @Id private String id; @Indexed private String name; } // No changes needed to repositories @Repository public interface ProductRepository extends RedisDocumentRepository<Product, String> { List<Product> findByName(String name); }

Opt-in Dynamic Features

// Only when explicitly configured @Document("orders") @IndexingOptions( dynamicIndexName = "#{@tenantService.currentTenant + '_orders_idx'}" ) public class Order { // Dynamic index only when SpEL expression is present }

Implementation Plan

Phase 1: Core Enhancements (2 weeks)

  • Enhance IndexingOptions with SpEL support
  • Add IndexNamingStrategy interface
  • Update RediSearchIndexer to evaluate dynamic expressions

Phase 2: Context Support (2 weeks)

  • Implement RedisIndexContext
  • Add context-aware methods to RediSearchIndexer
  • Create IndexResolver interface and implementations

Phase 3: Runtime Index Management (3 weeks)

  • Support for runtime index creation
  • Index aliasing support
  • Migration tools

Phase 4: Spring Data Redis Integration (4 weeks)

  • Implement ConfigurableIndexDefinitionProvider
  • Bridge RediSearch indexes with Spring Data Redis
  • Full integration testing

Testing Strategy

@Test public void existingCodeWorksUnchanged() { // All existing tests must pass without modification } @Test public void dynamicIndexNameResolution() { // Test SpEL evaluation in IndexingOptions } @Test public void contextAwareIndexing() { // Test context-based index resolution } @Test public void multiTenantScenario() { // Test isolation between tenants }

Comparison with Other Spring Data Projects

Feature Redis OM Spring MongoDB Elasticsearch JPA
Dynamic Collection/Index Names Via SpEL + Context Template methods SpEL in @Document N/A
Multi-tenancy Context-based Database switching Index per tenant Schema/Discriminator
Runtime Creation Supported Supported Supported Limited

Real-World Usage Examples

Example 1: Multi-Tenant SaaS Application

@Document("users") @IndexingOptions( namingStrategy = TenantAwareIndexNamingStrategy.class, creationMode = IndexCreationMode.SKIP_IF_EXIST ) public class User { @Id private String id; @Indexed private String tenantId; @Indexed private String email; @Searchable private String name; } @RestController public class UserController { @Autowired private UserRepository userRepository; @GetMapping("/users/{id}") public User getUser(@PathVariable String id, @AuthenticationPrincipal OAuth2User principal) { // Context automatically set from OAuth2 token String tenantId = principal.getAttribute("tenant_id"); try (var context = RedisIndexContext.withTenant(tenantId)) { // Repository uses tenant-specific index: tenant123_users_idx return userRepository.findById(id).orElseThrow(); } } }

Example 2: Blue-Green Deployment

@Service public class DeploymentService { @Autowired private IndexMigrationService migrationService; public void performBlueGreenDeployment() { // 1. Create new "green" index with updated schema String greenIndex = "products_idx_green_v2"; createIndexWithNewSchema(Product.class, greenIndex); // 2. Dual-write to both indexes enableDualWrite("products_idx_blue_v1", greenIndex); // 3. Backfill green index reindexInBackground(Product.class, greenIndex); // 4. Switch alias atomically switchAlias("products_current", greenIndex); // 5. Cleanup old index after verification scheduleCleanup("products_idx_blue_v1", Duration.ofHours(24)); } }

Example 3: Ephemeral Index for Batch Processing

@Component public class BatchProcessor { @Autowired private RediSearchIndexer indexer; public void processBatch(List<Order> orders) { // Create temporary index for batch String tempIndex = String.format("batch_%s_idx", UUID.randomUUID()); RedisIndexContext context = RedisIndexContext.builder() .attribute("indexName", tempIndex) .attribute("ttl", Duration.ofHours(2)) .build(); // Create ephemeral index indexer.createIndexForContext(Order.class, context); try { // Process batch with dedicated index processBatchWithIndex(orders, tempIndex); } finally { // Auto-cleanup after TTL or manual cleanup indexer.dropIndex(tempIndex); } } }

Example 4: CQRS with Separate Read/Write Indexes

@Document("events") @IndexingOptions( indexName = "events_write_idx", readIndexName = "events_read_idx", // Optimized for queries cqrsMode = true ) public class Event { @Id private String id; @Indexed private Instant timestamp; @Indexed private String type; @Searchable private String description; } @Service public class EventService { @Autowired private EventRepository repository; public void writeEvent(Event event) { // Writes go to write-optimized index repository.save(event); // Async projection to read index projectToReadIndex(event); } public List<Event> searchEvents(String query) { // Queries use read-optimized index with materialized views return repository.searchFromReadIndex(query); } }

Conclusion

This design addresses all community requirements by:

  1. Multi-tenancy & Security: Dynamic index naming with ACL support (#634, #638)
  2. DevOps & Lifecycle: Versioning, aliasing, and migration tools (#26, #590)
  3. Flexibility: Ephemeral indexes and config reuse (#376, #332)
  4. Spring Integration: Leveraging existing Spring Data Redis infrastructure

The solution provides:

  • 100% backward compatibility - existing applications work unchanged
  • Gradual adoption - features can be enabled incrementally
  • Production-ready patterns - blue-green deployments, CQRS, multi-tenancy
  • DevOps friendly - Maven/Gradle plugins for index management

All changes build upon Redis OM Spring's existing infrastructure (IndexingOptions, IndexCreationMode, RediSearchIndexer) while adding the dynamic capabilities the community needs.

Clone this wiki locally