Auto-Apply Agent Hyperparameter Recommendations and Expand Coverage

User Story

As a machine learning developer using AI-assisted model building, I want the agent's hyperparameter recommendations to be automatically applied to my model when possible, so that I can benefit from AI-guided optimization without manually configuring every parameter, while still maintaining full control over the configuration process.

Problem Statement

Current State:

The AI agent system (GetAgentRecommendationsAsync) analyzes datasets and provides intelligent recommendations including:

• Model type selection (SuggestedModelType)
• Hyperparameter values (SuggestedHyperparameters dictionary)
• Model selection reasoning (ModelSelectionReasoning)

However, these recommendations are underutilized:

// src/PredictionModelBuilder.cs - Current ApplyAgentRecommendations private void ApplyAgentRecommendations(AgentRecommendation<T, TInput, TOutput> recommendation) { // Applies model type recommendation (with console guidance only) if (_model == null && recommendation.SuggestedModelType.HasValue) { Console.WriteLine($"The AI agent recommends using: {recommendation.SuggestedModelType.Value}"); Console.WriteLine($"To use this recommendation, configure your builder:"); Console.WriteLine($" builder.UseModel(/* create {recommendation.SuggestedModelType.Value} instance */);"); } // Note: Hyperparameter recommendations are currently stored in recommendation.SuggestedHyperparameters // but not auto-applied. Future enhancement: Apply hyperparameters to compatible models. }

Problems with Current Approach:

1. Manual Work Required: Users must manually extract and apply hyperparameter values
2. Lost Optimization Potential: AI recommendations are ignored unless user reads console output
3. Inconsistent Coverage: Not all model types have hyperparameter recommendation support
4. No Reflection/Introspection: Can't automatically set properties on models
5. Complex Model Factory: 80+ model types with different constructor signatures makes auto-creation challenging
6. Silent Failures: No feedback when hyperparameters can't be applied

Missing Features:

• No automatic hyperparameter application to existing models
• No hyperparameter introspection API (IConfigurableModel or similar)
• No validation that recommended hyperparameters are compatible with model
• No reporting of which hyperparameters were/weren't applied
• Limited coverage across model types (not all models support agent-recommended hyperparameters)

User Impact Examples:

// Current workflow - manual application required var result = await builder .ConfigureAgentAssistance(options => options.EnableModelSelection()) .BuildAsync(data, labels); // User sees: "The AI agent recommends using: RandomForest" // User sees: "Recommendation details available in result.AgentRecommendation" // User must then manually do: var hyperparameters = result.AgentRecommendation.SuggestedHyperparameters; // How do I apply these? Which properties? What if model doesn't support them?

Proposed Solution

Create a comprehensive hyperparameter application system that:

1. Auto-applies hyperparameters when a model is already configured via UseModel()
2. Validates compatibility between recommendations and model capabilities
3. Reports what was applied successfully and what wasn't
4. Expands coverage to all ~80+ model types in the library
5. Maintains user control through opt-in configuration

Design Philosophy

1. Incremental Enhancement: Start with property-based approach, expand to interfaces later
2. Explicit Opt-In: Hyperparameter auto-application requires explicit configuration
3. Safe Defaults: Never override user-specified values, only fill in missing ones
4. Clear Reporting: User always knows what the agent did and why
5. Extensibility: Easy to add support for new hyperparameters and models

Architecture

Phase 1: Property-Based Hyperparameter Application (Foundation)

Goal: Apply agent hyperparameter recommendations using reflection on existing models.

AC 1.1: Create HyperparameterApplicator Service (8 points)

Requirements:

• Create src/Agents/HyperparameterApplicator.cs
• Define public class HyperparameterApplicator<T>
• Core method:

  public HyperparameterApplicationResult ApplyHyperparameters( IFullModel<T> model, Dictionary<string, object> hyperparameters, bool overrideExistingValues = false)

• Use reflection to discover writable properties on model
• Map hyperparameter names (case-insensitive) to property names
• Support common hyperparameter types: int, double, bool, enum
• Validate type compatibility before setting
• Track success/failure for each hyperparameter

Implementation Details:

public class HyperparameterApplicator<T> { public HyperparameterApplicationResult ApplyHyperparameters( IFullModel<T> model, Dictionary<string, object> hyperparameters, bool overrideExistingValues = false) { var result = new HyperparameterApplicationResult(); foreach (var (key, value) in hyperparameters) { try { // Find matching property (case-insensitive) var property = model.GetType() .GetProperties(BindingFlags.Public | BindingFlags.Instance) .FirstOrDefault(p => p.Name.Equals(key, StringComparison.OrdinalIgnoreCase) && p.CanWrite); if (property == null) { result.AddFailure(key, "Property not found or not writable"); continue; } // Skip if value already set and override not requested var currentValue = property.GetValue(model); if (currentValue != null && !IsDefaultValue(currentValue) && !overrideExistingValues) { result.AddSkipped(key, "Property already has non-default value"); continue; } // Convert and validate type object convertedValue = ConvertValue(value, property.PropertyType); // Set the property property.SetValue(model, convertedValue); result.AddSuccess(key, convertedValue); } catch (Exception ex) { result.AddFailure(key, ex.Message); } } return result; } private object ConvertValue(object value, Type targetType) { // Handle enums if (targetType.IsEnum && value is string enumString) { return Enum.Parse(targetType, enumString, ignoreCase: true); } // Handle nullable types var underlyingType = Nullable.GetUnderlyingType(targetType); if (underlyingType != null) { targetType = underlyingType; } // Standard conversion return Convert.ChangeType(value, targetType); } private bool IsDefaultValue(object value) { if (value == null) return true; var type = value.GetType(); if (type.IsValueType) { return value.Equals(Activator.CreateInstance(type)); } return false; } } public class HyperparameterApplicationResult { public List<(string Name, object Value)> Applied { get; } = new(); public List<(string Name, string Reason)> Skipped { get; } = new(); public List<(string Name, string Reason)> Failed { get; } = new(); public int SuccessCount => Applied.Count; public int TotalCount => Applied.Count + Skipped.Count + Failed.Count; public bool HasFailures => Failed.Count > 0; public void AddSuccess(string name, object value) => Applied.Add((name, value)); public void AddSkipped(string name, string reason) => Skipped.Add((name, reason)); public void AddFailure(string name, string reason) => Failed.Add((name, reason)); public string GetSummary() { var sb = new StringBuilder(); sb.AppendLine($"Hyperparameter Application Summary:"); sb.AppendLine($" Applied: {Applied.Count}"); sb.AppendLine($" Skipped: {Skipped.Count}"); sb.AppendLine($" Failed: {Failed.Count}"); if (Applied.Count > 0) { sb.AppendLine(" Successfully applied:"); foreach (var (name, value) in Applied) { sb.AppendLine($" - {name} = {value}"); } } if (Failed.Count > 0) { sb.AppendLine(" Failed to apply:"); foreach (var (name, reason) in Failed) { sb.AppendLine($" - {name}: {reason}"); } } return sb.ToString(); } }

Validation:

• Successfully sets int, double, bool properties
• Successfully sets enum properties from string values
• Handles nullable types correctly
• Respects overrideExistingValues flag
• Provides detailed success/failure reporting
• Case-insensitive property matching works
• Skips read-only properties gracefully

AC 1.2: Integrate HyperparameterApplicator into ApplyAgentRecommendations (5 points)

Requirements:

• Update src/PredictionModelBuilder.cs ApplyAgentRecommendations method
• Apply hyperparameters when _model is already set via UseModel()
• Output detailed application results to console
• Respect new configuration option: AutoApplyHyperparameters (default: true)

Updated ApplyAgentRecommendations:

private void ApplyAgentRecommendations(AgentRecommendation<T, TInput, TOutput> recommendation) { // Apply model type recommendation (existing logic) if (_model == null && recommendation.SuggestedModelType.HasValue) { Console.WriteLine($"\n=== AGENT RECOMMENDATION ==="); Console.WriteLine($"The AI agent recommends using: {recommendation.SuggestedModelType.Value}"); // ... existing console output } // NEW: Apply hyperparameter recommendations if (_model != null && recommendation.SuggestedHyperparameters != null && recommendation.SuggestedHyperparameters.Count > 0 && _agentConfiguration?.AutoApplyHyperparameters == true) { Console.WriteLine($"\n=== APPLYING AGENT HYPERPARAMETER RECOMMENDATIONS ==="); var applicator = new HyperparameterApplicator<T>(); var result = applicator.ApplyHyperparameters( _model, recommendation.SuggestedHyperparameters, overrideExistingValues: false); Console.WriteLine(result.GetSummary()); if (result.HasFailures) { Console.WriteLine("\nSome hyperparameters could not be applied. " + "This may be expected if your model type doesn't support them."); } Console.WriteLine("====================================================\n"); } }

Validation:

• Hyperparameters applied when model is set via UseModel()
• No application when AutoApplyHyperparameters = false
• Console output shows exactly what was applied
• User can see which hyperparameters failed and why

AC 1.3: Add AutoApplyHyperparameters to AgentConfiguration (2 points)

Requirements:

• Add public bool AutoApplyHyperparameters { get; set; } = true; to AgentConfiguration
• Add fluent configuration method to AgentConfigurationBuilder:

  public AgentConfigurationBuilder EnableHyperparameterApplication(bool enable = true) { _autoApplyHyperparameters = enable; return this; }

Usage Example:

var result = await builder .UseModel(new RandomForest<double>()) .ConfigureAgentAssistance(options => options .EnableModelSelection() .EnableHyperparameterApplication()) // NEW .BuildAsync(data, labels); // Agent will now auto-apply hyperparameters to the RandomForest model

Phase 2: Expand Hyperparameter Coverage (High Impact)

Goal: Ensure all major model types have comprehensive hyperparameter recommendation support.

AC 2.1: Audit Current Hyperparameter Coverage (3 points)

Requirements:

• Review GetAgentRecommendationsAsync to identify which hyperparameters are currently recommended
• Audit all ~80 model types to identify:
  • Which have configurable hyperparameters
  • Which hyperparameters are most impactful for each model family
  • Which hyperparameters are currently missing from agent recommendations
• Create coverage matrix documenting:
  • Model type
  • Available hyperparameters
  • Currently recommended by agent (yes/no)
  • Priority for recommendation support

Deliverable: HYPERPARAMETER_COVERAGE_MATRIX.md documenting current state

AC 2.2: Enhance Agent Prompt with Comprehensive Hyperparameter Knowledge (8 points)

Requirements:

• Update agent prompt in GetAgentRecommendationsAsync to include:
  • Neural network hyperparameters: hidden layers, activation functions, learning rate schedules
  • Tree-based hyperparameters: tree depth, min samples, max features, split criteria
  • SVM hyperparameters: kernel type, C, gamma, degree
  • Optimizer hyperparameters: batch size, epochs, early stopping criteria
  • Regularization hyperparameters: L1/L2 penalties, dropout rates
• Provide agent with model-specific hyperparameter ranges and defaults
• Include hyperparameter interaction guidance (e.g., "deeper trees need more min_samples")

Enhanced Prompt Structure:

var hyperparameterKnowledge = @" # Hyperparameter Recommendations by Model Type  ## Neural Networks (MLPRegressor, MLPClassifier) - HiddenLayers: Array of layer sizes (e.g., [100, 50] for 2 hidden layers) - ActivationFunction: ""ReLU"", ""Sigmoid"", ""Tanh"" - LearningRate: 0.001-0.01 (smaller for large datasets) - MaxEpochs: 100-1000 (use early stopping) - BatchSize: 32-128 (power of 2, based on dataset size)  ## Random Forest (RandomForestClassifier, RandomForestRegressor) - NumTrees: 100-500 (more trees = better but slower) - MaxDepth: null (unlimited) or 10-30 (prevent overfitting) - MinSamplesSplit: 2-10 (higher for noisy data) - MaxFeatures: ""sqrt"" (classification) or ""1/3"" (regression)  ## Support Vector Machines - KernelType: ""RBF"", ""Linear"", ""Polynomial"" - C: 0.1-100 (regularization strength) - Gamma: ""scale"" or 0.001-1.0 - Degree: 2-5 (only for polynomial kernel)  ## Gradient Boosting - NumEstimators: 100-1000 - LearningRate: 0.01-0.3 (inversely related to num_estimators) - MaxDepth: 3-8 (shallow trees for boosting) - Subsample: 0.5-1.0 (prevent overfitting)  [Include all 80+ model types with specific recommendations] "; var prompt = $@" Analyze this dataset and recommend hyperparameters.  {datasetAnalysis}  {hyperparameterKnowledge}  Return JSON with SuggestedHyperparameters as Dictionary<string, object>. ";

Validation:

• Agent provides hyperparameters for at least 20 most common model types
• Hyperparameters are contextually appropriate (e.g., deep trees for small datasets)
• Agent explains reasoning for each hyperparameter choice

AC 2.3: Add Model-Specific Hyperparameter Validation (5 points)

Requirements:

• Enhance HyperparameterApplicator to validate hyperparameter values
• Check ranges (e.g., learning rate must be > 0)
• Check dependencies (e.g., polynomial kernel requires degree parameter)
• Provide helpful error messages for invalid values

Validation Logic:

private void ValidateHyperparameter(string name, object value, Type propertyType) { // Example validations if (name.Equals("LearningRate", StringComparison.OrdinalIgnoreCase)) { var lr = Convert.ToDouble(value); if (lr <= 0 || lr > 1.0) throw new ArgumentOutOfRangeException( nameof(value), $"LearningRate must be in range (0, 1.0], got {lr}"); } if (name.Equals("MaxDepth", StringComparison.OrdinalIgnoreCase) && value != null) { var depth = Convert.ToInt32(value); if (depth < 1) throw new ArgumentOutOfRangeException( nameof(value), $"MaxDepth must be >= 1, got {depth}"); } // Add more validations for common hyperparameters }

Phase 3: IConfigurableModel Interface (Optional Future Enhancement)

Goal: Provide a formal interface for models to expose their hyperparameters.

AC 3.1: Design IConfigurableModel Interface (3 points)

Requirements:

• Define interface in src/Interfaces/IConfigurableModel.cs
• Interface signature:

  public interface IConfigurableModel<T> { /// <summary> /// Gets metadata about configurable hyperparameters /// </summary> IEnumerable<HyperparameterMetadata> GetHyperparameters(); /// <summary> /// Sets a hyperparameter value /// </summary> void SetHyperparameter(string name, object value); /// <summary> /// Gets current value of a hyperparameter /// </summary> object GetHyperparameter(string name); } public class HyperparameterMetadata { public string Name { get; set; } = string.Empty; public Type Type { get; set; } = typeof(object); public string Description { get; set; } = string.Empty; public object? MinValue { get; set; } public object? MaxValue { get; set; } public object? DefaultValue { get; set; } public string[]? AllowedValues { get; set; } }

Note: This is a future enhancement to enable:

• Discovery of available hyperparameters
• Validation constraints
• Better error messages
• Auto-generated UI for hyperparameter tuning

Not included in current scope - would require refactoring 80+ model classes.

Testing Requirements

Unit Tests (src/Tests/UnitTests/Agents/HyperparameterApplicatorTests.cs)

• Test_ApplyHyperparameters_SetsIntProperty: Verify int property is set correctly
• Test_ApplyHyperparameters_SetsDoubleProperty: Verify double property is set correctly
• Test_ApplyHyperparameters_SetsBoolProperty: Verify bool property is set correctly
• Test_ApplyHyperparameters_SetsEnumProperty: Verify enum property set from string
• Test_ApplyHyperparameters_CaseInsensitiveMatching: Property names matched case-insensitively
• Test_ApplyHyperparameters_RespectsOverrideFlag: Existing values not overridden when flag is false
• Test_ApplyHyperparameters_ReadOnlyPropertySkipped: Read-only properties gracefully skipped
• Test_ApplyHyperparameters_InvalidTypeReported: Type mismatches reported as failures
• Test_ApplyHyperparameters_NonExistentPropertyReported: Unknown properties reported as failures
• Test_ApplyHyperparameters_ResultSummary: Result object contains accurate counts and details

Integration Tests (src/Tests/IntegrationTests/Agents/)

• Test_AgentRecommendations_AppliedToRandomForest: Hyperparameters applied to RandomForest model
• Test_AgentRecommendations_AppliedToNeuralNetwork: Hyperparameters applied to MLP model
• Test_AgentRecommendations_NoApplicationWhenDisabled: Respects AutoApplyHyperparameters=false
• Test_AgentRecommendations_PartialApplicationReported: Some succeed, some fail, all reported correctly

Coverage Tests

• Test_HyperparameterCoverage_Top20Models: Verify agent recommends hyperparameters for 20 most common models
• Test_HyperparameterCoverage_ValidValues: Recommended values are within valid ranges

Migration Guide

For Library Users

Before (manual hyperparameter application):

var result = await builder .ConfigureAgentAssistance(options => options.EnableModelSelection()) .BuildAsync(data, labels); // Manual extraction and application var recommended = result.AgentRecommendation.SuggestedHyperparameters; // Now what? How do I apply these to my model?

After (automatic hyperparameter application):

var result = await builder .UseModel(new RandomForest<double>()) .ConfigureAgentAssistance(options => options .EnableModelSelection() .EnableHyperparameterApplication()) // Auto-apply hyperparameters .BuildAsync(data, labels); // Hyperparameters automatically applied to RandomForest // Console output shows exactly what was applied

Opt-out (if you want manual control):

var result = await builder .UseModel(new RandomForest<double>()) .ConfigureAgentAssistance(options => options .EnableModelSelection() .EnableHyperparameterApplication(false)) // Disable auto-application .BuildAsync(data, labels); // Manually inspect and apply var recommended = result.AgentRecommendation.SuggestedHyperparameters;

Definition of Done

• HyperparameterApplicator<T> class implemented with reflection-based property setting
• HyperparameterApplicationResult class provides detailed reporting
• ApplyAgentRecommendations integrates hyperparameter application
• AgentConfiguration includes AutoApplyHyperparameters option (default: true)
• Agent prompt enhanced with comprehensive hyperparameter knowledge for 20+ model types
• Hyperparameter validation implemented for common parameters
• HYPERPARAMETER_COVERAGE_MATRIX.md documents current coverage
• All unit tests pass with >= 90% code coverage for new components
• Integration tests confirm end-to-end hyperparameter application
• XML documentation complete with beginner-friendly examples
• Console output provides clear feedback on what was applied
• User can opt-in or opt-out of auto-application
• No breaking changes to existing API
• Code review approved

Success Metrics

Code Quality:

• Reduce manual hyperparameter configuration burden
• Achieve >= 90% test coverage for HyperparameterApplicator
• Zero breaking changes to existing model builders

Developer Experience:

• Users benefit from AI-recommended hyperparameters automatically
• Clear visibility into what hyperparameters were applied
• Easy opt-out for users who want manual control

Agent Effectiveness:

• Agent provides hyperparameter recommendations for 20+ model types
• Recommended values are contextually appropriate for dataset characteristics
• 80%+ of recommended hyperparameters successfully applied to compatible models

Related Issues

• PR Work on Issue Number Two #423 - Code review comments that identified ApplyAgentRecommendations limitation
• Future: IConfigurableModel interface for formal hyperparameter introspection

Labels

• enhancement - Improves existing functionality
• agent-assistance - AI agent system improvements
• high-priority - Critical for agent system value proposition

Estimated Effort: 31 story points across all phases

• Phase 1 (Foundation): 15 points (HyperparameterApplicator + integration)
• Phase 2 (Coverage): 16 points (audit + prompt enhancement + validation)
• Phase 3 (Future): 3 points (IConfigurableModel design only, not implementation)

Priority: High - Unlocks full value of agent recommendation system

Status: Ready for Implementation