Edge Case Detection

Visual analysis tools to identify and resolve problematic inputs that cause low performance. Use scatter plot visualization to find patterns and target specific problem areas for optimization.

Getting Started

📊Data Requirements

Edge Case Detection requires sufficient optimization data to generate meaningful visualizations.

Minimum: 15-20+ API calls (optimization runs)

Recommended: 30+ API calls for clear patterns

Best results: 50+ API calls with diverse inputs

🎯Access Method

Access Edge Case Detection through the Actions button on any task.

Actions → Edge Case Detection

Available after running optimizations

Updates as you add more data

Performance Scatter Plot

The scatter plot displays each API call as a point on a graph, showing performance patterns and problem areas visually.

Understanding the Visualization

X-axis: Represents one performance dimension

Y-axis: Represents another performance dimension

Each dot: Represents one input/output interaction

Colors: Indicate performance score ranges

Color-Coded Performance Levels

🔴 Red Dots (0-3)

Significant performance issues requiring immediate attention

🟠 Orange Dots (4-6)

Moderate performance issues with improvement potential

🔵 Blue Dots (7-8)

Good performance, may still benefit from fine-tuning

🟢 Green Dots (9-10)

Excellent performance, can serve as models for optimization

Common Performance Patterns

Scattered Performance

Indicates: Inconsistent performance across different inputs

Solution: Focus on building more specialized prompts

Clustered Problems

Indicates: Systematic issues with certain input types

Solution: Target specific clusters for optimization

Clear Separation

Indicates: Some inputs work well, others consistently fail

Solution: Create prompt family members for different input types

Gradual Distribution

Indicates: General improvement trend

Solution: Continue current optimization strategy

Interactive Interface Tools

Selection Tools

Click and drag: Select specific areas of the plot

Multiple selections: Choose several problem areas at once

"Clear selection" button: Reset any selected areas

Optimization Actions

"Optimize" button: Runs targeted optimization on selected inputs

Focused improvement: Addresses specific problem areas

Efficient targeting: More effective than general optimization

Systematic Edge Case Resolution

Review Overall Distribution

Get a sense of general performance patterns across all your data points.

Identify Red Clusters

Find groups of problematic inputs that consistently perform poorly.

Select Specific Areas

Choose dense problem clusters first - these have the highest impact potential.

Run Targeted Optimization

Focus improvement efforts on the selected areas using the "Optimize" button.

Reassess Results

Check if optimization improved the targeted areas and repeat for other problem clusters.

Problem Prioritization Strategy

🔴 High Priority

Dense red clusters affecting many inputs

Focus on these first for maximum impact

Create specific prompt family members

Add more manual inputs for these scenarios

🟠 Medium Priority

Orange clusters with improvement potential

Address after resolving red clusters

May benefit from targeted evaluations

Consider model optimization

🔵 Low Priority

Isolated red dots representing rare edge cases

Address if they represent important scenarios

May indicate data quality issues

Consider if worth optimizing

🟢 Monitor

Blue and green areas for performance maintenance

Maintain current performance levels

Use as examples for successful patterns

Monitor for regression

Integration with Other Tools

📊 + Input Optimization

Problem identification workflow:

Use scatter plot to identify problematic input patterns

Go to Input Optimization → End User Inputs to find specific examples

Add similar examples to Manual Inputs for systematic testing

Use these inputs for targeted prompt optimization

🎯 + Prompt Optimization

Targeted improvement workflow:

Select problem clusters in scatter plot

Run targeted optimization (creates focused events)

Review results in Prompt Optimization → Event Log

Use Manual Optimization to create specialized prompts

✅ + Evaluations

Criteria refinement workflow:

Identify consistent problem patterns in scatter plot

Create specific evaluations targeting these problem types

Use new evaluations to guide optimization of problem areas

Monitor improvement in subsequent scatter plot analysis

Best Practices

When to Use Edge Case Detection

•After initial optimization: Need baseline data to see patterns
•When performance plateaus: Identify specific areas needing attention
•Regular monitoring: Weekly or monthly review of performance patterns
•Before major changes: Understand current problem areas

Effective Selection Strategies

•Start with obvious clusters: Target dense red areas first
•Work systematically: Address one cluster at a time
•Document findings: Note what types of inputs cause problems
•Test improvements: Verify that optimization helps selected areas

Common Mistakes to Avoid

•Optimizing isolated points: Focus on patterns, not individual failures
•Ignoring successful areas: Learn from what works well
•Over-optimization: Don't endlessly optimize "good enough" areas
•Premature analysis: Wait for sufficient data before drawing conclusions

Next Steps

→ Input Optimization

Use identified problem patterns to improve your test data

→ Prompt Optimization

Create targeted prompt family members for problem clusters

→ Evaluations

Develop specific criteria for systematic problem resolution

Get Started View Documentation