Minimum Detectable Effect (MDE)

Master this essential documentation concept

Quick Definition

The smallest change in a metric that an experiment is designed to reliably detect with statistical significance.

How Minimum Detectable Effect (MDE) Works

graph TD A[Documentation Change Proposed] --> B[Define Current Baseline Metric] B --> C[Determine Practical Significance] C --> D[Set Minimum Detectable Effect] D --> E[Calculate Required Sample Size] E --> F[Design Experiment] F --> G[Run A/B Test] G --> H{Observed Effect ≥ MDE?} H -->|Yes| I[Implement Change] H -->|No| J[Maintain Current Version] I --> K[Monitor Performance] J --> L[Test Alternative Approach]

Understanding Minimum Detectable Effect (MDE)

Minimum Detectable Effect (MDE) represents the threshold of change that matters for your documentation goals. Rather than detecting any statistically significant difference, MDE focuses on changes large enough to justify implementation decisions and resource allocation.

Key Features

  • Defines the smallest improvement worth measuring in documentation metrics
  • Combines statistical power with practical significance
  • Requires advance planning before running experiments
  • Depends on baseline metrics, sample size, and statistical power
  • Varies across different documentation KPIs (time-to-completion, error rates, satisfaction)

Benefits for Documentation Teams

  • Prevents wasting resources on detecting trivial improvements
  • Enables proper sample size calculation for experiments
  • Clarifies success criteria before launching documentation changes
  • Improves decision-making by focusing on meaningful metrics
  • Reduces time and costs associated with underpowered studies

Common Misconceptions

  • MDE is not the same as statistical significance—it's about practical importance
  • Smaller MDEs don't always mean better experiments; they require larger sample sizes
  • MDE must be determined before the experiment, not after analyzing results
  • MDEs vary by metric and cannot be universally applied across all documentation initiatives

Real-World Documentation Use Cases

Optimizing Tutorial Completion Rates

Problem

Documentation team wants to improve tutorial completion rates but needs to know what improvement justifies redesigning the tutorial structure

Solution

Set an MDE of 5% improvement in completion rate. If baseline is 60%, the team only implements changes if new version achieves 65% or higher

Implementation

['Establish baseline completion rate from historical data (e.g., 60%)', 'Determine practical significance: 5% improvement represents meaningful business impact', 'Calculate sample size needed to detect 5% improvement with 80% statistical power', 'Split users between current and redesigned tutorial versions', 'Run experiment until reaching required sample size', 'Analyze results against the 5% MDE threshold']

Expected Outcome

Team invests redesign effort only if improvements exceed 5%, avoiding small changes that don't justify the work

Reducing Documentation Search Time

Problem

Documentation team wants to improve search functionality but needs to quantify what time reduction is worth implementing

Solution

Set MDE of 15-second reduction in average search-to-answer time. Only implement if new search algorithm achieves this threshold

Implementation

['Measure current average time from search query to finding answer (e.g., 45 seconds)', 'Define MDE as 15-second reduction (30-second target)', 'Calculate required sample size of search sessions', 'Deploy new search algorithm to percentage of users', 'Track search-to-answer time for both versions', 'Compare results against 15-second MDE threshold']

Expected Outcome

Resources directed toward search improvements only when they deliver meaningful time savings to users

Measuring Documentation Satisfaction Impact

Problem

Team redesigns documentation structure but needs clear criteria for determining if changes improved user satisfaction

Solution

Set MDE of 10-point improvement on 100-point satisfaction scale. Implementation only if new structure achieves this increase

Implementation

['Establish baseline satisfaction score through surveys (e.g., 65/100)', 'Determine 10-point improvement as practically significant', 'Calculate sample size for satisfaction survey with 80% power', 'Implement new documentation structure for test group', 'Collect satisfaction ratings from both groups', 'Compare average satisfaction scores against 10-point MDE']

Expected Outcome

Documentation redesigns are validated only when they produce substantial satisfaction improvements

Evaluating Documentation Video Effectiveness

Problem

Team produces video tutorials but needs to determine what improvement in engagement metrics justifies video production costs

Solution

Set MDE of 20% increase in video completion rate. Only produce videos if they achieve this engagement threshold

Implementation

['Measure baseline completion rate for text-based tutorials (e.g., 55%)', 'Define MDE as 20% relative increase (66% completion target)', 'Calculate required video and text viewers for statistical power', 'Deploy videos to test group while control group uses text', 'Track completion rates for both formats', 'Analyze whether video completion meets 20% MDE threshold']

Expected Outcome

Video production investments justified only when they demonstrate significant engagement improvements over text

Best Practices

✓ Define MDE Before Running Experiments

Establish your Minimum Detectable Effect during the planning phase, not after collecting data. This prevents p-hacking and ensures your experiment is properly powered to detect meaningful changes.

✓ Do: Collaborate with stakeholders to determine what improvement magnitude justifies implementation effort and costs before launching experiments
✗ Don't: Avoid setting MDE thresholds after seeing preliminary results or adjusting them based on observed effects

✓ Align MDE with Business Impact

Your MDE should reflect genuine business or user value, not just statistical significance. A 0.1% improvement might be statistically significant but practically meaningless for documentation changes.

✓ Do: Consult with product managers and user research teams to ensure MDE thresholds align with real user needs and business goals
✗ Don't: Don't set arbitrary MDE values without considering implementation costs and expected user benefits

✓ Calculate Appropriate Sample Sizes

MDE directly determines how many observations you need. Smaller MDEs require exponentially larger sample sizes. Understanding this relationship prevents underpowered experiments that can't detect real improvements.

✓ Do: Use statistical power calculators to determine required sample sizes for your chosen MDE, baseline metric, and desired power level (typically 80%)
✗ Don't: Don't assume you can detect small effect sizes with limited sample sizes

✓ Document MDE Rationale Clearly

Record why you chose specific MDE values for each metric. This documentation helps future teams understand decision criteria and enables consistent application across similar experiments.

✓ Do: Create experiment documentation that explains baseline metrics, chosen MDE, statistical power, and business justification for each threshold
✗ Don't: Don't leave MDE decisions undocumented or assume team members will remember the reasoning

✓ Use Metric-Specific MDEs

Different documentation metrics require different MDE thresholds. A 5% improvement in completion rate may be meaningful while a 5% improvement in page views might not be. Tailor MDEs to each specific metric.

✓ Do: Establish separate MDE thresholds for completion rates, time-on-page, search effectiveness, user satisfaction, and error reduction based on their distinct business impacts
✗ Don't: Don't apply uniform MDE percentages across all documentation metrics

How Docsie Helps with Minimum Detectable Effect (MDE)

Book a Demo

Build Better Documentation with Docsie

Join thousands of teams creating outstanding documentation

Book a Demo