Non-inferiority testing with Guardrail Cutoffs
There are a number of experiments where you want to monitor that a metric is not positively impacted, but you also want to ensure there is no negative effect either.
In this case, you can now set metrics to have a Guardrail Cutoff, which means they will appear with a downside threshold. The metric with the guardrail cutoff is highlighted in yellow until Eppo is fairly certain that the downside risk on this metric is sufficiently small.
You could previously create Guardrail Metrics in Eppo, which worked by adding the metric to all experiments. While helpful to ensure some metrics are consistently tracked, it's not how most experimenters think about Guardrails.
Today we're introducing two changes:
* Metrics set to be guardrails will now have "Add this metric to all new experiment analyses" enabled in their settings. These metrics will not have any changes in how they behave, but they will no longer be labeled as guardrail metrics.
* Metrics have a new option to set a Guardrail Cutoff value. This can be edited on the metric directly and will show the associated threshold on experiment results. These metrics will be labelled as Guardrail Metrics within experiments.
Learn more on how to setup Guardrail Metrics here. Guardrail Cutoffs can also be set via API and via Certified Metrics sync as well.
