Skip to content

model_tests.FEAT.MinMaxMetricThreshold

MinMaxMetricThreshold Objects

@dataclass
class MinMaxMetricThreshold(ModelTest)

Test if all the subgroups for a given attribute meets a certain level of expected performance.

The test also stores a dataframe showing the results of each groups. ROC curve plots For classification problem, plot ROC curves for every subgroup along with the points which maximises tpr-fpr.

Arguments:

  • attr - Column name of the protected attribute.
  • metric - Type of performance metric for the test, For classification problem, choose from 'fpr' - false positive rate, 'tpr' - true positive rate, 'fnr' - false negative rate, 'tnr' - true negative rate. For regression problem, choose from 'mse' - mean squared error, 'mae' - mean absolute error.
  • threshold - Threshold for the test. To pass, fpr/fnr/mse/mae has to be lower than the threshold. tpr/tnr has to be greater than the threshold.
  • proba_threshold - Arg for classification problem, probability threshold for the output to be classified as 1. By default, it is 0.5.
  • test_name - Name of the test, default is 'Min Max Threshold Test'.
  • test_desc - Description of the test. If none is provided, an automatic description will be generated based on the rest of the arguments passed in.

get_result_regression

def get_result_regression(df_test_with_output: pd.DataFrame) -> pd.DataFrame

Test the performance metrics of the groups of protected attribute passes specified thresholds.

Arguments:

  • df_test_with_output - Dataframe containing protected attributes with "prediction" and "truth" column.

Returns:

Dataframe with the results of each group.

get_result_classification

def get_result_classification(df_test_with_output: pd.DataFrame) -> pd.DataFrame

Test if at the current probability thresholds, for a particular attribute, the fpr/tpr of its groups passes the maximum/mininum specified metric thresholds.

Arguments:

  • df_test_with_output - Dataframe containing protected attributes with "prediction_probas" and "truth" column.

Returns:

Dataframe with the results of each group.

plot

def plot(alpha: float = 0.05, save_plots: bool = True)

For classification problem, plot ROC curve for every group in the attribute and mark the optimal probability threshold, the point which maximises tpr-fpr. For regression problem, plot the metric of interest across the attribute subgroups, and their confidence interval bands.

Arguments:

  • alpha - Significance level for confidence interval plot. Only applicable for regression problem.
  • save_plots - If True, saves the plots to the class instance.

run

def run(df_test_with_output: pd.DataFrame) -> bool

Runs test by calculating result and evaluating if it passes a defined condition.

Arguments:

  • df_test_with_output - Dataframe containing protected attributes with output and truth column. Protected attribute should not be encoded.