main module

class conformalopt.main.ConformalPredictor(alpha: float = 0.1, lr_type: str = 'fixed', quantile_tracker='linear', scorecaster=None, hypers={}, conformal_predictor_name='', expt_name='expt')

Bases: object

A class for implementing an online conformal predictor as in the paper Online Conformal Prediction via Online Optimization.

alpha

Target miscoverage level. If set to 0.1, for example, eventually 90% coverage will be achieved.

Type:

float

lr_type

Type of learning rate, can be fixed, decaying, or proportional. The decaying learning rate is of the form Theta(t**{-0.6}). The proportional option multiplies the learning rate by the range of scores over the last 20 scores.

Type:

str

quantile_track

Type of quantile tracker, can be scalar or linear.

Type:

str or None

scorecaster

Scorecasting method, either theta_scorecaster or ar_quantile_loss_scorecaster, or None.

Type:

str or None

hypers

Specified hyperparameters for the conformal predictor. If none are passed in, all appropriate hyperparameters will be tuned.

Type:

dict

conformal_predictor_name

Name of the conformal predictor to be used in out/ folder during evaluation. If the empty string is passed in, a name will be constructed based on the conformal predictor’s attributes.

Type:

str

expt_name

Experiment name for evaluation.

Type:

str

eval(checkpoint_name='')

Evaluates the model and outputs the results in out/expt_name/checkpoint_name. If another ConformalPredictor has already produced evaluation results in this location, the results are combined. If checkpoint_name is not provided, then it is set to the current time step.

Parameters:

checkpoint_name (str) – The checkpoint name for this evaluation.

fit(val_scores, tune_all_hparams=False, specific_hypers_to_tune: list = [], hyper_grid_overrides={})

Tunes the hyperparameters for the conformal predictor using a grid search on validation data. The hyperparameters leading to the best quantile loss, provided 1-alpha-0.01 coverage was achieved, are selected. If no selection achieves this coverage constraint, the coverage constraint is disregarded.

Parameters:

• val_scores (list) – The validation scores to use for tuning hyperparameters.

• tune_all_hparams (bool) – Whether to tune all hyperparameters or only those currently unspecified.

• specific_hypers_to_tune (list) – Specific hyperparameters to tune, if provided.

• hyper_grid_overrides (dict) – New grids to tune hyperparameters on, beyond default grid.

get_covariate(p_order=None)

Returns the covariate or feature vector based on past test scores. This is the last p_order scores along with a bias term (1 appended), and could be updated for richer feature vectors.

Parameters:

p_order (int, optional) – The number of past test scores to include. If None, defaults to self.hypers[“p_order_qt”].

Returns:

A numpy array of shape (p_order + 1,) containing the past test scores (if available) and the bias term.

Return type:

np.ndarray

get_predictions()

Retrieves the past predictions made by the conformal predictor.

Returns:

The list of past predictions.

Return type:

list

predict()

Makes a prediction by combining the quantile_tracker_predict and scorecaster_predict.

Returns:

The final prediction.

Return type:

float

reset_hypers()

Resets the hyperparameters to their original values.

set_hypers(hyper_dict)

Sets the hyperparameters for the model.

Parameters:

hyper_dict (dict) – A dictionary of hyperparameters to set.

step(prediction, realized_score)

Updates the quantile tracker based on the realized score and prediction. The update follows one step of gradient descent with the quantile loss on the quantile tracker’s parameter.

Parameters:

• prediction (float) – The predicted score.

• realized_score (float) – The actual observed score.