Most common machine learning models (linear, tree-based or neural network-based), optimize for the least squares loss when trained for regression tasks. As a result, they output a point estimate of the conditional expected value of the target: E[y|X].
In this presentation, we will explore several ways to train and evaluate probabilistic regression models as a richer alternative to point estimates. Those models predict a richer description of the full distribution of y|X and allow us to quantify the predictive uncertainty for individual predictions.
On the model training part, we will introduce the following options:
- ensemble of quantile regressors for a grid of quantile levels (using linear models or gradient boosted trees in scikit-learn, XGBoost and PyTorch),
- how to reduce probabilistic regression to multi-class classification + a cumulative sum of the
predict_probaoutput to recover a continuous conditional CDF. - how to implement this approach as a generic scikit-learn meta-estimator;
- how this approach is used to pretrain foundational tabular models (e.g. TabPFNv2).
- simple Bayesian models (e.g. Bayesian Ridge and Gaussian Processes);
- more specialized approaches as implemented in XGBoostLSS.
We will also discuss how to evaluate probabilistic predictions via:
- the pinball loss of quantile regressors,
- other strictly proper scoring rules such as Continuous Ranked Probability Score (CRPS),
- coverage measures and width of prediction intervals,
- reliability diagrams for different quantile levels.
We will illustrate of those concepts with concrete examples and running code.
Finally, we will illustrate why some applications need such calibrated probabilistic predictions:
- estimating uncertainty in trip times depending on traffic conditions to help a human decision make choose among various travel plan options.
- modeling value at risk for investment decisions,
- assessing the impact of missing variables for an ML model trained to work in degraded mode,
- Bayesian optimization for operational parameters of industrial machines from little/costly observations.
If time allows, will also discuss usage and limitations of Conformal Quantile Regressors as implemented in MAPIE and contrast aleatoric vs epistemic uncertainty captured by those models.
