PartialAUC
- class rectools.metrics.auc.PartialAUC(k: int, insufficient_handling: str = 'ignore')[source]
Bases:
_AUCMetricPartial AUC at k (pAUC@k). pAUC@k measures ROC AUC score for ranking of the top-k irrelevant items and all relevant items for each user. IMPORTANT: this metric requires more then k recommended items for each user. It fill be enough to have \(n^+\) (number of user positives) + k recommended items for each user. Read more in insufficient_handling parameter description.
Metric is averaged between users. For one user the formula is:
\[pAUC@k = \frac{1}{kn_+}\sum_{{x_i}\in S^+}\sum_{{x_j}\in S^-}\mathbb{1}[s(x_i)\geq s(x_j)]\]- where
\(k\) is the number of user top scored negatives for metric computation
\(s\) is a scoring function which provides scores to rank items for user
\(\mathbb{1}\) is the indicator function
\(n_+\) is the number of all user test positives
\(S^+\) is the set of all positives for user
\(S^-\) is the set of top \(k\) negatives for user acquired by \(s\)
\(x_i\) and \(x_j\) are user positives and negatives for metric computation
Analysed in “Rich-Item Recommendations for Rich-Users: Exploiting Dynamic and Static Side Information”: https://arxiv.org/abs/2001.10495, analysed in “Optimization and Analysis of the pAp@k Metric for Recommender Systems”: https://proceedings.mlr.press/v119/hiranandani20a.html
- Parameters
k (int) – Number of top irrelevant items for user to be taken for ROC AUC computation. This does not equal k for classic @k metrics.
insufficient_handling ({“ignore”, “raise”, “exclude”}, default “ignore”) – Method of handling users with insufficient recommendation lists for metric calculation. pAUC@k needs more then k recommendations for each user. This happens because this metris calculate ROC AUC score for specific number of user false positives and ranked test positives that is derived from provided k parameter but is not equal to it. It fill be enough to have \(n^+\) (number of user positives) + k recommended items for each user. The following methods are available: - ignore - don’t check for insufficient recommendations lists, handle all of insufficient cases as if algorithms are not able to retrieve users unpredicted test positives on any k level. This will understate the metric value if recommendation lists are not sufficient; - exclude - exclude all users with insufficient recommendations lists from metrics computation; - raise - raise error if there are any users with insufficient recommendations lists. Use this option very carefully because some of the algorithms are unable to provide full required lists because of their inference logic. So can get errors even if you requested enough recommendations in recommend method. For example, ItemKNN generates recommendations only until the model has non-zero scores for the item in item-item similarity matrix. So with small K for neighbours in ItemKNN and big K for recommend and AUC based metric you will still get an error when insufficient_handling is set to raise.
Examples
>>> reco = pd.DataFrame( ... { ... Columns.User: [1, 1, 2, 2, 2, 3, 3], ... Columns.Item: [1, 2, 3, 1, 2, 3, 2], ... Columns.Rank: [1, 2, 1, 2, 3, 1, 2], ... } ... ) >>> interactions = pd.DataFrame( ... { ... Columns.User: [1, 1, 2, 2, 3, 3], ... Columns.Item: [1, 2, 1, 3, 1, 2], ... } ... ) >>> PartialAUC(k=1).calc_per_user(reco, interactions).values array([1., 1., 0.]) >>> PartialAUC(k=3).calc_per_user(reco, interactions).values array([1. , 1. , 0.33333333]) >>> PartialAUC(k=3, insufficient_handling="exclude").calc_per_user(reco, interactions).values array([1., 1.])
- Inherited-members
- Parameters
k (int) –
insufficient_handling (str) –
Methods
calc(reco, interactions)Calculate metric value.
calc_from_fitted(fitted)Calculate metric value from fitted data.
calc_per_user(reco, interactions)Calculate metric values for all users.
calc_per_user_from_fitted(fitted)Calculate metric values for all users from from fitted data.
fit(reco, interactions, k_max, ...)Prepare intermediate data for effective calculation.
Attributes
insufficient_handling