PAP

class rectools.metrics.auc.PAP(k: int, debias_config: DebiasConfig = None, insufficient_handling: str = 'ignore')[source]

Bases: _AUCMetric

Partial AUC + precision@k (pAp@k) joint classification and ranking metric. pAp@k measures AUC between the top-k irrelevant items and top-β relevant items, where β is the minimum of k and the number of relevant items. The metric behaves like prec@k when the number of relevant items are larger than k and like pAUC otherwise. IMPORTANT: this metric requires more then k recommended items for each user. It fill be enough to have k * 2 recommended items for each user. Read more in insufficient_handling parameter description.

Metric is averaged between users. For one user the formula is:

\[pAp@k = \frac{1}{k\beta}\sum_{{x_i}\in S^+}\sum_{{x_j}\in S^-}\mathbb{1}[s(x_i)\geq s(x_j)]\]
where

  • \(k\) is the number of top scored negatives and border for top scored positives

  • \(s\) is a scoring function which provides scores to rank items for user

  • \(\mathbb{1}\) is the indicator function

  • \(\beta\) is the minimum between k and number of user test positives

  • \(S^+\) is the set of top \(\beta\) positives for user acquired by \(s\)

  • \(S^-\) is the set of top \(k\) negatives for user acquired by \(s\)

  • \(x_i\) and \(x_j\) are user positives and negatives for metric computation

Introduced in “Rich-Item Recommendations for Rich-Users: Exploiting Dynamic and Static Side Information”: https://arxiv.org/abs/2001.10495, analysed in “Optimization and Analysis of the pAp@k Metric for Recommender Systems”: https://proceedings.mlr.press/v119/hiranandani20a.html

Parameters

  • k (int) – Number of top irrelevant items for user to be taken for ROC AUC computation. This does not equal k for classic @k metrics.

  • insufficient_handling ({“ignore”, “raise”, “exclude”}, default “ignore”) – Method of handling users with insufficient recommendation lists for metric calculation. pAp@k needs more then k recommendations for each user. This happens because this metris calculate ROC AUC score for specific number of user false positives and ranked test positives that is derived from provided k parameter but is not equal to it. It fill be enough to have k * 2 recommended items for each user. The following methods are available: - ignore - don’t check for insufficient recommendations lists, handle all of insufficient cases as if algorithms are not able to retrieve users unpredicted test positives on any k level. This will understate the metric value if recommendation lists are not sufficient; - exclude - exclude all users with insufficient recommendations lists from metrics computation; - raise - raise error if there are any users with insufficient recommendations lists. Use this option very carefully because some of the algorithms are unable to provide full required lists because of their inference logic. So can get errors even if you requested enough recommendations in recommend method. For example, ItemKNN generates recommendations only until the model has non-zero scores for the item in item-item similarity matrix. So with small K for neighbours in ItemKNN and big K for recommend and AUC based metric you will still get an error when insufficient_handling is set to raise.

  • debias_config (DebiasConfig, optional, default None) – Config with debias method parameters (iqr_coef, random_state).

Examples

>>> reco = pd.DataFrame(
...     {
...         Columns.User: [1, 1, 2, 2, 2, 3, 3],
...         Columns.Item: [1, 2, 3, 1, 2, 3, 2],
...         Columns.Rank: [1, 2, 1, 2, 3, 1, 2],
...     }
... )
>>> interactions = pd.DataFrame(
...     {
...         Columns.User: [1, 1, 2, 2, 3, 3],
...         Columns.Item: [1, 2, 1, 3, 1, 2],
...     }
... )
>>> PAP(k=1).calc_per_user(reco, interactions).values
array([1., 1., 0.])
>>> PAP(k=3).calc_per_user(reco, interactions).values
array([1.        , 1.        , 0.33333333])
>>> PAP(k=3, insufficient_handling="exclude").calc_per_user(reco, interactions).values
array([1., 1.])
Inherited-members

Parameters

  • k (int) –

  • debias_config (DebiasConfig) –

  • insufficient_handling (str) –

Methods

calc(reco, interactions)

Calculate metric value.

calc_from_fitted(fitted[, is_debiased])

Calculate metric value from fitted data.

calc_per_user(reco, interactions)

Calculate metric values for all users.

calc_per_user_from_fitted(fitted[, is_debiased])

Calculate metric values for all users from outer merged recommendations.

fit(reco, interactions, k_max, ...)

Prepare intermediate data for effective calculation.

Attributes

insufficient_handling

calc_per_user_from_fitted(fitted: AUCFitted, is_debiased: bool = False) → Series[source]

Calculate metric values for all users from outer merged recommendations.

Parameters

  • fitted (AUCFitted) – Meta data that got from .fit method.

  • is_debiased (bool, default False) – An indicator of whether the debias transformation has been applied before or not.

Returns

Values of metric (index - user id, values - metric value for every user).

Return type

pd.Series