calc_metrics

rectools.metrics.scoring.calc_metrics(metrics: Dict[str, MetricAtK], reco: DataFrame, interactions: Optional[DataFrame] = None, prev_interactions: Optional[DataFrame] = None, catalog: Optional[Collection[Union[str, int]]] = None, ref_reco: Optional[Union[DataFrame, Dict[Hashable, DataFrame]]] = None) → Dict[str, float][source]

Calculate metrics.

Parameters

  • metrics (dict(str -> Metric)) – Dict of metric objects to calculate, where key is metric name and value is metric object.

  • reco (pd.DataFrame) – Recommendations table with columns Columns.User, Columns.Item, Columns.Rank.

  • interactions (pd.DataFrame, optional) – Interactions table with columns Columns.User, Columns.Item. Obligatory only for some types of metrics.

  • prev_interactions (pd.DataFrame) – Table with previous user-item interactions, with columns Columns.User, Columns.Item. Obligatory only for some types of metrics.

  • catalog (collection, optional) – Collection of unique item ids that could be used for recommendations. Obligatory only if ClassificationMetric or SerendipityMetric instances present in metrics.

  • ref_reco (Union[pd.DataFrame, Dict[Hashable, pd.DataFrame]], optional) – Reference recommendations table(s) with columns Columns.User, Columns.Item, Columns.Rank. For multiple intersection calculations we can pass multiple models recommendations in a dict: ref_reco = {"one": ref_reco_one, "two": ref_reco_two} Obligatory only if IntersectionMetric instances present in metrics.

Returns

Dictionary where keys are the same with keys in metrics and values are metric calculation results.

Return type

dict(str->float)

Raises

ValueError – If obligatory argument for some metric not set.

Examples

>>> from rectools import Columns
>>> from rectools.metrics import Accuracy, NDCG
>>> reco = pd.DataFrame(
...     {
...         Columns.User: [1, 1, 2, 2, 3, 3, 3, 3, 4, 4, 4],
...         Columns.Item: [7, 8, 1, 2, 1, 2, 3, 4, 1, 2, 3],
...         Columns.Rank: [1, 2, 1, 2, 1, 2, 3, 4, 1, 2, 3],
...     }
... )
>>> interactions = pd.DataFrame(
...     {
...         Columns.User: [1, 1, 2, 3, 3, 3, 4, 4, 4],
...         Columns.Item: [1, 2, 1, 1, 3, 4, 1, 2, 3],
...         Columns.Datetime: [1, 1, 1, 1, 1, 2, 2, 2, 2],
...     }
... )
>>> split_dt = 2
>>> df_train = interactions.loc[interactions[Columns.Datetime] < split_dt]
>>> df_test = interactions.loc[interactions[Columns.Datetime] >= split_dt]
>>> metrics = {
...     'ndcg@1': NDCG(k=1),
...     'accuracy@1': Accuracy(k=1)
... }
>>> calc_metrics(
...     metrics,
...     reco=reco,
...     interactions=df_test,
...     prev_interactions=df_train,
...     catalog=df_train[Columns.Item].unique()
... )
{'accuracy@1': 0.3333333333333333, 'ndcg@1': 0.5}