ranking metric for a specific use case

by lezebulon   Last Updated October 22, 2018 18:19 PM

I am trying to find the best "ranking metric" for my problem but could not find any existing suitable metric, can someone help.

I'm interested in information retrieval.

For item there is a "revelancy" between 0 and 1 that acts as a label. My algorithm gives a score to each item, and returns the 5 items with the highest score. I want to know how "good" my algorithm is to retrieve the real top 5 events, which are the 5 items with highest relevancy scores. Here are a few properties I'm interested in for my evaluation metric:

  • for items not selected (not in top 5), I don't care about their score
  • for items selected in top 5, I don't care about their order (ie returning A,B,C,D,E is the same as returning A,B,C,E,D)
  • I want that returning higher-ranked items is better than returning lower-ranking items. Ie the best result is to return the 5 items with highest relevancy score, and lowest result is to return the 5 items with lowest relevancy score.
  • I want to take into account that items with "close enough" relevancy score change the metric by not much. For instance, if I include in my result the item ranked 6th in relevancy instead of 5th in relevancy, I want this impact to be big is the relevancy score of 6 is way lower to 5, but almost nothing if the relevancy score of 6 and 5 are close.

Are their "famous" metrics that do this? I couldn't find any.

A criteria I was thinking about is to define my metric as : sum of relevancy of the 5 selected items / sum of relevancy of the 5 most relevant items.

This is somewhat doing what I want but it feels like I'm re-inventing an existing metric



Related Questions




Proper way to use NDCG@k score for recommendations

Updated April 19, 2018 23:19 PM