I am trying to find the best "ranking metric" for my problem but could not find any existing suitable metric, can someone help.
I'm interested in information retrieval.
For item there is a "revelancy" between 0 and 1 that acts as a label. My algorithm gives a score to each item, and returns the 5 items with the highest score. I want to know how "good" my algorithm is to retrieve the real top 5 events, which are the 5 items with highest relevancy scores. Here are a few properties I'm interested in for my evaluation metric:
Are their "famous" metrics that do this? I couldn't find any.
A criteria I was thinking about is to define my metric as : sum of relevancy of the 5 selected items / sum of relevancy of the 5 most relevant items.
This is somewhat doing what I want but it feels like I'm re-inventing an existing metric