DADA Auto-Rating for foobar2000

date and duration adjusted (DADA) auto-rating algorithm (DAR) for fb2k v.1.1+

Introduction: What is DAR?
Setting up DAR in foobar2000 v.1.1+
- Creating an autoplaylist ranked according to the DAR algorithm
The Auto-Rating Dilemma (How DAR Calculates Ratings)
Recent Changes: v.2.0.1 (05.04.15)
Any Other Business?

Introduction: What is DAR?

The DADA Auto-Rating algorithm (DAR) automatically rates music according to a variety of statistics collected by foobar2000's official "playback statistics" plugin.

If done well, an auto-rating algorithm will simply tell you what your favourite (and conversely, least favourite) music is, and the rating will be derived from and reflect actual listening behaviour. This is quite different from manual ratings (e.g. 1 - 5 stars set by the user), since there's very often a disparity between what we'd like ourselves to like, and what we actually like. For those interested in uncovering their actual musical tastes, a good auto-rating formula can be quite illuminating.

Setting up DAR in foobar2000 v.1.1+

Firstly, if you don't already have them, you'll need the following:

Note: Just follow the instructions in the text file linked to above.

Very simply you paste the code into Yirkha's foo_dynfil component:

click on images to enlarge

Create a custom column with %_dynamic_rating%, and under Library > (select) "Recalculate dynamic fields".

Creating an autoplaylist ranked according to the DAR algorithm

Go to: Library > Search > and paste: " %_dynamic_rating% GREATER 0 " (without the inverted commas) into the search box.
Click on the [...] button and select: "Create Autoplaylist"
Right click on the New Playlist and select "Autoplaylist properties"
Enter " $sub(99999,%_dynamic_rating%) " in the box as per below:

Congratulations. Your music is now being rated automatically by DADA Auto-Rating and foobar2000.

For those interested in what the DAR algorithm actually does, read on ...

The Auto-Rating Dilemma (How DAR Calculates Ratings)

The DAR algorithm solves a number of problems related to auto-rating music. In this section we'll explore how it does this, but first: "what's the problem?":

The Plays per Period problem:
"Total Playcount" / "No. Days in Library" will give an unfair advantage to new tracks.
The Total Playcount problem:
Total playcount will give an unfair advantage to old tracks.
The Track Duration & Playcount problem:
Song A = 30 mins, Song B = 3 mins, the opportunity cost of listening to Song A is listening to Song B 10 times. If playcount is at the root of the rating (and it has to be), somehow duration has to be factored in, because we are physically limited by time (lifetime/listening time). No matter how much one likes Song A you simply cannot fit as many plays of that track into a lifetime as you can with Song B. So all other things being equal, Song B has an advantage (due to its duration) and this has to be adjusted for.

The DAR formula attempts to equalise these issues so that regardless of the above, all tracks are rated on a level playing field. As you can see below, new tracks (e.g. 29 days old, playcount = 8) are competing with old tracks (3312 days old, playcount = 41), and short tracks (2:58) are competing with much longer tracks (22:21).

(obviously this data would normally accompany Track Title & Artist etc...

Duration

As the graph below illustrates, if duration is not taken into account at all, then a track listened to for 2,700 seconds achieves the same rating (9889) as one listened to for 27,000 seconds (10 times as long). However, if one takes absolute account of duration then the playcount of a song becomes almost secondary. DAR strikes an even balance between playcount and total listening time.

The graph below is a useful way to express the problem. From the "playcount" point of view it seems as though long tracks have a distinct advantage (e.g. to achieve a rating of 10500 after 900 days, a 4 min song has to be played approx 57 times), however, for the same rating, a 20 minute song need only be played about 25 times, so it seems unfair. Yet from the "total listening time" point of view, the 20 min song is playing for more than double the time.

Age

The DAR algorithm uses two independent formulas (one for old and one for new tracks) to suppress or boost the effect of time on a tracks rating. Below, the 30 min track shows the effect of boosting or suppressing both simultaneously; the 3 min track shows an example of suppressing one (i.e. old tracks) while boosting the other (i.e. new tracks) and vice versa. Adjusting these variables may be of use for people with very small (1,000 tracks) or very large collections (i.e. 50,000 tracks).

NOTE: The old track suppression variable has changed slightly since version 1.6.4 (see the graph beneath this one).

Since v1.6.4 the old track suppression multiplier is now a dynamic variable. The effect has been to reduce the "performance burden" of older tracks with each additional year in the collection.

The "play per period problem" especially in the first few days can have a profoundly distorting effect. For example, you may have a track you've played 100 times in 2 years (that's a lot!), which means you play that track approx. 1 day in every 7. However, a new song played twice, straight away (has been played 2 times in 1 day), when extrapolated this suggests it will be played 14 times every 7 days, i.e. it's 14 times more popular (on a "plays per day" measure) than the track you played 100 times. Yet, it's highly unlikely you'll end up listening to that song 1400 or so times over 2 years.

For this reason, the DAR formula suppresses this effect in proportion to the age of the track. You can see this below in the playcount required to achieve a constant rating (for a rating of 10109 a brand new track has to be played almost 6 times, for the same rating after 9 weeks, the same track only has to be played 10 times).

Low Playcounts & Predictability

When tracks have a very low playcount (i.e. < 5) the effects of the low playcount penalty are a dominant factor in the DAR rating scheme. As the playcount increases this effect diminishes quickly (ultimately to zero). It's primary purpose is to spread the rating over a wide range. Furthermore, a track played just once or twice, whether long or short, young or old cannot be said to be a favourite, and with so few plays predicting its popularity is problematic.

Recent Changes: v.2.0.1 (05.04.15)

The Low Playcount Penalty

The Problem:

New tracks were getting too high a rating too easily
Long tracks were getting too high a rating on too low a playcount

What the changes mean:

A significant drop in ratings for tracks with playcounts between 2 and 10, while tracks with high playcounts (20+) will remain pretty much the same.
The penalty differential between long and short tracks has been reduced, so long duration tracks still reduce their penalty quicker than short tracks but this duration benefit has been significantly reduced.
No major jump going from 1 play to 2 plays (as there was in the prior version).
The role of delaying first playing a track (no. days between added and first played) has been reduced.
The new penalty reduces to zero slightly faster than previously.

Here's the maths (click to enlarge):

Here's the difference between the new and old penalties in isolation:

And here's the effect on the overall DAR rating:

Note the x axis jump from 1 - 5 to 10 - 50

The Floor, the Fish & the Auto-Boost

The Problem:

If someone has a very old library, an old track with a low playcount could get a negative rating. In versions prior to v.2.0.0 this was blocked by putting a floor on the ratings:

$puts(r4,$ifgreater($get(r3),0,$get(r3),1))

i.e. if the rating is less than 1 give it a rating of 1. However this kind of floor creates a distortion since a real rating of -2000 gets the same rating as one which is -5.

Analogy:

The DAR Rating works like salmon swimming upstream. Time is the stream's flow and each play is the energy the track requires to move against that flow. However, if one listens to some music only rarely and their library is old; that's like salmon getting really tired. The floor on the ratings is like a dam. Over time all these old tired salmon get squished up against the wall of the dam and it's impossible to know which one's a real stinker and which is just rubbish.

The new DAR version removes the dam (the ratings floor) and instead allows users to a) slow the stream's flow and b) move all the fish further upstream. Thus all these old tired salmon now have miles of river before they hit the site of the old dam. To stretch the analogy, the athletic salmon (the tracks you play a lot) will now be even further from the old dam than they were prior to the boost. The boost lifts all fish ... but not equally.

The rubbish salmon now have more river (in the example above 4580's worth) to slide further downstream. Meanwhile the athletic salmon keep swimming further upstream. Consequently, the boost results in increasing the (high-low) range of the overall ratings.

The graph below shows why it's necessary for the Auto-Boost to be connected to the Low Playcount Penalty. Since the bulk of one's ratings are slowly dropping, a new track (the pink dash) needs a lower point of entry (since that new track also benefits from any boost).

Note the long time frames (5 year periods) on the x axis

An example of how the Auto-Boost works

Auto-Boost "clock value" set at: 1980
Number of days between 1980 and now (05/02/15) = 12819.
12819 / 5 = 2564
All tracks will have 2564 added to their DAR Rating.
2% of 2564 = 51.
51 is then added to one of the coefficients of the Low Playcount Penalty equation making the penalty slightly larger.

Playing around with the Auto-Boost

To change the "clock value" year (in bold):
$puts(b1,$add($div($date_diff(1900),5),0))
To switch this auto-boost off use this line:
$puts(b1,$add($div($date_diff(2015),500000000),0))
To increase the rate at which the auto-boost is added change the 5 to 4 or 3, but don't go any lower than 3:
$puts(b1,$add($div($date_diff(2015),4),0))
You can also add a static boost. So, if you wanted to add a one off 6000 to all ratings, you could use this:
$puts(b1,$add($div($date_diff(2015),500000000),6000))
Or you could change all 3 variables at once. Putting back the "clock value" (to the year 2000), adding a static boost (1000) and changing the rate of the clock (4). This would add: 1380 (days since 1/1/2000 / 4) + 1000 = 2380, and add an additional 1 point every 4 days:
$puts(b1,$add($div($date_diff(2000),4),1000))

The DAR Index

The Auto-Boost & DAR Indexing

The DAR Index Dots

Finally, a brief look at the effect of all this on the DAR Index. The DAR Index code is used to turn the DAR values into pretty, user-friendly dots:

The graph below shows the effect of applying different auto-boost levels to the spread of indexed DAR ratings (dots). The mindar to maxdar range of 5000 (e.g. 13000 to 18000) is used in each case to show how the DAR Index values may need to be looked at after a change to the auto-boost "clock value".

This graph should be helpful for those struggling to get the right spread of ratings dots (i.e. how to set the right mindar and maxdar values).

As a rule. If you want a reasonably equal spread of DAR Indexed dots (a flat line in the graph above).
I'd recommend doing something like this:

mindar: Take the lowest DAR Rating value, round up to the nearest 1000 and add 1000.
maxdar: Take the highest DAR Rating value, subtract 2000 and round down to the nearest 500.

That's pretty much it for the update.

Any Other Business?

Special thanks go to Yirkha (at Hydrogen Audio) for creating his excellent and elegant component (currently in beta) foo_dynfil.

Any feedback, suggestions, questions visit the DAR thread on the foobar2000 forum at Hydrogen Audio.