Recommendation Algorithms: Reaching the Promised Land

This week in ISYS6621, I will be sharing my findings on Data Mining on social media data. From data generation by users sharing information and spending times on their favorite platforms, to how this data is stored and collected to how this data is finally computed upon and turned into insight, I’ve stumbled upon some astounding information recently. It turns out Facebook processes nearly 2.5 million posts per minute while 72 hours of video are uploaded YouTube nowadays.

Data never sleeps. Via Domo

Data centers storing our senseless photo albums from the Summer of 2007 have military-grade security. Predictive algorithms analyzing the data we’re producing have made leaps and bounds since the term ‘big data’ came about around 2008, but even though computers may understand parts of our lives better than we do, there’s one thing computers have continued to struggle with.

What I found most interesting about all of my research was the topic of recommendation algorithms. Just off the top of my head, I can picture seeing “Suggested Apps…” in my Twitter timeline, “People you may know..” on LinkedIn, “Recommended Games” on Facebook and plenty more. Whether supporting the service’s growth model or advertising platform, recommendations have long been at the backbone of social media platforms. Realistically though, how often are these recommendation actually things we like though? In the case of advertisements, they’re often so obnoxious that even if I were being recommended an app destined to be the next Instagram I wouldn’t care. And with “people you may know,” the results are often so trivial that a good match doesn’t surprise anyone anymore. Oh we have over 100 friends in common? Yea we might know each other. But what about when the software we use is genuinely giving us its best guess at what we might like though?   

It’s a paradigm that has to take a multitude factors into account to be accurate, the kind of puzzle that has consistently stumped data scientists. A few years back, Netflix even hosted a $1,000,000 challenge to whoever could build the best recommendation software to keep its users on the couch even longer than they were. Until recently, I was convinced that there never really would be a program capable of knowing what I liked better than I could myself.

When I first saw the “Discover Weekly” playlist pop up on my Spotify my gut feeling was that it was a PR move to take the fresh launch of Apple’s Beats Radio out of the spotlight. Regardless, I tried it and found myself pleasantly surprised. In it’s debut, the 30 song playlist worked as advertised for me. I wound up saving about one third of the playlist’s songs over the course of the week, and the following Monday by the time I woke up I had clean Discover Weekly waiting for me.

Somehow Spotify had done it again, finding bands I had never heard of yet musicI loved. On the third week however, a “known bug” kept me from Monday pickup of a new Discover Weekly playlist. That afternoon I stumbled upon a Buzzfeed article entitled Spotify’s Discover Weekly Updated Late And People Were Furious, and that’s when I knew that Spotify was onto something big.

via buzzfeed

via buzzfeed

I clearly wasn’t the only one one relying on Spotify’s computers to tell me what I love and for me at least, this felt like the first major triumph for recommendation algorithms. My hunch was all but confirmed when today I saw that The Verge had published a longform article called Tastemaker: How Spotify’s Discover Weekly cracked human curation at internet scale. The article itself is quite thorough and well worth the read, but I’ve decided to included a few paragraphs from it does a great job of getting at the ‘how’ Discover Weekly has become so successful for so many.  

The technology that makes Discover Weekly possible comes in part from a Boston-based startup called The Echo Nest, which Spotify acquired in March of 2014… The company became one of the best in the business and helped power recommendation systems for Rdio, Spotify, Deezer, iHeartRadio, and Rhapsody. But it never had a massive user base of its own that it could leverage to build new tools. “You have really good people, you have some really good algorithms. They’re only [as] good as the data that you have,” says professor Downie. That changed when The Echo Nest became part of Spotify and could tap its 75 million users.

The combination of The Echo Nest technology and Spotify’s massive data trove led to Discover Weekly. Here’s how it works: Spotify has built a taste profile for each user based on what they listen to. It assigns an affinity score to artists, which is the algorithm’s best guess of how central they are to your taste. It also looks at which genres you play the most to decide where you would be willing to explore new music.

The algorithms behind Discover Weekly finds users who have built playlists featuring the songs and artists you love. It then goes through songs that a number of your kindred spirits have added to playlists but you haven’t heard, knowing there is a good chance you might like them, too. Finally, it uses your taste profile to filter those findings by your areas of affinity and exploration. 

In a sense, the system works like the original Page Rank, (named for Larry Page), the technique Google used to revolutionize web search. Page Rank crawled the web to find hyperlinks and treated each one as a vote pointing toward useful information. A big batch of links pointing to a website about Elvis indicated to Google that site was a good resource on the The King. In Discover Weekly, each time a user with similar taste playlists a certain song, it’s a vote that the song will sound good to you when paired with other tracks on that playlist.

I may not have the credentials of Spotify’s Discover Weekly computers, but I highly recommend giving this playlist a chance.


  1. This is an interesting topic. I agree that many existing recommendation algorithms don’t work that well. Despite advances in data mining, there is still so much untapped data, with the potential for better and more relevant recommendations. Companies such as Google and Facebook have large amounts of data that can be harnessed to provide a better user experience. It is interesting how Spotify relies on how your preferences align with those of other users to provide music recommendations. Spotify has an interesting vision for the future, using your past behavior (as well as that of other users) to predict what music you want to hear, when you want (based on your location, what activity are engaged in, etc.). Spotify’s Discover Weekly and Apple News are some examples of where we can go with this, but it may take a while to get there.

  2. As a big data nerd, I found this post extremely interesting and informative. I love that you mentioned Netflix’s challenge. Learning about that challenge was the first time I really understood the power of predictive analytics… Now, almost every major company is trying to “reach the promised land.” Predictive analytics really draw customers in and, like you said, encourage them to buy things that they didn’t even know they wanted until the algorithm recognized they wanted it. There is just so much money to be made by investing in predictive analytics, so I think we will see the quality of these algorithms improve rapidly. I think we will soon reach a point where computers know us better than we know ourselves. This could potentially be a good thing with all of the time we will save looking for the perfect movies, songs, and other items, but will it hinder our ability to grow and change? Will algorithms tell us who to be instead of letting us grow ourselves? I think this is an issue generations will face in the future. Ultimately, technology can’t become smarter without having a human impact.

  3. Hey Ryan, Great post. As a big Spotify user, I was a similarly pleasantly-surprised by my Discover Weekly Playlist. It is just something that we inherent, but the results are great Thank you for sharing the post from The Verge. Last year I had the opportunity to visit the Echo Nest on a Boston BC Tech Trek and learn about their incredible technology. I was blown away but the impressive tools that they are cooking up over in Somerville, and I had assumed the Discover Weekly was powered by their magic. The amount of metadata they have for each song in their catalogue is astounding, so it does not surprise me that they are able to satisfies individual tastes so well!

  4. I have been a spotify user for a long time now and to be honest I had never noticed this feature. I think that i have probably looked at it but i never really saw that it might help me. I always search for the specific music that i want to hear. With your blog in mind i am definitely going to try this out from now on. These algorithms that not only spotify use but other platforms use as well seem to be extremely effective. I will be trying this out in the next coming days and will try to post back what i think about it. Great article!

  5. ashleighpopera · ·

    Really interesting post, Ryan. I use Spotify daily but have never used this feature. I decided to give it a go after reading your article and have to agree wholeheartedly with you that I was pleasantly surprised and ended up saving several of the songs. I think what makes Spotify’s approach stand out from others is the idea of finding content that you have never heard of, but like, while companies like Facebook provide you with recommendations that you likely have already encountered. Other companies and social media platforms could certainly gain some big takeaways from Spotify’s approach to recommendation algorithms in order to make better use of their users’ data.

  6. Though I listen to tons of music, I typically buy or download in other ways than using Spotify. I think in general though I have noticed that social media has been improving when it comes to recommendations for friends, apps, or things in general I might like. I actually still find Netflix to be the best at giving spot-on recommendations to me. Sometime I seriously think that it knows me better than I know myself. I hope in the coming years algorithms keep improving. Who knows, maybe one day we won’t have to do any searching for what we like and sites will just provide us with near perfect recommendations.

  7. I agree with you that predictive analytics is an essential tool for today. There are obviously imperfections but the fact that we can judge what a person may or may not like just by going over their data (with any amount of accuracy), would have blown people’s minds 15 years ago. Facebook’s ‘People you may know’ still is a hit and miss and I agree that its one of the easier predictive analytics given that we all have mutual friends and going from there is more probability than data science. But the analytics used by Netflix or retail companies such as Amazon, CVS or Walmart are amazing on a lot of levels. The example I used in my presentation of Target prediction pregnancies in women might be a one off good case, but the science is definitely getting better. And as more and more corporations get access to more and more of our data, these prediction analytics will get much better. The best example I have observed is Google Now. It crosses the lines in terms of privacy and creepiness by observing everything you do on your phone from contacts to search to maps to mail. But the constant help it gives in terms of parking locations and topics of interest, sports scores and everything without even asking or searching is just pure amazing. I’m waiting for the next steps they take with this.

  8. I think recommendation engines are going to be more essential in the future as it “recommends” people for certain tasks goal at work

%d bloggers like this: