Author: Jack Beckwith
Back in June, we started to contemplate doing an analysis of how the popularity of various music genres has evolved over time. We wanted to look at the trajectory that major genres have followed to reach the mainstream — the years that genres like rock or rap or country spent climbing onto America’s radar, the golden era of each genre when it was adopted and enjoyed by the masses, and the decline older genres suffered as they fell out of favor. In this way, we hoped to capture the “lifecycle” of major genres in American popular music.
So we set about gathering a dataset to suit our needs. Each week since 1958, Billboard magazine has released its Hot 100 chart — a list of the 100 most popular songs in the U.S. according to airplay, streams, and physical sales. We figured that was a good place to start. We then paired each of the 27,000+ songs that have appeared on the Hot 100 with an appropriate genre. To determine a genre for each song, we leaned heavily on the Spotify API, with supplemental data from EveryNoise.com, AllMusic.com, and Wikipedia for songs missing from Spotify. We think that this is the most complete dataset of Hot 100 songs matched to their respective genres publicly available. For more on our data collection process, check out the “About This Project” section at the end.
Here’s a look at every single song that has reached the Hot 100, stretching back to 1958. Each tiny rectangle denotes a different song. The larger the rectangle, the longer the song stayed on the Hot 100. Click on one of the rectangles to pull up the song in the Spotify player (if it’s on Spotify that is). Use the filters at the right to narrow your search to a particular artist or genre.
A few notes here:
Once each song was categorized, we looked at how a genre’s representation on the Hot 100 chart changed over time. To do so, we calculated the total number of Hot 100 spots occupied by each of our 16 major genres in a given year. Consider just 2016 for a second. Our dataset stretches through August 13, meaning that we have 33 Hot 100 charts for this year. Each chart has 100 spots. Thus, there have been 3,300 spots up for grabs so far in 2016. So what percentage of those 3,300 were nabbed by artists from each genre?
Performing the same calculation for each year since 1958, we produced this chart:
There’s obviously a lot to digest here. You can toggle genres on and off the graph by adjusting the “Filter Genres” option in the upper left corner.
Given the trajectories that various genres have followed, it’s clear that there’s no “one-size-fits-all” genre lifecycle. Here’s a brief look at how a few genres have fared:
So if 2016 really is the year of rap, then which rappers are driving a cultural shift towards the genre and away from pop and country? First and foremost, there’s Drake. The Toronto MC has 24 Hot 100 songs to his name in 2016 alone, not even counting an array of guest features that have landed on the charts too. Kanye West, Fetty Wap, and Future are other top contributors from rap with upwards of five solo Hot 100 hits apiece this year. And it’s not just established acts getting in on the fun; there are fourteen hip-hop artists with Hot 100 hits in 2016 that have never charted before.
To explore the data by year, we’ve created this interactive bubble chart. In its preset format, each bubble represents a song that has charted thus far in 2016. You can toggle the year parameter to change the year shown. Again, click on a bubble to pull up the corresponding song in the Spotify player.
A final point: 2016 has provided a little bit of music nostalgia. Even as pop, rap, and country have collectively occupied 86% of the Hot 100 spots this year, there have been flashes of music from bygone eras. Prince, the Minneapolis icon who died in April, has eight songs to his credit that have reached the Hot 100 this year. That’s tied for the most among everyone not named Drake or Beyonce. Then there’s David Bowie, another legend whose career began in the early ’60s. Three of his songs have touched the Hot 100 in 2016, more than any other active rock band can claim. Though rock and soul have fallen out of the mainstream, Americans continue to hold onto artists from each genre’s peak years.
If you have any comments or want to explore the raw data yourself, reach out to us at firstname.lastname@example.org. And if you really want to get on our good side, like us on Facebook.
The data collection process for this project was intensive. We had already scraped data from Billboard’s website for this DataFace post about the hometowns of popular American musicians. So after refreshing the dataset compiled for that post in October 2015, we had every Hot 100 list dating back to 1958. All we had to do was find a way to categorize the songs into their respective genres. Surely there was a website, an API, a random spreadsheet floating around somewhere on the Internet that would have this data for free. Yet despite our best efforts, we couldn’t find anything that provided genre categorization for every song in our dataset, let alone even close.
We ended up using the Spotify API as our primary source of genre data. We grabbed Spotify data about 79% of the songs in our dataset using this Python project (shout out to Allen who maintains the GitHub repository). We filled in the blanks with information from EveryNoise.com, AllMusic.com, and many Wikipedia searches.
Spotify’s genre classification system provided additional challenges. The streaming service categorizes artists into over 1,300 specific, and often unheard of, music genres (anybody familiar with “zydeco”?). Moreover, a given artist may fall into as many as 23 different Spotify genres. We used a two-step process to translate Spotify’s genres to our own genre definition. First, we mapped each of Spotify’s genres to an “overarching” genre — “atmospheric post rock” became Rock, “deep northern soul” turned into Soul. Then, for any artist that fell in multiple Spotify genres, we wrote a short Python script to “vote” on which genre to place the artist in. For instance, Spotify classifies Drake as “pop rap”, “indie r&b”, “alternative hip hop”, and “hip hop”. According to our mapping system, three of those genres fall under rap/hip-hop and one under R&B. Thus, Drake goes under rap/hip-hop.
This is by no means a perfect system. Spotify categorizes music into genres at the artist, not the song, level. In other words, for a musician like Prince — whose catalogue arguably spanned pop, R&B, and soul — we give the same genre designation to all his songs. Moreover, genres are inherently subjective. And often they bleed together as a new genre emerges and is influenced by another. Generally, we think the results are consistent with our expectations. But if you see any song/artist that seems grossly misclassified, please send us an email.