{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Cargar los datos\n",
    "\n",
    "Vamos a empezar cargando los datos de artistas, del fichero **artists.dat**. Para facilitar el visualizado de los datos ordenaremos alfabéticamente los grupos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import csv\n",
    "pd.set_option('display.max_rows', None, 'display.max_columns', None)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('artists.dat', header=0, sep='\\t', usecols=[0,1])\n",
    "df.rename(columns={'id': 'artistID'}, inplace=True)\n",
    "df = df.sort_values('name')\n",
    "df = df.reset_index(drop=True)\n",
    "\n",
    "assert list(df['artistID'][df['name'] == 'MALICE MIZER'])[0] == 1\n",
    "assert list(df['artistID'][df['name'] == 'Katatonia'])[0] == 370\n",
    "assert list(df['artistID'][df['name'] == 'Eluveitie'])[0] == 837\n",
    "assert list(df['artistID'][df['name'] == 'Windy & Carl'])[0] == 6155\n",
    "assert list(df['artistID'][df['name'] == 'Emilie Autumn'])[0] == 5079\n",
    "assert list(df['artistID'][df['name'] == 'Hamlet'])[0] == 10302\n",
    "assert list(df['artistID'][df['name'] == 'Goot'])[0] == 14011"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Visualizar los datos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>artistID</th>\n",
       "      <th>name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>14846</td>\n",
       "      <td>!!!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10635</td>\n",
       "      <td>!DISTAIN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8209</td>\n",
       "      <td>!deladap</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>759</td>\n",
       "      <td>#####</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10004</td>\n",
       "      <td>#2 Orchestra</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   artistID          name\n",
       "0     14846           !!!\n",
       "1     10635      !DISTAIN\n",
       "2      8209      !deladap\n",
       "3       759         #####\n",
       "4     10004  #2 Orchestra"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.set_option('display.max_rows', None, 'display.max_columns', None) #Permite mostrar todas las filas del dataframe\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Limpieza de datos y Data Enrichment\n",
    "\n",
    "En este apartado vamos a atajar principalmente dos puntos:\n",
    "1. Realizar una limpieza de artistas, usando el número de usuarios del que disponemos información y el número de oyentes. Este último lo recopilaremos usando la API que pone last.FM a nuestra disposición.\n",
    "2. Por otra parte, asignaremos a cada artista tags o etiquetas, que corresponderán principalmente a géneros musicales. Para ello usaremos también la API de last.FM, y realizaremos una selección de tags manual y una posterior limpieza, de manera que se use un número limitado de etiquetas y además con la seguridad de que son válidas y útiles.\n",
    "\n",
    "\n",
    "### 3.1 Purga de grupos\n",
    "\n",
    "Queremos:\n",
    "\n",
    "- Comprobar que no hay grupos repetidos (mismo grupo, distinto formato de string, mayúsculas, minúsculas, símbolos o abreviaturas).  \n",
    "- Comprobar que no hay grupos incorrectos: fechas, títulos de vídeos de youtube, colaboraciones o símbolos inválidos.\n",
    "- Eliminar grupos poco relevantes (con un número muy bajo de oyentes).\n",
    "\n",
    "Para ello, eliminar los artistas con menos de X 'listeners' usando la API de last.fm parece ser una opción viable, ya que los grupos incorrectos suelen tener un número bajo de oyentes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://www.last.fm/api/show/artist.getInfo\n",
    "\n",
    "Get the metadata for an artist. Includes biography, truncated at 300 characters.\n",
    "\n",
    "**artist (Required (unless mbid)] :** The artist name  \n",
    "    **mbid (Optional) :** The musicbrainz id for the artist  \n",
    "    **lang (Optional) :** The language to return the biography in, expressed as an ISO 639 alpha-2 code.  \n",
    "    **autocorrect[0|1] (Optional) :** Transform misspelled artist names  into correct artist names, returning the correct version instead.   The corrected artist name will be returned in the response.  \n",
    "    **username (Optional) :** The username for the context of the request. If supplied, the user's playcount for this artist is included in the response.  \n",
    "**api_key (Required) :** A Last.fm API key.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def artist_getInfo(artist, api_key='3675d2cb82f8d59bcc885774869aca18', mbid=None, lang=None, autocorrect=None, username=None,  maxTags = 10):\n",
    "    import http.client,  urllib.parse, json\n",
    "\n",
    "    #Parametros necesarios\n",
    "    params_dict = {\n",
    "            'method':'artist.getInfo',\n",
    "            'artist' : artist,\n",
    "            'api_key' : api_key,\n",
    "            'format' : 'json'\n",
    "    }\n",
    "\n",
    "    #Parametros opcionales\n",
    "    if mbid:\n",
    "        params_dict['mbid'] = mbid\n",
    "    if lang:\n",
    "        params_dict['lang'] = lang\n",
    "    if autocorrect:\n",
    "        params_dict['autocorrect'] = autocorrect\n",
    "    if mbid:\n",
    "        params_dict['username'] = username\n",
    "    \n",
    "\n",
    "    params = urllib.parse.urlencode(params_dict) #Parametros de la llamada\n",
    "    conn = http.client.HTTPConnection('ws.audioscrobbler.com') #URL de la API\n",
    "    conn.request('GET', '/2.0/', params, {}) #Llamada a la API\n",
    "    res = conn.getresponse() #Obtenemos la respuesta\n",
    "    body = json.loads(res.read().decode('utf-8')) #Leemos el body y lo convertimos en dict\n",
    "    conn.close() #Cerramos la conexion\n",
    "\n",
    "    #Extras\n",
    "    #headers = res.getheaders() #Leemos los headers\n",
    "    #code = res.getcode() #Leemos el codigo (200 Ok)\n",
    "    #reason = res.reason #Leemos la descripcion del codigo\n",
    "    if 'artist' in body.keys():\n",
    "        listeners = int(body['artist']['stats']['listeners'])\n",
    "    else:\n",
    "        listeners = -1\n",
    "        \n",
    "    return listeners"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Llamamos a la API tantas veces como sea necesario, ya que en ocasiones falla la conexión y se interrumpe el proceso. Para ello, lo único que debemos hacer es copiar el contenido conseguido hasta ahora ejecutando la celda inferior, y a continuación cambiar el índice 'idx' para seguir por donde lo hemos dejado.\n",
    "\n",
    "Realmente podríamos hacerlo todo seguido y copiarlo al final, pero debido a lo lento del proceso, me parece más conveniente y seguro ir copiando en el fichero de texto a medida que conseguimos los datos. \n",
    "\n",
    "Por seguridad, prefiero añadirle un entero aleatorio al inicio del fichero, por si ejecutamos accidentalmente la celda."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "#import random\n",
    "#number = random.randint(0,1000000)\n",
    "#with open(str(number)+\"artist_listeners.dat\", \"a\") as f:\n",
    "#    writer = csv.writer(f, delimiter='\\t')\n",
    "#    writer.writerow(('artistID', 'name', 'listeners'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "En total deberemos ejecutar las dos celdas de a continuación unas 10 veces. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#artist_listeners = []\n",
    "#idx = 0\n",
    "#for index, row in df[idx:].iterrows():\n",
    "#    print(index)\n",
    "#    artist_listeners.append((row['id'], row['name'], artist_getInfo(row['name'])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#with open(str(number)+\"artist_listeners.dat\", \"a\") as f:\n",
    "#    writer = csv.writer(f, delimiter='\\t')\n",
    "#    writer.writerows(artist_listeners)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ahora que ya hemos conseguido el número de oyentes de cada grupo, vamos a ordenarlos de más a menos oyentes, y vamos a **decidir** cual es **el umbral** que nos resulta más adecuado para descartar los artistas.\n",
    "\n",
    "Deberemos repetir los pasos vistos en el cargado de datos para el nuevo fichero construido."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>artistID</th>\n",
       "      <th>name</th>\n",
       "      <th>listeners</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>65</td>\n",
       "      <td>Coldplay</td>\n",
       "      <td>5369132</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>154</td>\n",
       "      <td>Radiohead</td>\n",
       "      <td>4718748</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>220</td>\n",
       "      <td>Red Hot Chili Peppers</td>\n",
       "      <td>4608144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>288</td>\n",
       "      <td>Rihanna</td>\n",
       "      <td>4542147</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>475</td>\n",
       "      <td>Eminem</td>\n",
       "      <td>4504111</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   artistID                   name  listeners\n",
       "0        65               Coldplay    5369132\n",
       "1       154              Radiohead    4718748\n",
       "2       220  Red Hot Chili Peppers    4608144\n",
       "3       288                Rihanna    4542147\n",
       "4       475                 Eminem    4504111"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "#Cargamos los datos\n",
    "df_listeners = pd.read_csv('artist_listeners.dat', header=0, sep='\\t')\n",
    "#Los ordenamos\n",
    "df_listeners = df_listeners.sort_values('listeners', ascending=False).reset_index(drop=True)\n",
    "#Los visualizamos\n",
    "pd.set_option('display.max_rows', None, 'display.max_columns', None)\n",
    "display(df_listeners.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "También vamos a usar el número de usuarios del que disponemos datos para cada grupo, por lo que vamos a cargar el fichero **user_artists.dat** y vamos a contar las veces que aparece cada artista."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userID</th>\n",
       "      <th>artistID</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>51</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>54</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>55</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userID  artistID\n",
       "0       2        51\n",
       "1       2        52\n",
       "2       2        53\n",
       "3       2        54\n",
       "4       2        55"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "#Cargamos los datos\n",
    "df_ua = pd.read_csv('user_artists.dat', header=0, sep='\\t', usecols=[0,1])\n",
    "#Los visualizamos\n",
    "pd.set_option('display.max_rows', None, 'display.max_columns', None)\n",
    "display(df_ua.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>artistID</th>\n",
       "      <th>freq</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>89</td>\n",
       "      <td>611</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>283</th>\n",
       "      <td>289</td>\n",
       "      <td>522</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>282</th>\n",
       "      <td>288</td>\n",
       "      <td>484</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>221</th>\n",
       "      <td>227</td>\n",
       "      <td>480</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294</th>\n",
       "      <td>300</td>\n",
       "      <td>473</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     artistID  freq\n",
       "83         89   611\n",
       "283       289   522\n",
       "282       288   484\n",
       "221       227   480\n",
       "294       300   473"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_freq = df_ua[['artistID']].copy()\n",
    "df_freq = df_freq.groupby(['artistID']).size().reset_index()\n",
    "df_freq.rename(columns = {0: 'freq'}, inplace = True)\n",
    "df_freq = df_freq.sort_values('freq', ascending=False)\n",
    "df_freq.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ahora que ya tenemos tanto el número de oyentes como el número de veces que aparece el artista en nuestros datos, vamos a construir un nuevo dataframe que reuna ambos campos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>artistID</th>\n",
       "      <th>name</th>\n",
       "      <th>freq</th>\n",
       "      <th>listeners</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>65</td>\n",
       "      <td>Coldplay</td>\n",
       "      <td>369</td>\n",
       "      <td>5369132</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>154</td>\n",
       "      <td>Radiohead</td>\n",
       "      <td>393</td>\n",
       "      <td>4718748</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>220</td>\n",
       "      <td>Red Hot Chili Peppers</td>\n",
       "      <td>222</td>\n",
       "      <td>4608144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>288</td>\n",
       "      <td>Rihanna</td>\n",
       "      <td>484</td>\n",
       "      <td>4542147</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>475</td>\n",
       "      <td>Eminem</td>\n",
       "      <td>204</td>\n",
       "      <td>4504111</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   artistID                   name  freq  listeners\n",
       "0        65               Coldplay   369    5369132\n",
       "1       154              Radiohead   393    4718748\n",
       "2       220  Red Hot Chili Peppers   222    4608144\n",
       "3       288                Rihanna   484    4542147\n",
       "4       475                 Eminem   204    4504111"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_freq_listeners = df_freq.join(df_listeners.set_index('artistID'), on='artistID').reset_index(drop=True)\n",
    "df_freq_listeners = df_freq_listeners.sort_values('listeners', ascending=False).reset_index(drop=True)\n",
    "df_freq_listeners = df_freq_listeners[['artistID', 'name', 'freq', 'listeners']]\n",
    "df_freq_listeners.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Y ahora sí vamos a decidir qué umbrales aplicamos. Podemos jugar con los dos campos, frequency y listeners, y ver qué nos resulta más conveniente.\n",
    "Después de hacer varias combinaciones y de analizar un poco los datos, creo que lo más sensato es eliminar los artistas de los que no disponemos datos de al menos de 3 usuarios y de los muy poco relevantes (menos de 5000 oyentes).\n",
    "\n",
    "Por qué **eliminar los que tienen menos de 3 usuarios en nuestros datos**? Porque cuando tengamos que hacer los ejes entre grupos usaremos el número de usuarios en común. En el caso de los artistas con 1 solo usuario, tendrán 100% de usuarios en común con todo el resto de artistas de ese usuario. En el caso de los artistas con 2 usuarios, tendremos exactamente el mismo problema si ponemos el umbral de unión a menos del 50%. Por lo tanto, creo que tener en cuenta esos artistas puede ensuciar los datos y causar resultados erróneos.\n",
    "\n",
    "Por qué **eliminar los artistas muy poco relevantes**? Porque esto además nos asegura que no hay artistas con nombres incorrectos, suelen tener entre 0 y 100 oyentes (o -1 en el caso de los artistas no encontrados). No queremos ruido en nuestros datos.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>artistID</th>\n",
       "      <th>name</th>\n",
       "      <th>freq</th>\n",
       "      <th>listeners</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>89</td>\n",
       "      <td>Lady Gaga</td>\n",
       "      <td>611</td>\n",
       "      <td>3798959</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>289</td>\n",
       "      <td>Britney Spears</td>\n",
       "      <td>522</td>\n",
       "      <td>3243214</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>288</td>\n",
       "      <td>Rihanna</td>\n",
       "      <td>484</td>\n",
       "      <td>4542147</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>227</td>\n",
       "      <td>The Beatles</td>\n",
       "      <td>480</td>\n",
       "      <td>3658694</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>300</td>\n",
       "      <td>Katy Perry</td>\n",
       "      <td>473</td>\n",
       "      <td>3721531</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   artistID            name  freq  listeners\n",
       "0        89       Lady Gaga   611    3798959\n",
       "1       289  Britney Spears   522    3243214\n",
       "2       288         Rihanna   484    4542147\n",
       "3       227     The Beatles   480    3658694\n",
       "4       300      Katy Perry   473    3721531"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_purga = df_freq_listeners[~((df_freq_listeners.freq < 3) | (df_freq_listeners.listeners < 5000))]\n",
    "df_purga = df_purga.sort_values('freq', ascending=False).reset_index(drop=True)\n",
    "df_purga.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4562"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df_purga.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert list(df_purga['artistID'][df_purga['name'] == 'MALICE MIZER'])[0] == 1\n",
    "assert list(df_purga['artistID'][df_purga['name'] == 'Katatonia'])[0] == 370\n",
    "assert list(df_purga['artistID'][df_purga['name'] == 'Eluveitie'])[0] == 837\n",
    "assert list(df_purga['artistID'][df_purga['name'] == 'Windy & Carl'])[0] == 6155\n",
    "assert list(df_purga['artistID'][df_purga['name'] == 'Emilie Autumn'])[0] == 5079"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "df_purga.to_csv(\"artists-v2.dat\", sep='\\t', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Obtención de tags\n",
    "\n",
    "Para empezar vamos a obtener los 10 tags más frecuentes de cada artista. Para ello hemos creado una función que nos permite recopilar los tags más frecuentes de cada artista (10 por defecto). Esta función puede modificarse para que devuelva otra información si se desea."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://www.last.fm/api/show/artist.getTopTags\n",
    "\n",
    "Get the top tags for an artist on Last.fm, ordered by popularity.  \n",
    "\n",
    "**artist (Required (unless mbid)] :** The artist name  \n",
    "**mbid (Optional) :** The musicbrainz id for the artist  \n",
    "**autocorrect[0|1] (Optional) :** Transform misspelled artist names into correct artist names, returning the correct   version instead. The corrected artist name will be returned in the response.  \n",
    "**api_key (Required) :** A Last.fm API key.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "def artist_getTopTags(artist, api_key='3675d2cb82f8d59bcc885774869aca18', mbid=None, autocorrect=None, maxTags = 10):\n",
    "    import http.client,  urllib.parse, json\n",
    "\n",
    "    #Parametros necesarios\n",
    "    params_dict = {\n",
    "            'method':'artist.getTopTags',\n",
    "            'artist' : artist,\n",
    "            'api_key' : api_key,\n",
    "            'format' : 'json'\n",
    "    }\n",
    "\n",
    "    #Parametros opcionales\n",
    "    if autocorrect:\n",
    "        params_dict['autocorrect'] = autocorrect\n",
    "    if mbid:\n",
    "        params_dict['mbid'] = mbid\n",
    "\n",
    "    params = urllib.parse.urlencode(params_dict) #Parametros de la llamada\n",
    "    conn = http.client.HTTPConnection('ws.audioscrobbler.com') #URL de la API\n",
    "    conn.request('GET', '/2.0/', params, {}) #Llamada a la API\n",
    "    res = conn.getresponse() #Obtenemos la respuesta\n",
    "    body = json.loads(res.read().decode('utf-8')) #Leemos el body y lo convertimos en dict\n",
    "    conn.close() #Cerramos la conexion\n",
    "\n",
    "    #Extras\n",
    "    #headers = res.getheaders() #Leemos los headers\n",
    "    #code = res.getcode() #Leemos el codigo (200 Ok)\n",
    "    #reason = res.reason #Leemos la descripcion del codigo\n",
    "    tags = []\n",
    "    #Obtenemos únicamente los tags\n",
    "    i = 0\n",
    "    for d in body['toptags']['tag']:\n",
    "        if i == maxTags:\n",
    "            break;\n",
    "        tag = d['name']\n",
    "        tags.append(tag.lower())\n",
    "        i+=1\n",
    "        \n",
    "    return tags\n",
    "\n",
    "RHCP_tags = artist_getTopTags('Red Hot Chili Peppers')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"['rock', 'alternative rock', 'alternative', 'funk rock', 'funk', 'seen live', 'red hot chili peppers', 'punk', 'american', 'indie']\""
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "str(RHCP_tags)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Antes de nada, cargamos en un dataframe el nuevo archivo *artists.dat* obtenido en el apartado anterior."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('artists-v2.dat', header=0, sep='\\t', usecols=[0,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Y repetimos los mismos pasos que para la obtención del número de oyentes, pero esta vez para las etiquetas de los artistas,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "#import csv\n",
    "#import random\n",
    "#number = random.randint(0,1000000)\n",
    "#with open(str(number)+\"artist_tags.dat\", \"a\") as f:\n",
    "#    writer = csv.writer(f, delimiter='\\t')\n",
    "#    writer.writerow(('artistID', 'name', 'tag1', 'tag2', 'tag3', 'tag4', 'tag5', 'tag6', 'tag7', 'tag8', 'tag9', 'tag10'))\n",
    "#    \n",
    "#\n",
    "#with open(str(number)+\"tags.dat\", \"a\") as f:\n",
    "#    writer = csv.writer(f)\n",
    "#    writer.writerow(('tag'))   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "#artist_tags = []\n",
    "#all_tags = []\n",
    "#idx = 3531\n",
    "#for index, row in df[idx:].iterrows():\n",
    "#    print(index)\n",
    "#    tags_list = artist_getTopTags(row['name']) \n",
    "#    artist_tags.append([row['artistID'],row['name']]+tags_list)\n",
    "#    all_tags = all_tags+tags_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Debido a que posteriormente necesitaré contabilizar el número de apariciones de cada tag para hacer una limpieza de tags, he optado por hacer otro archivo complementario, a medida que realizo el de artistas y etiquetas, que contiene únicamente las etiquetas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "#with open(str(number)+\"artist_tags.dat\", \"a\") as f:\n",
    "#    writer = csv.writer(f, delimiter='\\t')\n",
    "#    writer.writerows(artist_tags)\n",
    "#    \n",
    "#with open(str(number)+\"tags.dat\", \"a\") as f:\n",
    "#    writer = csv.writer(f, delimiter='\\n')\n",
    "#    writer.writerow(all_tags)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Una vez ya tenemos los dos archivos, **artist_tags.dat** y **tags.dat**, los cargamos para trabajar con ellos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_artist_tags = pd.read_csv('artist_tags.dat', header=0, sep='\\t')\n",
    "df_tags = pd.read_csv('tags.dat', header=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculamos la frecuencia de cada tag y los ordenamos de más a menos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_tags_freq = df_tags.groupby(['tag']).size().reset_index()\n",
    "df_tags_freq.rename(columns = {0: 'freq'}, inplace = True)\n",
    "df_tags_freq = df_tags_freq.sort_values('freq', ascending=False).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creamos un nuevo fichero **tags_freq.dat** con los tags y sus frecuencias."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "df_tags_freq.head()\n",
    "df_tags_freq.to_csv(\"tags_freq.dat\", sep='\\t', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Manualmente he realizado una selección de tags de los 420 tags más frecuentes. Por un lado están los tags a eliminar **tags_to_delete.dat**, que son tags poco útiles o incorrectos. La mayoría corresponden a nacionalidades u otras características del artista no relevantes.  \n",
    "Por otro lado estan los tags a sustituir **tags_to_replace.dat**, que son tags incorrectos pero que pueden substituirse por un tag correcto, como géneros repetidos escritos de distinta forma (hip hop en lugar de hip-hop) o géneros que se han acompañado con una nacionalidad, sin tener distinción alguna del género original (*rock brasileño* o *rock ruso* en lugar de simplemente *rock*)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_tags = pd.read_csv('tags_freq.dat', header=0, sep='\\t', usecols =[0])\n",
    "df_tags_replace = pd.read_csv('tags_to_replace.dat', header=0, sep='\\t')\n",
    "#df_tags_delete = pd.read_csv('tags_to_delete.dat', header=0)\n",
    "df_tags_delete = pd.read_csv('tags_to_delete-v2.dat', header=0)\n",
    "df_tags = df_tags[:420]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>seen live</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>alternative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>pop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>indie</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           tag\n",
       "0    seen live\n",
       "1         rock\n",
       "2  alternative\n",
       "3          pop\n",
       "4        indie"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_tags.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>incorrect tag</th>\n",
       "      <th>tag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>abstract hip-hop</td>\n",
       "      <td>hip-hop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>alternative rap</td>\n",
       "      <td>alternative hip-hop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>avantgarde</td>\n",
       "      <td>avant-garde</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>brazilian rock</td>\n",
       "      <td>rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>chanson francaise</td>\n",
       "      <td>chanson</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       incorrect tag                  tag\n",
       "0   abstract hip-hop              hip-hop\n",
       "1    alternative rap  alternative hip-hop\n",
       "2         avantgarde          avant-garde\n",
       "3     brazilian rock                 rock\n",
       "4  chanson francaise              chanson"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_tags_replace.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>invalid tag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>00s</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4ad</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>50s</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>60s</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>70s</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  invalid tag\n",
       "0         00s\n",
       "1         4ad\n",
       "2         50s\n",
       "3         60s\n",
       "4         70s"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_tags_delete.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ahora en lugar de los tags que queremos eliminar, queremos los tags válidos. Para ello eliminamos los tags incorrectos de la lista de tags."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "for index, tag in df_tags_delete.iterrows():\n",
    "    df_tags = df_tags.loc[~(df_tags == tag[0]).sum(axis=1).astype(bool)]\n",
    "#df_tags.to_csv(\"tags-v2.dat\", index=False)\n",
    "df_tags.to_csv(\"tags-v3.dat\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ya tenemos los tags válidos **df_tags** y los tags a reemplazar **df_tags_replace**. Ahora por fin vamos a realizar la limpieza, creando un nuevo dataframe que contenga únicamente los tags válidos seleccionados."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_tags = pd.read_csv('tags-v3.dat', header=0)\n",
    "df_tags = df_tags.sort_values('tag').reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8-bit</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>acid jazz</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>acoustic rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>aggrotech</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>alt-country</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>alternative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>alternative hip-hop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>alternative metal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>alternative rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>ambient</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>anime</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>art rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>atmospheric black metal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>avant-garde</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>avant-garde metal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>bebop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>black metal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>blues</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>blues rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>bossa nova</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>breakbeat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>breakcore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>britpop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>brutal death metal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>celtic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>chanson</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>chicago blues</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>chillout</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>christian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>christian metalcore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>christian rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>classic rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>classical</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>classical crossover</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>composer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>contemporary christian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>contemporary classical</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>contemporary country</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>country</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>country pop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>country rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>crunk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>dance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>dancehall</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>dark ambient</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>dark electro</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>darkwave</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>death metal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>deathcore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>deathrock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>disco</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>disney</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>doom metal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>dream pop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>drone</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>drum and bass</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>dub</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57</th>\n",
       "      <td>dubstep</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>east coast rap</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>easy listening</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>ebm</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61</th>\n",
       "      <td>electric blues</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>electro house</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63</th>\n",
       "      <td>electro-industrial</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>electroclash</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>electronic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66</th>\n",
       "      <td>electropop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>emo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>ethereal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69</th>\n",
       "      <td>ethnic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>eurodance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>experimental</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72</th>\n",
       "      <td>experimental rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>folk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>74</th>\n",
       "      <td>folk metal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75</th>\n",
       "      <td>folk punk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>76</th>\n",
       "      <td>folk rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77</th>\n",
       "      <td>freak folk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>funk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>futurepop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80</th>\n",
       "      <td>gangsta rap</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>garage rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>82</th>\n",
       "      <td>glam metal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>glam rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84</th>\n",
       "      <td>glitch</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85</th>\n",
       "      <td>gospel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>86</th>\n",
       "      <td>gothic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87</th>\n",
       "      <td>gothic metal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88</th>\n",
       "      <td>gothic rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <td>grime</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90</th>\n",
       "      <td>grindcore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>91</th>\n",
       "      <td>groove metal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>92</th>\n",
       "      <td>grunge</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>93</th>\n",
       "      <td>hard rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94</th>\n",
       "      <td>hardcore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>hardcore punk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>heavy metal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>hip-hop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>horror punk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>house</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>idm</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>101</th>\n",
       "      <td>indie</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102</th>\n",
       "      <td>indie folk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>103</th>\n",
       "      <td>indie pop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>104</th>\n",
       "      <td>indie rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>105</th>\n",
       "      <td>indietronica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>106</th>\n",
       "      <td>industrial</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>107</th>\n",
       "      <td>industrial metal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>108</th>\n",
       "      <td>industrial rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>109</th>\n",
       "      <td>instrumental</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110</th>\n",
       "      <td>j-pop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>111</th>\n",
       "      <td>j-rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>112</th>\n",
       "      <td>jazz</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>113</th>\n",
       "      <td>jazz hop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>114</th>\n",
       "      <td>k-pop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>115</th>\n",
       "      <td>latin pop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>116</th>\n",
       "      <td>latin rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>117</th>\n",
       "      <td>lo-fi</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>118</th>\n",
       "      <td>lounge</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>119</th>\n",
       "      <td>math rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>120</th>\n",
       "      <td>mathcore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>121</th>\n",
       "      <td>medieval</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>122</th>\n",
       "      <td>melodic death metal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>123</th>\n",
       "      <td>melodic hardcore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124</th>\n",
       "      <td>melodic metalcore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>125</th>\n",
       "      <td>metal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>126</th>\n",
       "      <td>metalcore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>127</th>\n",
       "      <td>minimal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>128</th>\n",
       "      <td>minimal techno</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>129</th>\n",
       "      <td>minimalism</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>130</th>\n",
       "      <td>mpb</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>131</th>\n",
       "      <td>musical</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>132</th>\n",
       "      <td>neo soul</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>133</th>\n",
       "      <td>neoclassical</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>134</th>\n",
       "      <td>neofolk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>135</th>\n",
       "      <td>new age</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>136</th>\n",
       "      <td>new rave</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>137</th>\n",
       "      <td>new romantic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>138</th>\n",
       "      <td>new wave</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>139</th>\n",
       "      <td>noise</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>140</th>\n",
       "      <td>noise pop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>141</th>\n",
       "      <td>noise rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>142</th>\n",
       "      <td>noisecore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>143</th>\n",
       "      <td>nu jazz</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>144</th>\n",
       "      <td>nu metal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>145</th>\n",
       "      <td>opera</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146</th>\n",
       "      <td>orchestral</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>147</th>\n",
       "      <td>pagan metal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>148</th>\n",
       "      <td>piano rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>149</th>\n",
       "      <td>pop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>150</th>\n",
       "      <td>pop punk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>151</th>\n",
       "      <td>pop rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>152</th>\n",
       "      <td>post-grunge</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>153</th>\n",
       "      <td>post-hardcore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>154</th>\n",
       "      <td>post-metal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>155</th>\n",
       "      <td>post-punk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>156</th>\n",
       "      <td>post-rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>157</th>\n",
       "      <td>power metal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>158</th>\n",
       "      <td>power pop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>159</th>\n",
       "      <td>progressive metal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>160</th>\n",
       "      <td>progressive rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>161</th>\n",
       "      <td>progressive trance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>162</th>\n",
       "      <td>proto-punk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>163</th>\n",
       "      <td>psybient</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>164</th>\n",
       "      <td>psychedelic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>165</th>\n",
       "      <td>psychedelic rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>166</th>\n",
       "      <td>psychobilly</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>167</th>\n",
       "      <td>psytrance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>168</th>\n",
       "      <td>punk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>169</th>\n",
       "      <td>ragga</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>170</th>\n",
       "      <td>rap</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>171</th>\n",
       "      <td>rapcore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>172</th>\n",
       "      <td>reggae</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>173</th>\n",
       "      <td>reggaeton</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>174</th>\n",
       "      <td>rnb</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>175</th>\n",
       "      <td>rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>176</th>\n",
       "      <td>rock n roll</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>177</th>\n",
       "      <td>rockabilly</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178</th>\n",
       "      <td>roots reggae</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>179</th>\n",
       "      <td>samba</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>180</th>\n",
       "      <td>screamo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>181</th>\n",
       "      <td>shoegaze</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>182</th>\n",
       "      <td>ska</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>183</th>\n",
       "      <td>ska-punk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>184</th>\n",
       "      <td>sleaze rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>185</th>\n",
       "      <td>sludge metal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>186</th>\n",
       "      <td>soft rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>187</th>\n",
       "      <td>soul</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>188</th>\n",
       "      <td>soundtrack</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>189</th>\n",
       "      <td>southern rap</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>190</th>\n",
       "      <td>southern rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>191</th>\n",
       "      <td>space rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>192</th>\n",
       "      <td>speed metal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>193</th>\n",
       "      <td>stoner</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>194</th>\n",
       "      <td>surf rock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195</th>\n",
       "      <td>swing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>196</th>\n",
       "      <td>symphonic black metal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>197</th>\n",
       "      <td>symphonic metal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>198</th>\n",
       "      <td>synth pop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>199</th>\n",
       "      <td>technical death metal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200</th>\n",
       "      <td>techno</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>201</th>\n",
       "      <td>teen pop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>202</th>\n",
       "      <td>thrash metal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>203</th>\n",
       "      <td>trance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>204</th>\n",
       "      <td>trancecore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>205</th>\n",
       "      <td>trip-hop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>206</th>\n",
       "      <td>twee</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>207</th>\n",
       "      <td>underground hip-hop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>208</th>\n",
       "      <td>urban</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>209</th>\n",
       "      <td>viking metal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>210</th>\n",
       "      <td>vocal trance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>211</th>\n",
       "      <td>west coast</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>212</th>\n",
       "      <td>world</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>213</th>\n",
       "      <td>worship</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         tag\n",
       "0                      8-bit\n",
       "1                  acid jazz\n",
       "2              acoustic rock\n",
       "3                  aggrotech\n",
       "4                alt-country\n",
       "5                alternative\n",
       "6        alternative hip-hop\n",
       "7          alternative metal\n",
       "8           alternative rock\n",
       "9                    ambient\n",
       "10                     anime\n",
       "11                  art rock\n",
       "12   atmospheric black metal\n",
       "13               avant-garde\n",
       "14         avant-garde metal\n",
       "15                     bebop\n",
       "16               black metal\n",
       "17                     blues\n",
       "18                blues rock\n",
       "19                bossa nova\n",
       "20                 breakbeat\n",
       "21                 breakcore\n",
       "22                   britpop\n",
       "23        brutal death metal\n",
       "24                    celtic\n",
       "25                   chanson\n",
       "26             chicago blues\n",
       "27                  chillout\n",
       "28                 christian\n",
       "29       christian metalcore\n",
       "30            christian rock\n",
       "31              classic rock\n",
       "32                 classical\n",
       "33       classical crossover\n",
       "34                  composer\n",
       "35    contemporary christian\n",
       "36    contemporary classical\n",
       "37      contemporary country\n",
       "38                   country\n",
       "39               country pop\n",
       "40              country rock\n",
       "41                     crunk\n",
       "42                     dance\n",
       "43                 dancehall\n",
       "44              dark ambient\n",
       "45              dark electro\n",
       "46                  darkwave\n",
       "47               death metal\n",
       "48                 deathcore\n",
       "49                 deathrock\n",
       "50                     disco\n",
       "51                    disney\n",
       "52                doom metal\n",
       "53                 dream pop\n",
       "54                     drone\n",
       "55             drum and bass\n",
       "56                       dub\n",
       "57                   dubstep\n",
       "58            east coast rap\n",
       "59            easy listening\n",
       "60                       ebm\n",
       "61            electric blues\n",
       "62             electro house\n",
       "63        electro-industrial\n",
       "64              electroclash\n",
       "65                electronic\n",
       "66                electropop\n",
       "67                       emo\n",
       "68                  ethereal\n",
       "69                    ethnic\n",
       "70                 eurodance\n",
       "71              experimental\n",
       "72         experimental rock\n",
       "73                      folk\n",
       "74                folk metal\n",
       "75                 folk punk\n",
       "76                 folk rock\n",
       "77                freak folk\n",
       "78                      funk\n",
       "79                 futurepop\n",
       "80               gangsta rap\n",
       "81               garage rock\n",
       "82                glam metal\n",
       "83                 glam rock\n",
       "84                    glitch\n",
       "85                    gospel\n",
       "86                    gothic\n",
       "87              gothic metal\n",
       "88               gothic rock\n",
       "89                     grime\n",
       "90                 grindcore\n",
       "91              groove metal\n",
       "92                    grunge\n",
       "93                 hard rock\n",
       "94                  hardcore\n",
       "95             hardcore punk\n",
       "96               heavy metal\n",
       "97                   hip-hop\n",
       "98               horror punk\n",
       "99                     house\n",
       "100                      idm\n",
       "101                    indie\n",
       "102               indie folk\n",
       "103                indie pop\n",
       "104               indie rock\n",
       "105             indietronica\n",
       "106               industrial\n",
       "107         industrial metal\n",
       "108          industrial rock\n",
       "109             instrumental\n",
       "110                    j-pop\n",
       "111                   j-rock\n",
       "112                     jazz\n",
       "113                 jazz hop\n",
       "114                    k-pop\n",
       "115                latin pop\n",
       "116               latin rock\n",
       "117                    lo-fi\n",
       "118                   lounge\n",
       "119                math rock\n",
       "120                 mathcore\n",
       "121                 medieval\n",
       "122      melodic death metal\n",
       "123         melodic hardcore\n",
       "124        melodic metalcore\n",
       "125                    metal\n",
       "126                metalcore\n",
       "127                  minimal\n",
       "128           minimal techno\n",
       "129               minimalism\n",
       "130                      mpb\n",
       "131                  musical\n",
       "132                 neo soul\n",
       "133             neoclassical\n",
       "134                  neofolk\n",
       "135                  new age\n",
       "136                 new rave\n",
       "137             new romantic\n",
       "138                 new wave\n",
       "139                    noise\n",
       "140                noise pop\n",
       "141               noise rock\n",
       "142                noisecore\n",
       "143                  nu jazz\n",
       "144                 nu metal\n",
       "145                    opera\n",
       "146               orchestral\n",
       "147              pagan metal\n",
       "148               piano rock\n",
       "149                      pop\n",
       "150                 pop punk\n",
       "151                 pop rock\n",
       "152              post-grunge\n",
       "153            post-hardcore\n",
       "154               post-metal\n",
       "155                post-punk\n",
       "156                post-rock\n",
       "157              power metal\n",
       "158                power pop\n",
       "159        progressive metal\n",
       "160         progressive rock\n",
       "161       progressive trance\n",
       "162               proto-punk\n",
       "163                 psybient\n",
       "164              psychedelic\n",
       "165         psychedelic rock\n",
       "166              psychobilly\n",
       "167                psytrance\n",
       "168                     punk\n",
       "169                    ragga\n",
       "170                      rap\n",
       "171                  rapcore\n",
       "172                   reggae\n",
       "173                reggaeton\n",
       "174                      rnb\n",
       "175                     rock\n",
       "176              rock n roll\n",
       "177               rockabilly\n",
       "178             roots reggae\n",
       "179                    samba\n",
       "180                  screamo\n",
       "181                 shoegaze\n",
       "182                      ska\n",
       "183                 ska-punk\n",
       "184              sleaze rock\n",
       "185             sludge metal\n",
       "186                soft rock\n",
       "187                     soul\n",
       "188               soundtrack\n",
       "189             southern rap\n",
       "190            southern rock\n",
       "191               space rock\n",
       "192              speed metal\n",
       "193                   stoner\n",
       "194                surf rock\n",
       "195                    swing\n",
       "196    symphonic black metal\n",
       "197          symphonic metal\n",
       "198                synth pop\n",
       "199    technical death metal\n",
       "200                   techno\n",
       "201                 teen pop\n",
       "202             thrash metal\n",
       "203                   trance\n",
       "204               trancecore\n",
       "205                 trip-hop\n",
       "206                     twee\n",
       "207      underground hip-hop\n",
       "208                    urban\n",
       "209             viking metal\n",
       "210             vocal trance\n",
       "211               west coast\n",
       "212                    world\n",
       "213                  worship"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_tags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>artistID</th>\n",
       "      <th>name</th>\n",
       "      <th>tag1</th>\n",
       "      <th>tag2</th>\n",
       "      <th>tag3</th>\n",
       "      <th>tag4</th>\n",
       "      <th>tag5</th>\n",
       "      <th>tag6</th>\n",
       "      <th>tag7</th>\n",
       "      <th>tag8</th>\n",
       "      <th>tag9</th>\n",
       "      <th>tag10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>89</td>\n",
       "      <td>lady gaga</td>\n",
       "      <td>pop</td>\n",
       "      <td>dance</td>\n",
       "      <td>electronic</td>\n",
       "      <td>female vocalists</td>\n",
       "      <td>female vocalist</td>\n",
       "      <td>lady gaga</td>\n",
       "      <td>electropop</td>\n",
       "      <td>seen live</td>\n",
       "      <td>american</td>\n",
       "      <td>gaga</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>289</td>\n",
       "      <td>britney spears</td>\n",
       "      <td>pop</td>\n",
       "      <td>dance</td>\n",
       "      <td>female vocalists</td>\n",
       "      <td>britney spears</td>\n",
       "      <td>legend</td>\n",
       "      <td>american</td>\n",
       "      <td>seen live</td>\n",
       "      <td>sexy</td>\n",
       "      <td>female</td>\n",
       "      <td>90s</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>288</td>\n",
       "      <td>rihanna</td>\n",
       "      <td>pop</td>\n",
       "      <td>rnb</td>\n",
       "      <td>female vocalists</td>\n",
       "      <td>dance</td>\n",
       "      <td>hip-hop</td>\n",
       "      <td>rihanna</td>\n",
       "      <td>r&amp;b</td>\n",
       "      <td>seen live</td>\n",
       "      <td>reggae</td>\n",
       "      <td>hip hop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>227</td>\n",
       "      <td>the beatles</td>\n",
       "      <td>classic rock</td>\n",
       "      <td>rock</td>\n",
       "      <td>british</td>\n",
       "      <td>60s</td>\n",
       "      <td>pop</td>\n",
       "      <td>psychedelic</td>\n",
       "      <td>the beatles</td>\n",
       "      <td>oldies</td>\n",
       "      <td>psychedelic rock</td>\n",
       "      <td>beatles</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>300</td>\n",
       "      <td>katy perry</td>\n",
       "      <td>pop</td>\n",
       "      <td>female vocalists</td>\n",
       "      <td>pop rock</td>\n",
       "      <td>indie</td>\n",
       "      <td>rock</td>\n",
       "      <td>katy perry</td>\n",
       "      <td>american</td>\n",
       "      <td>dance</td>\n",
       "      <td>seen live</td>\n",
       "      <td>female vocalist</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   artistID            name          tag1              tag2              tag3  \\\n",
       "0        89       lady gaga           pop             dance        electronic   \n",
       "1       289  britney spears           pop             dance  female vocalists   \n",
       "2       288         rihanna           pop               rnb  female vocalists   \n",
       "3       227     the beatles  classic rock              rock           british   \n",
       "4       300      katy perry           pop  female vocalists          pop rock   \n",
       "\n",
       "               tag4             tag5         tag6         tag7       tag8  \\\n",
       "0  female vocalists  female vocalist    lady gaga   electropop  seen live   \n",
       "1    britney spears           legend     american    seen live       sexy   \n",
       "2             dance          hip-hop      rihanna          r&b  seen live   \n",
       "3               60s              pop  psychedelic  the beatles     oldies   \n",
       "4             indie             rock   katy perry     american      dance   \n",
       "\n",
       "               tag9            tag10  \n",
       "0          american             gaga  \n",
       "1            female              90s  \n",
       "2            reggae          hip hop  \n",
       "3  psychedelic rock          beatles  \n",
       "4         seen live  female vocalist  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_artist_tags.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vamos a crear una lista de listas, donde cada lista será una fila del nuevo dataframe. Cada fila será una fila del dataframe df_artist_tags manteniendo únicamente los tags válidos y/o reemplazados."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/luna/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:3: UserWarning: DataFrame columns are not unique, some columns will be omitted.\n",
      "  This is separate from the ipykernel package so we can avoid doing imports until\n"
     ]
    }
   ],
   "source": [
    "valid_tags = df_tags.values\n",
    "replace_tags = df_tags_replace['incorrect tag'].values\n",
    "dictionary = df_tags_replace.set_index('incorrect tag').T.to_dict('records')[0]\n",
    "rows = []\n",
    "for index, data in df_artist_tags.iterrows():\n",
    "    artistID = data[0]\n",
    "    name = data[1]\n",
    "    tags = list(data[2:])\n",
    "    row = list(data[:2])\n",
    "    row_tags = []\n",
    "    valid_tag = None\n",
    "    for tag in tags:\n",
    "        if tag in replace_tags:\n",
    "            valid_tag = dictionary[tag]\n",
    "        elif tag in valid_tags:\n",
    "            valid_tag = tag\n",
    "        if (valid_tag is not None) and (valid_tag not in row_tags):\n",
    "            row_tags.append(valid_tag)\n",
    "    row = row+row_tags\n",
    "    rows.append(row)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Por último, escribimos en el fichero **artist_tags-v2.dat** las nuevas filas del dataframe. \n",
    "Hemos conseguido obtener un fichero con únicamente los artistas y tags válidos seleccionados."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"artist_tags-v2.dat\", \"w\") as f:\n",
    "    writer = csv.writer(f, delimiter='\\t')\n",
    "    writer.writerow(('artistID', 'name', 'tag1', 'tag2', 'tag3', 'tag4', 'tag5', 'tag6', 'tag7', 'tag8', 'tag9', 'tag10'))    \n",
    "\n",
    "with open(\"artist_tags-v2.dat\", \"a\") as f:\n",
    "    writer = csv.writer(f, delimiter='\\t')\n",
    "    writer.writerows(rows)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_artist_tags = pd.read_csv('artist_tags-v2.dat', sep = '\\t', header=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>artistID</th>\n",
       "      <th>name</th>\n",
       "      <th>tag1</th>\n",
       "      <th>tag2</th>\n",
       "      <th>tag3</th>\n",
       "      <th>tag4</th>\n",
       "      <th>tag5</th>\n",
       "      <th>tag6</th>\n",
       "      <th>tag7</th>\n",
       "      <th>tag8</th>\n",
       "      <th>tag9</th>\n",
       "      <th>tag10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>89</td>\n",
       "      <td>lady gaga</td>\n",
       "      <td>pop</td>\n",
       "      <td>dance</td>\n",
       "      <td>electronic</td>\n",
       "      <td>electropop</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>289</td>\n",
       "      <td>britney spears</td>\n",
       "      <td>pop</td>\n",
       "      <td>dance</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>288</td>\n",
       "      <td>rihanna</td>\n",
       "      <td>pop</td>\n",
       "      <td>rnb</td>\n",
       "      <td>dance</td>\n",
       "      <td>hip-hop</td>\n",
       "      <td>reggae</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>227</td>\n",
       "      <td>the beatles</td>\n",
       "      <td>classic rock</td>\n",
       "      <td>rock</td>\n",
       "      <td>pop</td>\n",
       "      <td>psychedelic</td>\n",
       "      <td>psychedelic rock</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>300</td>\n",
       "      <td>katy perry</td>\n",
       "      <td>pop</td>\n",
       "      <td>pop rock</td>\n",
       "      <td>indie</td>\n",
       "      <td>rock</td>\n",
       "      <td>dance</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   artistID            name          tag1      tag2        tag3         tag4  \\\n",
       "0        89       lady gaga           pop     dance  electronic   electropop   \n",
       "1       289  britney spears           pop     dance         NaN          NaN   \n",
       "2       288         rihanna           pop       rnb       dance      hip-hop   \n",
       "3       227     the beatles  classic rock      rock         pop  psychedelic   \n",
       "4       300      katy perry           pop  pop rock       indie         rock   \n",
       "\n",
       "               tag5 tag6 tag7 tag8 tag9 tag10  \n",
       "0               NaN  NaN  NaN  NaN  NaN   NaN  \n",
       "1               NaN  NaN  NaN  NaN  NaN   NaN  \n",
       "2            reggae  NaN  NaN  NaN  NaN   NaN  \n",
       "3  psychedelic rock  NaN  NaN  NaN  NaN   NaN  \n",
       "4             dance  NaN  NaN  NaN  NaN   NaN  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_artist_tags.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert list(df_artist_tags['artistID'][df_artist_tags['name'] == 'malice mizer'])[0] == 1\n",
    "assert list(df_artist_tags['artistID'][df_artist_tags['name'] == 'katatonia'])[0] == 370\n",
    "assert list(df_artist_tags['artistID'][df_artist_tags['name'] == 'eluveitie'])[0] == 837\n",
    "assert list(df_artist_tags['artistID'][df_artist_tags['name'] == 'windy & carl'])[0] == 6155\n",
    "assert list(df_artist_tags['artistID'][df_artist_tags['name'] == 'emilie autumn'])[0] == 5079"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
