{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# News Categorization using Multinomial Naive Bayes\n",
"## by [Andrés Soto](https://www.linkedin.com/in/andres-soto-villaverde-36198a5/)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Once upon a time, while searching by internet, I discovered [this site](https://www.kaggle.com/uciml/news-aggregator-dataset), where I found this challenge: \n",
"* Using the News Aggregator Data Set, can we predict the category (business, entertainment, etc.) of a news article given only its headline? \n",
"So I decided to try to do it using the Multinomial Naive Bayes method."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The News Aggregator Data Set comes from the UCI Machine Learning Repository. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Lichman, M. (2013). [UCI Machine Learning Repository](http://archive.ics.uci.edu/ml). Irvine, CA: University of California, School of Information and Computer Science. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This specific dataset can be found in the UCI ML Repository at [this URL](http://archive.ics.uci.edu/ml/datasets/News+Aggregator)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This dataset contains headlines, URLs, and categories for 422,937 news stories collected by a web aggregator between March 10th, 2014 and August 10th, 2014. News categories in this dataset are labelled:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Label | Category\t| News \n",
"-------|------------|----------\n",
"b\t| business\t| 115967
\n",
"t\t| science and technology\t| 108344
\n",
"e\t| entertainment\t| 152469
\n",
"m\t| health\t| 45639
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Multinomial Naive Bayes method will be used to predict the category (business, entertainment, etc.) of a news article given only its headline. The paper is divided in four sections. The first section is dedicated to importing the data set and getting some preliminary information about it. Second section explains how to divide data in two sets: the training set and the test set. Section number 3 is about training and testing the classification algorithm and obtaining results. Results analysis constitute the last section. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Import data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To import the data from the CSV file, we will use [Pandas library](http://pandas.pydata.org/) which also offers data structures and operations for manipulating data tables. Therefore, we need to import Pandas library. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To embed plots inside the Notebook, we use the \"%matplotlib inline\" [magic command](http://ipython.readthedocs.io/en/stable/interactive/magics.html#)."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"%matplotlib inline\n",
"import pandas as pd "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, we have to initialize some variables that will be used. They will be used to collect the news titles, its categories, as well as a list of the different possible categories (without repetitions)."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"titles = [] # list of news titles\n",
"categories = [] # list of news categories\n",
"labels = [] # list of different categories (without repetitions)\n",
"nlabels = 4 # number of different categories\n",
"lnews = [] # list of dictionaries with two fields: one for the news and \n",
" # the other for its category"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The code for this section will be organized in two functions: one which imports the data and the other which counts the news in each category, its percentage and plots it. "
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"def import_data():\n",
" global titles, labels, categories\n",
" # importing news aggregator data via Pandas (Python Data Analysis Library)\n",
" news = pd.read_csv(\"uci-news-aggregator.csv\")\n",
" # function 'head' shows the first 5 items in a column (or\n",
" # the first 5 rows in the DataFrame)\n",
" print(news.head())\n",
" categories = news['CATEGORY']\n",
" titles = news['TITLE']\n",
" labels = sorted(list(set(categories))) "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's see how long it takes to import the data by %time [magic command](https://ipython.org/ipython-doc/3/interactive/magics.html)."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" ID TITLE \\\n",
"0 1 Fed official says weak data caused by weather,... \n",
"1 2 Fed's Charles Plosser sees high bar for change... \n",
"2 3 US open: Stocks fall after Fed official hints ... \n",
"3 4 Fed risks falling 'behind the curve', Charles ... \n",
"4 5 Fed's Plosser: Nasty Weather Has Curbed Job Gr... \n",
"\n",
" URL PUBLISHER \\\n",
"0 http://www.latimes.com/business/money/la-fi-mo... Los Angeles Times \n",
"1 http://www.livemint.com/Politics/H2EvwJSK2VE6O... Livemint \n",
"2 http://www.ifamagazine.com/news/us-open-stocks... IFA Magazine \n",
"3 http://www.ifamagazine.com/news/fed-risks-fall... IFA Magazine \n",
"4 http://www.moneynews.com/Economy/federal-reser... Moneynews \n",
"\n",
" CATEGORY STORY HOSTNAME TIMESTAMP \n",
"0 b ddUyU0VZz0BRneMioxUPQVP6sIxvM www.latimes.com 1394470370698 \n",
"1 b ddUyU0VZz0BRneMioxUPQVP6sIxvM www.livemint.com 1394470371207 \n",
"2 b ddUyU0VZz0BRneMioxUPQVP6sIxvM www.ifamagazine.com 1394470371550 \n",
"3 b ddUyU0VZz0BRneMioxUPQVP6sIxvM www.ifamagazine.com 1394470371793 \n",
"4 b ddUyU0VZz0BRneMioxUPQVP6sIxvM www.moneynews.com 1394470372027 \n",
"Wall time: 7.72 s\n"
]
}
],
"source": [
"%time import_data()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The time to import the dat was 3.54 seconds. Let's analyze how many news we have from the different categories and its percentage. We will use the [class Counter](https://docs.python.org/3/library/collections.html#counter-objects) from the collections library, which keeps track of how many values contains a collection. Then we will tabulate the different categories and its percentage via a DataFrame."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" category news percent\n",
"0 b 115967 0.274531\n",
"1 e 152469 0.360943\n",
"2 m 45639 0.108042\n",
"3 t 108344 0.256485\n",
"total \t 422419\n"
]
}
],
"source": [
"from collections import Counter\n",
"\n",
"def count_data(labels,categories): \n",
" c = Counter(categories)\n",
" cont = dict(c)\n",
" # total number of news\n",
" tot = sum(list(cont.values())) \n",
" d = {\n",
" \"category\" : labels,\n",
" \"news\" : [cont[l] for l in labels],\n",
" \"percent\" : [cont[l]/tot for l in labels]\n",
" }\n",
" \n",
" print(pd.DataFrame(d)) \n",
" print(\"total \\t\",tot) \n",
" \n",
" return cont\n",
"\n",
"cont = count_data(labels,categories)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's show a [pie plot](http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html#sphx-glr-auto-examples-model-selection-plot-confusion-matrix-py) with the proportion of news by category."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAAD8CAYAAADUv3dIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4HNXVx/Hv2aLdVZfcewPLGGNjisHB4BAwHYEpoYWY\nFiC09IQUEhISEieUlx46hOIASQg1hE7AgGkGGxeBjXuVZSRLq7blvn/ckbUWki3L0s6W83meebbv\nnl3Lv7lz584dMcaglFIqOTxuF6CUUtlEQ1cppZJIQ1cppZJIQ1cppZJIQ1cppZJIQ1cppZJIQzdL\nichrInJeN77fHSLyy+56v3QnIkNEZIuIiNu1qNSioZvBRGS5iNQ7//nXicj9IpK7k+8xTETiIuJJ\nuG+GiLyZ+DxjzHeNMX/ortp3VXt1J5MxZpUxptDoQHjVhoZuZjPAscaYQmAfYD/gVzv5HuK8j7Rz\nXyprr+7kfLCIN9mfqdKHhm7mEwBjzDrgP8C4rzzB+pXTMl4vIg+ISIHz8BvOZbXTYj4QuAOYLCK1\nIrLZeY/7ReR3zvWpIrJKRH4oIhtEZI2InJPweaUi8oyI1IjIHBG5pm3LuU19U0Rktoh8KSIrROTb\nzv3HiMhHzvusEJHfJLysbd0HOK85T0QWikiViPxHRIYmfM4RIrLY+ZzbROT1li6YDn6jQuexllb1\neSKyAnilbUtbRApF5B4RWev8Nte0dD2IyCjns6pFZKOIzNrxP6tKVxq6WUJEhgDHAB+18/C5wLeB\nqcBIoAC4zXnsEOey0Nlcfhe4GHjHGFNgjCnt4CP7O+8zELgAuE1EipzHbgdqgb7AOcAMOmg5O6H4\nPHAT0BvYG/jYebgOONsYUwQcC1wsIuUd1D1HRE4ArgROBPoAbwKznM/pDTwB/AzoBVQAk3fwG93a\nptxDgDHAkc7txO/0INDsvHYiMM35XQCuAf5rjCkGBgO3tPdbqAxhjNElQxdgGbAF2OxcvwUIOI+9\nBpznXH8ZuDjhdaOxAeEBhgMxwJPw+Azgf20+637gd871qUC4zWs2AJOc92wGdkt47Jq275fw2JXA\nPzv5fW8ErneuD2un7ueBcxNue5w6hwBnA7PbvN/KTv5GLZ81LOHxrZ8P9AMaW3575/HTgVec6w8C\nfwUGuf03o0vPL9rSzXwnGGNKjTEjjDGXG2Oa2nnOQGBFwu0VgA8bFl3tu60yxsQTbtcD+dgWphdY\nnfDYqu28zxBgaXsPiMgkEXnV2SSvBi7CtoY7Mgy4SUQ2O90iVdjvNwj7G7StI7HG7f1G7T0/0VDA\nD6xzPvtLbMj2cR7/CTac3xOR+SJy7na+g0pzGrqZrzM7ktZiA6nFMCCCbZ22F7q7shOtEohiN6Nb\nDNnO81cBu3Xw2KPAv7EtxGLgTlq/b3s1rgQuclZCpcaYEmNMvrFdJuvaqSOxxu39Ri06+l1WYVu6\nvRI+t9gYMx7AGLPRGHOhMWYQtuvmdhEZ2cF7qTSnoavA9mv+QESGi0g+8Afg705LtRKIA6MSnr8B\nGCwi/p39IOc9/wVcLSIhERmD7SvtyCPAYSJyioh4nZ1wE5zH8oEvjTEREZkEnJnwuvbqvhP4hYiM\nBRCRIhE5xXnsOWCciJQ7n3MZ27Zit/cbQfsrt5admOuBF4EbRaTA2Sk3UkQOceo4RUQGOa+pduqO\nt/N+KgNo6Ga27bVIEx+7D3gI+B92U74euALAGNOADZjZzqbxJOBVYAGwXkQ2dqGWy4FibOvyQWyL\ntb1uD4wxq7A7AH+M7ZueC4x3Hr4UuEZEarBD4R5LeN1X6jbG/Bv4E/B3pztiHnCU8/wq4FTgL8Am\n7A6xDxLq6vA3auf7tXfft4EcYKHzPZ7A7mwE2B+YIyJbsC33K4wxy9v7PVT6E2NSfbilynQi8ieg\nnzEmZfoyneFcq4EzjTFv7Oj5SnWWtnRV0olImYjs5VyfBJyP7XJwlTNOt0hEAkDLIc3vulmTyjw+\ntwtQWakAmCUiA7D9w38xxjzjck1gx+U+ih1psBA78qPdbg+lukq7F5RSKom0e0EppZJIQ1cppZJI\nQ1cppZJIQ1cppZJIQ1cppZJIQ1cppZJIQ1cppZJIQ1cppZJIQ1cppZJIQ1cppZJIQ1cppZJIQ1cp\npZJIQ1cppZJIQ1cppZJIQ1cppZJIQ1cppZJIQ1cppZJIQ1cppZJIQ1cppZJIQ1cppZJIQ1cppZJI\nQ1cppZLI53YBKruJiBcoBkqBXm0uiwAf+Pzg8YPHt/XSFx2CX9ZgTARjosTjUeLxluvNQDWwGfgy\nYdkMfGmMaXLhqyoFaOiqHiAiAvQHdreLDIK8gZDTHzx9IN4LosXQVACeEASaoSgCxTHoDfTxQn8/\nlOZAjoCXryzeK+H00yEUgnjcLrGYvYxEDDU1TdTURKmpiVFTA3V1HurrfTQ2BsTni+H31+Hz1eLx\nVANVxOOV1Nd/Rjy+BFjqLOuNMcadX1FlKtG/KdVVItILG6yjwVcGhRPBjIa6wRAwMLIJxvpgVC70\nlvYbsyXYIN1JgRA8/BD07r1zrzMGGhuhtnbbZcsW2LAhzooV9axcGWPjxhyamnwEg+vxepfS2Pgp\nzc2LaQ3kFcaY5p0vXGU7bemqHRKRAcBk8OwFRXuDjIHwUAj5YFgD7OGFvfJgtMBobA4XAwRdLbw9\nIrZ1HApB375tH/UA+Vtv1dfD2rVDnOXrrFzZyIoVTaxb56WmJiS5uV/i9y8mHH6NWGwO8L4xZmMS\nv41KQ9rSVdsQET8wAZgMxdMgOhlMAezXBAfmwRhva7D2BsSdQrva0u0u0Shs2ABLl8KiRVE++STM\n0qUhvN4t+HwfUlv7KvAe8KExptadIlUq0tDNciLSF5gMgYMhbxrUjoFBTTDVD1OD8DVsyLoUrh1x\nO3TbE4/D6tVQUQELFjQzb14Dq1blEghsQGQOdXWvA+8Dc7VrIntp6GYZERkNHA7Fh0PsIIgWwb5N\n8I18mOKBSdhBAykuFUO3PdEoLFsGixfDp582MH9+hMrKAKHQu9TW/gt4CVisO+yyh4ZuhhORADAV\n8k4Ez3TwFcJRAt8I2VbsGNJyuHa6hG57amrgww9hzpwG3n3X0NzciMfzIvX1TwIvGmOq3S5R9RwN\n3QzkdBmUQ8kZUP81GB2FU3PheI/trk2xroKuSOfQTWQMrFoFH3wAb75Zy4IFAYLB+dTVzcKYZ4wx\nn7ldoupeGroZQkSGgEyH4nOhcSxMi8E3Q3AkdodXhsmU0G2roQHmzoU332xg9mxDLFZDLPY4TU33\nGWPmuV2e2nUaumlMRIaB9wwoOAeiw+F44MwAHE4qjtbqVpkauomMgSVL4LXXojz/fDPR6HoaGm4n\nHn/EGLPe7fJU12jophlnSNexUPxDiEyC0wTOzIFDAL/b5SVPNoRuongc5s2D556r5803vfj971FX\ndzvwlDGmwe3yVOdp6KYJERkOwYvB810Y44fvh+AUIORyZS7JttBN1NAAb70FzzxTS0WFF5/vSerr\n7wLeMsbE3S5PbZ+GbgpzWrXHQ/GPIbovzPDCJV4Y63Zp7svm0E1UWQkvvxzn6afrqalpIBq9m0jk\nHmPMMrdLU+3T0E1BIjLSadVeDHv4bKv2ZLK2VdseDd1ttfT/Pv98My+8EMfrfZlw+LfGmA/cLk1t\nS0M3RYiIByiH4ishNhFmeOASH+zhdmmpSUO3Y/X18OyzcR55pJFY7FPC4V9jx//qf/YUoKHrMids\nT4DCG2FQP/hl0LZqM3z0wa7S0N2xaBRefRUeeKCOmpqN1Nf/GnjcGBNxu7RspqHrEjvnrJwIRTdB\n/37w5xw4jow4cCEZNHQ7zxh47z148ME6li1rpLn5WuLxu40xdW6Xlo00dJPMCdtyKLoZ+g2AmX4o\nR8N2J2nods3ixfDQQ2E+/NBgzG00N9/Q09NRikgRcKYx5o6e/Jx0oaGbJE7YHmfDts9AmJkDJ6Jh\n20UaurtmzRqYNauRl1+OE4//iUjkup4a72uHO/KMMWavnnj/dKOh28OcsD3Whm2vwbZlO520nGQm\nlWjodo81a+C22+qZOzdMY+MVwGPdvcNNRGZhN+cqgJeMMT/rzvdPNxq6PUjEMwWKHoDSIbZlexIa\ntt1EQ7d7ffwx3HBDmKqqL6iv/44xZk53vbU9XJ1njDHju+s901lWJICIDBOR+cn7PH+JSK+noOQ1\nuG0UfJ5jjx7Lip9bpaO994YHHsjj8svHUVDwmuTl/ctOoqS6WzalQI836UVERIoug+BamH4sfOGD\nM8mun1mlLY8HjjpKeOyxENOnH0cgUCGBwLUikr/jF6vOyqY08IvIwyKyUEQeF5FuHQgrUjQGei+A\nfjfCi0G4x5sWZ2BQqq1QCC64wM+DD4Y48MDvEwyuFI9nht0/0SW1QEF3lpjOsil0y4BbjTFjsX8E\nl3THm4oU+EV63Qyx+fCDMbDAB5O7462Vcle/fvDb34a4/voShgy5jdzcN7rS5WCM2QzMFpF5IjKz\nBypNK1mxI83pyH/DGDPcuX0ocLkx5qRde9/iI8DzCIwvhvt8MLIbqlWdojvSkisahUceiTJrVhOR\nyBXE4/frYcVdk00t3bZ/IF3+gxEpKBHp+zx4n4Pbe8NrGrgqs/l8MGOGj1tvzWPQoJvJzX1VRAa5\nXVY6yqbQHSYiBzjXzwTe6sqbiPQ9CWQ1HHMELPHB6egBDipr7LYb3HtvHiedNIVAYJGI7NLWYjbK\nptBdDFwqIguBYmCnDkkUOSQkMvARaH4cZuXCA14o6ZFClUppfj+cf76PG24ooLT0IcnNfVBE8twu\nK11kRZ/urhLZcxxUPQeDBsK/faDDF12nfbqpIRyG669v4N13N9HQcIIxZq7bJaW6bGrp7jSRchEZ\neRGs/gDOGgzvauAqlSgvD3796xA/+MFggsHZ4vN9x+2SUp2GbgdEykPwyYNQdRs8EoDrPVl14kel\ndsa0acJdd4UoLf0/CYX+z5knWrVDf5h2iEwdBh/Nsac3/8Br57lVSm3XkCFw9925DB9+Abm5z2s/\nb/s0dNsQmXgoLPwQxu0BH/tgd7dLUip9FBXBzTfnMXnyVEKhD0RkoNslpRoNXYdIuUdkr0th6X/g\nohJ43geFbpelVPrx++GXvwxyxhm7EQzOE5EJbpeUSjR0AZHyXFj1B1h+I/w1AL/36E+j1C4QgbPP\n9vHTn5YSCLwtIse6XVKqyPpkESkvhjU3w5IfwYN+e9yEUqpbHHqocMMNueTnPyE5OT/YhUlzMkZW\nh65IeSmsvhk+mwGz/HaScaVUtxo7Fu6+O0Tv3r8nFLol24M3a0NXpLw3rLwNPj8D/uHTEQpK9aD+\n/eGuu3Lp0+ccgsE/uV2Om7IydEXK+8CK22HpKfCUD450uySlMl9+Ptx0Ux7FxZdJIPBLt8txS9aF\nrkh5P1h2J3wx3Y5Q+IbbJSmVPYqL4eabcyko+IXk5FzmdjluyKrQFSkfAEvvhhXHw4s+ONjtkpTK\nPn36wC235BIK/Vm83hlul5NsWRO6IuWDYfmdsOZoeFXP7qCUmwYMgJtvDhEK3SEez8lul5NMWRG6\nIuVDoWomLDsK/uOD/dwuSSk1bBjceGOIUOhvInKU2+UkS8aHrkh5f2i8ChaWw/XawlUqley+O8yc\nmUsw+E8ROcTtcpIho0PXHmkWvwLmHw2nBuHCrB4fqFRKGjcOfv/7XAKB50RkuNvl9LSMDV2Rcg9w\nHiw8Bob0hVt9bteklOrAvvvCOeeECIWeEZEct8vpSRkbusCxsOxkCI+FZ/2Q0f+OSqW/007zsuee\nIwkGr3e7lJ6Uka0/kfIJsOkCWHYQvOGHvm6XpLJZZSX88Y+weTN4PHDccXDSSfDgg/Dss1DinGvv\nggtg0qSvvr6uDq67DpYtsxPJ/PSn9tDau+6COXNsv+iVV9rnvvQSbNkCJ6fhgAARuOqqXGbMOE9E\nXjLGPO12ST0h40JXpHwgNPwQFh4Gt/lgX7dLUtnO64VLLrFn0m1ogIsugv2cETSnngrf/Ob2X3/r\nrXDAAXD11RCLQWOjPTfZ55/Dvfe2BvKgQfDf/8LMmT3+lXpMYaHt3/3Rjx4Wkb2MMSvcLqm7ZVT3\ngkh5PsS/D/OnwdlBmJFGO85WY4+O2xPYC7jFuf90YB9nGeFctmc4MAGYCCS2lq507j8n4b5HgJu7\np2y1Y6WlNnABQiEYOtS2fjsjHIb58+Hoo+1tr9eel8zjsQEMNoR9PnjsMZg+3T4nne25J8yYESI3\n99lM7N/NmNAVKfcCF0DFwTC0N9yYZn95PuAGYAHwDnAr9qzxfwc+cpaT6XgmNA/wOjAXeM+5b4tz\n+xPs+d0WAI3AA8Cl3f8V1I6tXw9LltjuAYAnn7TdCn/5i+1GaGvdOtv6mzkTLrzQtmqbmmx4T5oE\n3/mOPSNyXh4sXgwHHZTc79NTTjvNxx57jCQYvM7tUrpbxoQucDzUfQ3W7mvnxU23npP+wN7O9Xxg\nD2BNm+c8DpzRwesNEG9znweIONfrscF7HXA5kGbrpEzQ0AC/+Q1cdpkNzRNOgEcfhXvusa3h22//\n6mtiMduNcOKJtg83GLSvATj9dLj7brj4YrjvPjj3XHjuOfjtb+Hhh5P73bqbx2P7d4PB80XkeLfL\n6U4ZEboi5cOBE2HxeLjEYwMrnS0HPgYOSLjvTWwwj+rgNQJMA/YH7nbuyweOxnY5DMKefug9oLzb\nK1Y7EIvZwJ02DaZMsfcVF9udRwDHHgsVFV99XZ8+0LcvlJXZ21On2hBO1HJ78GB44w37OWvW2CWd\nFRXBNdfkEgg8LCKlbpfTXdI+dJ1uhbNhVTHER8LVad6EqwNOAW7ChmaLWXTcygWYje2CeB64DXjL\nuf8n2C6GPwNXAb8D7gVOA67tzsLV9sycaQ97PeWU1vs2b269/uabMHz4V19XWmqDd9Uqe/ujj+z7\nJLr/fjjvPIhGwRh7n8djuyHS3bhxcPjhOYRCf3a7lO6Sbtvg7ZkMsdGw/FB4wA/pfNbnKDZwzwZO\nSLg/BvwLG6odGeBc9gGmY1u0UxIen+tcjsbuXHsBOA9YSsetZ9Ut5s+HV16BESNsH6yI7cd9+WVY\nutTe7t8ffvhD+/yqKtt3+8c/2tuXXw5/+IMN1YED7ZCxFm+9ZVvBpU5DcNQoOP98ezlyZHK/Z0+5\n8MIgr79+pojcZoyZu+MXpDYxLWvGNCRSXgT8EeaPg1EHwkt+u5mdrr4N9MbuUEv0AjATeK2D19Vj\n+3PzgTBwBPAb57LF8dhuhxBwKvAicAHwPexoiTQTCMHDD9mdSCrzPfus4Y475lNfP9EY03bnRVpJ\n9+6F6VBTCpUHwl1pHrizsUO5XsX2we6DDVuAx/hq18I6Wk8xtAHbqp0IHIgN2MTAfQrb19sfKMIO\nIRsPNJGWgauyzzHHCP36jUTkLLdL2VVp29IVKd8NuArenwIXD4ffpPsKRO0Mbelmn3nz4Gc/q6Sx\ncagxptHtcroqLYNKpNwHnAPLSsA3BK5My++hlNoJ48fDuHG5eL1pPcg8XcPqEDCDYe0UuMMPAbfr\nUUolwyWX5OHzXS0iRW6X0lVpF7oi5SXAabAiAMV5cKzbJSmlkmXECDj4YC+BwM/dLqWr0i50sRMU\n+GDjFLgqJ713nimldtq3vx0CLknXeRnSKnRFyvOAI2G9gWg/ONPtkpRSyTZkSMsY5BPdLqUr0ip0\nseOhfLBmMvzIpxOTK5WlTjmlgPz8H7tdRlekTeiKlPuBcjvXXfVouEj7FZTKVlOmQDw+TkR2d7uU\nnZU2oYsd0V8ES/awWxUZM/+FUmpn5eTAscf6CATSbvhYWoSuSLkAx4Cpgdr94dJMmDNCKbUrysv9\nGHOeiKTVmNG0CF1gCDACNuSALwST3a5HKeW2wYNbzsgx3e1Sdka6hO7+QAzW7QVneXSYmFIKSMsd\naikfuiLlHuBgMJugdgKclebz5Sqlus2UKRCL7SkiQ90upbNSPnSxZ2MsgvXFkBvQs/sqpbby+2HC\nhCgw1e1SOisdQndfIAobdoPp2rWglNrWpEn55OYe6XYZnZXSoet0LUwBNkG0DKZp14JSalvjxwMc\n6nYZnZXSoYuddTvfnqdk84A02oJQSiXLiBEQj/cSkf5ul9IZqR66I+zF2kEwIgol7lajlEo9Hg+M\nHdsEHOJ2KZ2R6qE7EWiAqmFwpN/tYpRSKWrSpAJCoWlul9EZKRu6zqnV9wSqgSGwf8rWqpRy2YQJ\ngterobuL+mGnEYtAQz8Y53Y9SqlUtfvu0NQ0QER6uV3KjqRy6Pa1FxEvbMmHMe5Wo5RKXV4vjBzZ\ngD2NdkpL5dAdZC+qesPAqJ4HTSm1Xf36ebEjnlJaKofuKKAeanrBmPQ8T7xSKnn69AlguyVTWiqH\n7nAgDI0FMEwPilBKbV+vXn5ycga7XcaOpGToOmeJKAEaIZoPg3X+XKXU9pWUQCAwzO0ydiQlQxfI\nA+L2qrc4DbpplFJuKy0FkYFul7EjqRq6+YDTjxsv1NBVSu1QSQnE433dLmNHUjl0HbEgFLtXiVIq\nPZSUQCSS8idPTOXQdeZwNB7Q/WhKqR0oLobm5nwRSenASNUdVH5aQ1c0dJUVBSqAefbmJ59Ar152\nIuuWJSfHXvp8rdf9fjspispsPh/4fFEikXygxu1yOpKqoZvwP0RbutlhLfAxsAj4HFgB3uUGX6WB\nWiEeEaLGro5zMdIcEHPjjQjgBxMH4sZgjBGMwcTjkLiI2KOWfL7WS58P/H7jXLaEtCEnx14PBFou\nhUCg9TIx5NuGfWdv+/22JtW9YjEf0Oh2GduTqqEraEs3Q9QBnwALsK3U5SArDP61BqkRTKMQdQaq\n5AJ5YijGUGKEIoQChEKgwFnsX6yYeBO82kRoNnzNwJ0gIzuoIA7UG8OWaJTaaJRap6o6IAxSB9S3\nLlIPNDhLo3PZ5CyNQL3HYxq9XppETJPHQ8TjIQLERIiCxIEYdk9w3BgxLSsBY7ZdEXg87a8IfD7T\nJrC3XRHYlYFdAQSD0qXAb+8xny+9VwSxGMTjAjS7Xcr2pHLoOnwR+2evUksU+AyYj22dfgGsAN/K\nON7NQL0Qiwox7BHceWIoAooNlDhh2hKkhc5zBOxKthPnZPIAh0P9QfDa47DnMrgM+A3b7IXd+tT8\ndu7vsnhciMfpVJ0diAHheJwt8Ti1kQh1sHVlEAYJ28sOVwQtS1PL/V6vafJ4aPJ4TLMIzSJEPR6i\n9rMkhl35xI3BgJh4HNOyEjDGBpYxNvxblrYrgbZbBC1bAq1bBdvfItjZlUNOjq2js5qbweuNmGg0\npY9gTeXQdf6gvQ2w2dViss96YC7bbuqvMPg2GKgT4s12U98H5IqhgNYwLcSzTcs0D6ezyPRMEyoE\nsRlIbDXc9nfMfXXILcAZpPbZ9LzYdU1hd71hLCbEYrALXzsKbInFqI3FqKXjlYBzKS0rgfZWBPUi\ndkXgbA00ezxERIjarQFizhZBhyuClgXa3yKwKwGTENZxvF7weFI6cCF1Q7eZrQdHSL2GbnepZ9tN\n/WUgK8C3No6n2m7qx+J2uzgEFIih0NnUL05onRZim41+6LEw3VmDoeHHSMN7cOELcF0ccy/IRLfr\nSiM+oNRZdpkxEI12bqtlO5pJWBE0N3+layhha8C7Ephl599Oaakaug1sPTiCOg3dHYljQzRxU3+5\nwbfK4K2i3U39QgzFRra2TltapgVAkJ3b1E8lkyC8D3z8FBw0H04Fcx1IH7frUl2SA/R2lh15C/gX\nbOjZinZdqoZuI61HpNVBpSHd/vN3m420bup/xta9+t6NBqlts6mPoUCgyEApQmGbftNtN/Uz9/f0\ngTkZafgG/P0R+Ocm+D2Yy0BS9Q9e7bpKQOx/mJSWqn+DCXvOQtWwKIJd6WWQemzL9FNaN/VXgm+1\ns6nfZPvoWjb1853WaamzVz8xUAtwNvWR1g0ERQk0X4Y0L4Zf/QtuaoZ7gMPcrkv1iA1AFNa4XceO\npGro1rO1JVZYBYvSKEni2J1P84GFtO7VXxXHuwm7qR/ZdlO/AEOJgWI8Wzf1W3ZGpfOmfqoYA+Er\nkfArUD4bDgFzB8hwl8tS3WsuNNbAB27XsSOpGrotneECJVWwyGdbcG5nzibspv4Cttmr713/1U39\nlh1RRcZQioeiNv2m+WTHpn6q8ADT7BCzlx+DsSvge2CuAsl1uzbVLd61Ayfmul3HjogxqdmIFCmf\n6VxtgFd+Dktzem62sUZaN/UX07qpvyaO50vBNNohOW039Vv36rPNXv0M6wjJSKshdxbxUBjPbcA3\n0TVfOosBeRBpgr7GmJQewZCqLV2A1cBooAEKq2Fh350P3TiwFHus/iLnesumfiXbbOrnkLhXH0oS\nNvVblhC6qZ8pBkP9T/DUvwvnvwh/cYaYTXC7LtUlSwE/VDemeOBCaofuCmBvoAq8y+HdPvCNhKDb\nROuY08+B5eBZYfA5m/qmSYgYOwo9l9ZN/ZKETf2WftM8nCONNUyzzoEQ3g8+ehImL4AzwPwZUv88\n3mobc4EcGwgpL5VDdw1bAzC0Gt81++G9FnsGH2dTP8hXN/Xb7tXfuqmvgao64ANzqh1i9sij8HgV\nXAvmuzrELG18BNEaO1Q35aXy39Qato5/6ruK2BIP3wSK0E191TN6QdPlSNNC+PmT8H8RuBf4utt1\nqR16G8Ix+MjtOjojlUN3I6Wf9SNQcwCeaA5VEqXQ+FL/BMsq7Y2F8Bjki5fg2HfgUDC3gaT8GQ+z\nVBPwod3ufc/tWjojZWd2NubpOCVLXiNv4xxyN83CbxawzO2qVNbwAEdC/U/gv0NgD+AqMG7Nd3c+\n0A8Yn3Dfl8ARQBlwJB3P2n0jMM557Vm0znt4JTABOCfhuY8AN3dX0UnyChCExcaYlD8EGFI4dAHw\nxt7HExcEQw4VLKDJ7ZJUlsmD6PlIw3lwQy5mGPAPkn/c37nAf9vc9yfgcOzxjN8A/tjO69YCt2C3\nu+dhZxL7O7AFu/PpE+zBjAuwAycfAC7t9up71t+hoQYedLuOzkrt0CWhbVvM56zHS62L1ajsNRTq\nf4qn8kg5mpshAAAUjklEQVQ4xwMHgpmfxI+fApS0ue8pYIZzfQbw7w5eG8NOxxjFHuo5EPsfP+I8\nXo8N3uuAy0mvUwZEgSftNJH/cruWzkr10F2DXSmH8BIlxGcscLskldUmQ/jn8P4ecABwIRi35sDb\nCFt3cfSn/ZleBgI/AoYCg7Dn1T4cewzP0cBE5/5CbIdoec+W3O3eAnywyhizwu1aOiulQ9dUmDjw\nOmCHTQb4mLnaxaBc5gdzGtJwOTxUihkO3A4m5nJZ7Q3hqca2iFdguxrqgEedx36C7WL4M3AV8Dvs\naI3TgGt7uthu8jg0heFvbtexM1I6dB0f0bLFU8wXbMbDl+4WpBQAvaDxCjy1p8BP/TAG+F8SP74f\nrZPHrgf6tvOcl4GR2InJvcBJwNttntMyWcFo4AngMWAJ9iivVBYHHoNYxHazp410CN1VQBWQh4cY\nQRayQOcvVClkHIR/jiw5wG6ynwBmVQ98jGHbHXjl2B1fYPcindDOa4YC79I6QfUr2JEYiX4NXIPt\n43VO14IH29ebymYDUagyxix2u5adkfKhayqMAV6jpYshl494j+jWvw6lUoEHONoOMXt+sB3GdTWY\n7joX+JnA17DT2A8F7scO+XoJ+1mvOLcB1gHHOdcnAadg+24nYIP3woT3fQrYH9snXOQ8Zzx27Ote\n3VR7T/kDhOtg5o6fmVpSdpaxRFImA4E/ACswwBou5Xh6f2WVrVSqWA65j2HyG5C/Aieih012pwpg\nItQ2QH9jTKo3yreR8i1dxzrs8LFSBAjxGq+n9rntVZYbDvU/QzZOg7MFDgKz0O2aMsgfoSEON6db\n4EKahK7TxfA0dtYFKGUR1URY6WpZSu3YQRD+Bcwpg/2A70Lqzz2Y4jYAjwNNcJPbtXRFWoSuYz52\nPsd8BEOIN3hj6/hupVKXH+JnIA2XwgMl9qi2O7EHLaiddxNEvDDLGFPpdi1dkTahaypMDHvQjd2h\n1ouPWYlhk6tlKdV5faDxe3i2nAQ/8mHGYvfAq84LA7dCtK79o57TQtqEruMD7EiWIF4i5PIOL2lr\nV6WZ8RD+BfLZ/nbCmpPBpPwpbFPEnRD3wP+MMUvcrqWr0ip0TYVpAp6lZRx4b2azjChpcwCgUg4P\ncCzU/wieGWgPTLimG4eYZaLNwNXQVGMPpktbaRW6jrewY71tazef53mGZh23q9JSAUQuROpnwJ9C\nMAK7xzj1B3Im36+gydi+3GTONdTt0mKcbltSJl/HTgO63Bm3exGH0Y99dSikSnNvQt6rsLfB3A2i\nQ9GthcB+EG6A4caYtN6Tk44tXbCt3bVAMQIU8TQvEUO3zVS6O9jOYvbOaNgXuAxMR5OTZwsDXADh\nKFyV7oELaRq6psJEgYexU4wKBazDz2Je11E4KgPkQPxMpOG7cG8RDAPugaztQXsIzKewJmLnY98p\nIjJMRBaJyP0iUiEiD4vIYSLylnN7v56oeXvSMnQdi4APaZlStJT/8iExVrtak1Ldpx80/gCpORG+\n78OMA/OO2zUlWRVwBTTUwlnGmGgX32YU8BdjTBl2MrgzjDFTsDvkftlNpXZa2oauc5Ta49iTrPsJ\nUEchT/E4EZ1xV2WUve0sZov2QQ4Hvglmnds1Jcnl0BiFvxljPtiFt1lmzNajsBdg5wcCe8BV0s83\nmrahC2AqzHrsaToGA9CLhcT5jP/Q1TWiUqnJC5RD/Q/h3wNgN+BaMJncvngIzNOwKQw/3cW3SvyZ\n4gm347hwRvS0Dl3HC8DntIzd7cszLKKJtJphU6lOKoTIRUj92XBt0A4xe87tmnrAIuC70BCGY40x\nu3pmxO2Nakr6iKe0D11np9o92HPrBfHRRDGP82+i1LlcnFI9ZRSEr0TWHQqnCUwFU+F2Td2kHjge\n6pvg+8aYed3wlqaD6+3d7nFpOU63PVImU7DzMy8DDOuZRin782386b9qUWo7msDzBCawBPkOmGtA\nCt2uaRd8Cxqfgmfr4JsmUwIqQSbF0WzsCU0HAtCXV9lIJS/rMDKV4QIQ/xbScDHcXWj3DN1Peg4x\newDMU7CxDs7NxMCFDGrpAkiZFGFP9xQFttBMLuv4LkeTx956tJrKEh9B3nOY4TG4D2SS2/V00gJg\nEtTXwwHGmE/drqenZFJLF1NharADqEuAIDnU05u/8bxOeK6yyD52FrMFE5GvA6eDWe92TTuwCTjG\n9uNensmBCxkWugCmwnyO3bE2EPCSRyVFPMGjRPXU7SpreIEToOEH8K/+9uiAmWBS8RxXdcA3oKEK\n/ho15j636+lpGdW90ELKRICTgeOB5YBhI5PxcCgX4SfganlKJd/nkPcPTHETcjf2VPGpoAmYBk1z\n4Yk6+Ham9uMmysjQBZAy8QKXAHsDqzDAek6kmLGcjZ8cd+tTyhWvQd7/YJLB3AWym4ulxICToPk1\neLUWjt+Fw3zTSsaGLoCUSS7wc6A3sAGDsI5TKWU3zsaP3+UC00UUuzs8ht0lPhb4uvPYHOB9bEfV\n7sC0dl7/FPAZkIddDbZ4CVgC9AemO/fNww7UPLA7v4DaRhN4H8fkLEW+C+ZqkIIkl2CA70D0cfio\nFqYakz3zt2d06AJImfQBfoE9eGITcTys55v0YSRn4U/+QYBpqhk7y0UcuBe7fRoB3gTOwvYhhrHB\n2tYK57VP0hq6jcATwNnYWbsPxO7+nAV8iwzc25CC1kFoFia4BbkR+0+RrJ/9FxC7FZbUwiRjzJYk\nfWxKyPg/bVNhKoE/Y+OiNx7i9OdxNrKcWUR0loZOaumOiWJ/ScGesW4KNnCh/cAFO3A01OY+ofV0\nuBHsX+LbwCSy4K8yRQyAhh8iXx4Pl3oxE2GXZpXprOshfjNsqIVDsi1wIUv+vE2FWQf8CbtV0wsP\ncQbwGOtZxWNE9PCJTogDfwWuw+4KH4Sdd28FcDfwALAzZ1cMYLsj/goUOLfXYCfeU8m1rx1iNm8C\nMhX4FpiNPfAxBvgVxH8DG8NwkDE98jEpL+O7FxJJmQwGrsRGyGZieFnPtxjAIE7TPt5OaQQew3Yv\n/AM748rR2MB8Avh+B6+rBh5l2z7dRE8D+wPrgKXYWZIP6baqVWdVQ86jGP9G5Gow3wPpjv8WMeBC\niD0BX9TCwcaYDd3wtmkpK1q6LUyFWQ3MxE7nVoKXGP15mPUs5T4ihF0uMB0EgeHYHWCFQMtJvAZh\nuwzqu/CeLZPD9sIelnQq9tSvm3elUNUlxdB8CRI+E67OsVNIvriLb9kInADRf8LcWtgvmwMXsix0\nAUyFWYUNXj/QGy8xBvA4YT7gTiL6H70dYdh6/rkItiXaG9sVsMy5fxN2+yG3g/fY3gbVa8Chzutb\nnifOZyl3jLazmK08GKYLHAHmiy68TTUwFSJvwss1MCUb+3DbyqruhURSJgOBHwL5gD1KciP708gR\nfAufMy26AtiAHXlgnGUcdtM/hh0Oth67M+1IbCu4FttdcJbz+n9gD1FpwO5sOxSY6Dy22Hn9153b\nL9I6jOyknvpCaqc0OEPMliGXgfk1SH4nXrYW+Do0r4eHa+E7xph0nIOn22Vt6AJImZQAVwBDgVUA\nbGY0WziFk/DrTh2lEqyF3EeJB+vw3AKcQcczgFdgA7cGrm2A32XDkWadldWhCyBlEgIuwra9VgBx\ntjCQzZzNIeRwEB6dn0ypBO9D3n8wu8ftLGYT2zz8FHA2NDfCpc3G3ONGiaks60MXQMrEB5wJHA6s\nBKI0UMQmvsUQijkZn87XoFSCKMhTmOB85FSIXweeUuDnEL8D6sIwLW7Me26XmYo0dB3OJDnHAKdh\nezHDxPCxkePwMJaz8DtnYVNKtfgSAo9ifJXI7hBfAYu+hEONMZVul5aqNHTbkDKZgB1NGgPs4O1K\nJhLmaI7Cx0REuxuUSrAceJSoL8oL0TjTs2Ximq7S0G2HlEk/4DLsqd1XAXHC9KGKMxlJHifo9JBK\nEQdex3jepiknypkNxjzpdknpQEO3A1ImQWxXw2HY4fsNxPCzkeOAPTgJPyNcLVEp92wC/kFUqlmZ\n08jURmNWu11SutDQ3Q6nn3d/4ALsVC+2u8EOKzuRcfg5UneyqSwSA97G8D9iePkzjVyl4293joZu\nJ0iZ9McOKxuBHfPdTIQgmziGOGOYjh83Z4NWKhk2Av8gSh1rgBNM2HzidknpSEO3k5xhZdOAU7AH\nxdq9s18yii2cRBk5HI3vK1MYKpXuYsCbxJlNHD83EORKU6XB0VUaujvJOXz4POxcIOuAJqLkUMlR\nRBnH8fjZg44P1VEqnawCniJKPSvxcKKpNfPdLindaeh2gXP+tUOB07HTsthZk6oZxhZOoBf5HIef\nAS4WqdSuqAH+S5QlxPBxLUGu0dZt99DQ3QVOX+852AkONwD1xPGwiX2o53DG4GUaPpJ9AiqluqoZ\neIs472AI8goBzjWVZq3bZWUSDd1d5LR6J2Hn/8jHdjlEiBCgikNpZF+m4GEyHp0kXaUsA8wHXiCG\nh2WEuIQSXjYVGhDdTUO3mzhnHj4COB6762EdYKinlGqOAYZyFH7GkoWzGKuUthz4D1G2UEeQX9OH\nv5oKo7MZ9xAN3W7mnH34ZGAysAV7JjGoZgR1HE0ORRxGDnui4avcY7Bh+zIRqoiSw/2U8nOzTCcZ\n72kauj1EymR37DTeI7DBuwUDVDOKMEfgo4TD8DOO1rPpKtXTDPAF8AoRNhMlyL8p4UrzhVnpdmnZ\nQkO3Bzn9vROxZ/3qhz3rlw3fGkZQxxF46cU38DMeDV/Vcwz2NEsvE6GaCCGeo4Rr8PGp9tsml4Zu\nEjjhuzf2wIoBwJfYQTlQzXDCTEPowxR87I0QdK9WlWGiwCLgTSLU0uyE7e/wslDD1h0auknkhO94\nbMt3IPa8fdUA1DCEMAfTxAjGIxyIlz7u1arSXC3wPnHeJ46XanJ4gRKuxctiDVt3aei6QMrEA+yF\nbfkOxp6ysRIwNFJINfvTwP70RziIHEajO93UjhlgNfA2ET7HQy7LyOdp8rkT+FzDNjVo6LrICd/R\n2KFme2P/22wAmonhZTN70sQUhGImO10PnTkNq8ouTcBCYDbN1BIjxHyKeJAAzwErNWxTi4ZuinCG\nmh2MnVQniO12sP2+WxhEmMk0UMZg4uxLDmVAjmvlKrfFsKMQ5hLhMzyEqCTIHIq4HQ/vmgod+pWq\nNHRTjDN5+kTs+doGYw/MrASiRPFTzRia2Z8mBlJGnInOZOra/ZD5DPaQm4+JMQ+DlzA5LKaQlwjw\nD2CBqdBT5aQ6Dd0U5UygPgrb+p0M+IB67LCzOE3kU804IuyHoZAJeBiPlwHoDGeZpgpYgOEjIjQQ\nJ8jnFDCPXJ4C3jEVetaGdKKhmwac1u9YYCowDhurNbR0P4TpQy0TaGY8QpAyhLH4GAE630MaimOn\nVFxEnIVEaQSCrCSPReTzDMJs4DNt1aYnDd00I2VShB12djgwFPtftBo7SAjC9KKWMmLsRRN9GEqM\nPZ0REDrbWepqApYAC4nwOYKPevwsJ4/l5PE2wqvY7oN6lytVu0hDN0053Q/9sP2/X8P2/xogjD34\nIk6EEDXsRoRxNDCSEmKMxs8IPAwBPbebi6LYEz8tx/A5zazDR4hKfCyjkFUE+Bh4B5hnKsxmd4tV\n3UlDN0NImZQCZcAB2C4ID3aC9SqgmThetjCUBkYSZ3ca6U0vooxyQngo6JFwPSgKrGHbkA1Qi4+1\nBFhLAavwMQd4HzumttbdglVP0dDNQFImIezphPbGzvWbi+0Hrsf2A0eI4aOWwTQwAsPuNNCXEqKM\nwsdQvAwEitGdcl1Vi23JrsGwlGbW4yNADT7WEGQD+azHRyXwNnYm2y90OsXsoKGb4ZxDjwcCw7Ah\nvCd2hK+H1hC2B2PUMYgGhmMYThMDAB/9iDEEPwPw0B/ohQ5PS2QnL4L1wHoMK2lmHR4iQIhNCBsJ\nUumEbBU2YD/FTqy4UQ9cyD4aulnGOQpuADaEx2MPRw5i27RRbBstTMuwtDADaGIgMJQI/YgQopQI\nffHQFx+9EEqBUjK7eyKC7Smvwg7a20iEjcTZhA8hSoBKhA3kUEMumwlQi1ADzKM1ZDdpyCoN3Szn\nhHBf7I64UcAYYIjzsAc7OqLOWaJECRCmL030JkYvoD8xetFIAX4MxcTojYe++CnBnsCoZQmRmt0V\nBnsISi2t33QLsJEolcT4Eg+NeAkQxsdmoBIvtQRoJEQ1OTRgjxFbhp3Tazl2FoQqDVnVloau+gop\nEz92ZER/7LC03WGbUb+CjaqGrYshThMFNFJKM6XE6I3QhzgFxMgjQog4XoJEySNOPlCEl0J8BLAd\nHttbOjPXcBzbIm1OWJrauV1HlC3E2AKEERrwYTDk0IiXMF5qMVQ7s3PVE6TJabnGsSuiMDZgP8eO\nqF0PVJoKE+vaL66yiYau6hRniFoRtle3FBvKQ7H9xf2wsRjHBnLLyInmbZYYHprJJ+IsUfKJkY8Q\ndGYRDgIBDDnEty5+YvgwnWgj20+O4iWChwhCs7M0Ao0YZ/HSiI8G/DQ7SxQvxglVk1D/OuyYgxXA\nJlo7F+q0Bau6SkNX7TKni6IIG8YlQCE2nHsl3FeMDWbjLGBjMnGJtbPs8OOxIelNWFp29Rm++nkt\ngfolNkA3Y8N0E7ZjoaXntlaDVfUEDV2VFE5LOYjt3S2AdjsV8rDD20LOZYDWwNyeJuxIjMSlbaeC\n7ViwS5MGqnKLhq5CRM4CrsD22c4BLjH6h6FUj9ARl1lORMYApwFfM8bsg+2XPcvdqpTKXD63C1Cu\nOwzYB3hfZGsXwAZ3S1Iqc2noKgEeNMb80u1ClMoG2r2gXgFOEZE+ACJSIiJDXa5JqYyloZvljDGL\ngF8BL4rIJ8CL2IMilFI9QEcvKKVUEmlLVymlkkhDVymlkkhDVymlkkhDVymlkkhDVymlkkhDVyml\nkkhDVymlkkhDVymlkkhDVymlkkhDVymlkkhDVymlkkhDVymlkkhDVymlkkhDVymlkkhDVymlkkhD\nVymlkkhDVymlkkhDVymlkkhDVymlkkhDVymlkuj/ATYGI9Qc+at7AAAAAElFTkSuQmCC\n",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import pylab as pl # useful for drawing graphics\n",
"\n",
"def categories_pie_plot(cont,tit):\n",
" global labels\n",
" sizes = [cont[l] for l in labels]\n",
" pl.pie(sizes, explode=(0, 0, 0, 0), labels=labels,\n",
" autopct='%1.1f%%', shadow=True, startangle=90)\n",
" pl.title(tit)\n",
" pl.show()\n",
" \n",
"categories_pie_plot(cont,\"Plotting categories\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As we can see, the entertainment (e) category is the biggest one, which is more than three times bigger than health (m) category. In second place we have business (b) and technology (t), which are more than two times bigger than health category."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Splitting the data "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we should split our data into two sets:\n",
"1. a training set (70%) used to discover potentially predictive relationships, and\n",
"2. a test set (30%) used to evaluate whether the discovered relationships hold and to assess the strength and utility of a predictive relationship."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Before splitting it, the data should be first permuted. [Shuffle](http://scikit-learn.org/stable/modules/generated/sklearn.utils.shuffle.html) is a method included in scikit-learn library which allows to do random permutations of collections. Then data could be splitted into a pair of train and test sets."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from sklearn.utils import shuffle # Shuffle arrays in a consistent way\n",
"\n",
"X_train = []\n",
"y_train = []\n",
"X_test = []\n",
"y_test = []\n",
"\n",
"def split_data():\n",
" global titles, categories\n",
" global X_train, y_train, X_test, y_test,labels\n",
" N = len(titles)\n",
" Ntrain = int(N * 0.7) \n",
" # Let's shuffle the data\n",
" titles, categories = shuffle(titles, categories, random_state=0)\n",
" X_train = titles[:Ntrain]\n",
" y_train = categories[:Ntrain]\n",
" X_test = titles[Ntrain:]\n",
" y_test = categories[Ntrain:]"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Wall time: 1.06 s\n"
]
}
],
"source": [
"%time split_data()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Time required to split data is 1.28 seconds. Now let's analyze the proportion of news categories in the training set."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" category news percent\n",
"0 b 81238 0.274738\n",
"1 e 106844 0.361334\n",
"2 m 31930 0.107984\n",
"3 t 75681 0.255945\n",
"total \t 295693\n"
]
}
],
"source": [
"cont2 = count_data(labels,y_train)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Percentage are very much close to the ones obtained for the whole data set. "
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAAD8CAYAAADUv3dIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd8W+XZ//HPpWFL3nEmCWQR4kASwgx7hU3A7F02LS2j\niw7o0zJ+fShQKJRNGWWW1T5QRqFlU0JZZYaQBMgme3vJtsb1++M+dhTHTpzE1pGs6/16nZe1dUmW\nvrrPfe5zH1FVjDHGZEbA7wKMMSafWOgaY0wGWegaY0wGWegaY0wGWegaY0wGWegaY0wGWeiaThOR\nWhEZ6ncdLUTkRRE5w8fnv0tE/qerb2t6NrFxupklIqcBPwFGATXAp8DvVPWdTtw3BYxQ1ZndW2Vm\niEgQ+AtwKPAucJKq1nnXXQ7EVPWP3fTcs4DzVPX17nj8bCciZwHnq+o+fteSb6ylm0Ei8lPgJuB/\ngX7AYOAO4KhOPoQvv5BeOHaH44Ak0Bv3A/Q97/mG4d6TW7vpeTeoG19zthB8+jzlPVW1JQMLUAbU\nAset5za7Av8BVgLzgduAkHfdW0AKqMMF1Ine5UcCn3j3mQSMTXu8nYCPgdXAU8ATwP9Lu/67wNfA\nMuDvwBZp16WAC4GvgBlplw33ThcANwJzgIXAnUChd11v4HmvpuXAWx283l8A3/VOXwDc7p1+Dti9\nE+/pG8C53umzgLeBG4AVwAzgsA7u9zAu7Ou99/JnwBDv9Z3rvaY3vds+5b2+lcCbwHZpj/NAy/sJ\n7AfMA34KLPb+f2dv4m0rvfdvNfA+8Fvg7Q5eSyHwiPc/XOndvm/aZ+4+YIH3fL/Fhe0oIAbEcZ/J\nFX5/P/JpsZZu5uyB+4L8fT23SQI/xn3p9gAm4IIPVd3Pu81YVS1T1b+KyI7A/bjwrAT+BDwnImER\nCQNPA3/2rnscOLbliURkAvA74ARgC2AuLpTTHQ2MB7bzzqe3jK4HRgDbe38HAVd4112K+5L3xrXo\nf9XB6/0CmCAiBcABwBQROQZYqqrvred96sh4YKr3vDfg3pt1qOqZuNd7pPde3ph29b64UDrUO/8i\nsLX3Oj7GdYd0ZABQCgwEzgfuEJHyTbjtnbgw7AecjftB6ahVehYuXAfh/s/fxwUqwENAMzAc2BE4\nGNelMM273buqWqqqlet5Taar+Z36+bIApwELNvI+PwL+L+18a0vTO38ncHWb+0wD9vGWeW2ue5s1\nra37gOvSrivGfUEHpz3Xfm3un97SrQOGpV23BzDTO3018AywdSde47XAZ8BduND4BBea1+Ba97fj\ntfbbuW/blu5XaddFcT9i/Tq47yxgQtr5Id7th6yn1grvPSj1zrdtvdYDgbTbLwbGb8xtcV1+zbi+\n+5brfgv8u4OazqHNGo53eT+gEW/tw7vsFOD1tPer3ce0pXsXa+lmznKgj4h0+J6LyDYi8ryILBSR\nVbjg6bOexxwCXCoiK7xlJbAlrvU0ELfamm5e2umBuNVoAFS13qtxUNptvu2gzr5AEfBRy3MDL+HC\nElwrcwbwsoh8IyK/7OgFqOrlqjpOVX8AXIYL3/HATupa94W4Vf7OWJT2uDHcqnRJJ+/bovU1i0hA\nRK7zXsMqXFArHf9PlqtqKu18w3qev6Pb9gWCrP3ep//f2noY+BfwhIh869UbxH02wsDCtM/G3eup\n3WSIhW7mvAs0Aces5zZ34VaPt1bVCuB/cMHRkXnANapa6S29VLVEVZ/E9UMOanP7rdJOL8B9MQEQ\nkWJcaKZ/2TtapV2GC4nRac9doarlAKpap6o/U9WtgWrgpyJywHpeByIyFthDVe8BxgIfeVd9iOvC\n6Godvbb0y0/DbdCb4P0/huL+H+v7n2yupUAC9+PZYqsObouqJlX1t6o6GtgTV++ZuM9GI9A77bNR\noaot76VtRPOJhW6GqGoNcCWu7+5oEYmKSEhEDheR67yblQI1qtogIqOAH7R5mEW4/rkW9wLfF5Hx\n4IJTRI7wAvRdICkiF4lIUERa+mdbPA6cIyLbi0ghrn/3PVVdX6uq5bWo99x/9Fq9iMggETnEOz1R\nRLb2bl6LC5FUuw+2xm3AJd7pWcDeXr/0fkB3DJFr+17CumFaivuhXOm9p9fSzWHltX6fBq7yPiOj\ncCHaLhHZX0TGeGtQdbiNY0lVXQS8DNwsIqXiDBeRfb27Lga29N5jk0EWuhmkqjfhtlj/GliC25hz\nIWs2rv0MOF1EanAbxdpu2LoKeNhbXTxBVT/CbUS73VvF/wrXV4eqxnFDss7HbdU+DbdFvMm7/jXg\nN7gv+HxgGK7Pr7Xc9l5C2ulfAt8A73mr3i8DI73rtgFeFZFa4B3gDlV9q6P3RUTOASar6qfeRU/j\nWupLgV7APR3cdUMBuL7rrwN+472XP+3g9g/j/kfzcRv9/rOB59vY+jq67SW4/uOFuI1hj+H939ox\nAPgbbqTDFFw/96PedWfiRpl8iRvR8Vfv9gCve7dfJCJLNqJOs5ls54g8IiLvAXep6kN+12I6z1sT\n6q+q5/hdi9l81tLtwURkXxHp73UvnIXrK/2n33WZ9RORKq+PG6/r6Dxc69/0ACG/CzDdqgo3uL8I\n1y96vKou9rck0wmlwOMisgWu7/UGVX3e55pMF7HuBWOMySDrXjDGmAyy0DXGmAyy0DXGmAyy0DXG\nmAyy0DXGmAyy0DXGmAyy0DXGmAyy0DXGmAyy0DXGmAyy0DXGmAyy0DXGmAyy0DXGmAyy0DXGmAyy\n0DXGmAyy0DXGmAyy0DXGmAyy0DXGmAyy0DXGmAyy0DXGmAyy0DXGmAyy0DXGmAyy0DXGmAwK+V2A\nyW8iEgQqgN5AZZu/ZUAIQgUQCHlLGAIhQomtCMt8VOOoJkilEqRSLaebgVXACmClt7SeVtUmH16q\nMYCFrukGIiLAAGAbYCTIICgaAIVbgPQF7Q2JCmgqBYlANA5lceiVdFnbLwj9w1BZAAUCQdZZgpfB\nKadANAqplFuSSfc3HldWr25i9eoEq1YlqamBuroADQ0hGhsLJRRKEg7XEQrVEgi4cE6lltDQ8BWp\n1DfADG9ZpKrqz7toeiqxz5TZVCLSGxgJbAOhKijbEXQk1G0JkRQMbYYxIRheBH2k/cZsBS5IN1Jh\nFB59BPr02bj7qUJjI9TWrr3U1MDixSnmzGlg7twkS5YU0NQUJBJZRDA4k8bGL2hungbMxAXybFVt\n3vjCTb6zlq7ZIBHZAtgdAmOhfAeQbaF+MERCMCQG2wVhTBGMCrjG7Ta4MCXqa+HtEXGt42gU+vVr\ne20AKGk919AACxYM9pb9mTs3xpw5zSxcGGT16qgUFa0kFJpGff3rpFLvAx+q6tIMvhqTg6yla9Yi\nImFgHLAH9DoI4ntAqgx2aYI9iqEq2Nq4pS8g/hS6qS3drpJIwOLFMGMGTJ2a4LPP6pkxI0owWEMo\n9F9qa98APgQ+UtUaf4o02chCN8+JSD9gDyjcB4oPgtptYVAT7BeG/SKwBy5ks2ygi9+h255UCr79\nFqZPhylTmvn88xjz5hVRWLgEkfepq3sT+AD4WFXjPldrfGKhm0e8DVwjgQOh4iBI7gWJcti5CSaU\nwF4B2A0o97nSTsjG0G1PIgGzZ8O0afDFFzEmT46zdGkBkcj71NY+DbwCTLMNdvnDQreHE5FCYD8o\nOhaCx0CoDA4XOCDqWrHbknWt2M7IldBtz+rV8NFH8N57MT74IEVTUxOBwCs0NDwNvKyqq/wu0XQf\nC90eyOsyqIZep0LDnlCVgBOK4KiA6671qR+2K+Vy6KZThXnzXAj/+981TJkSIRKZTF3d46g+r6pf\n+V2i6VoWuj2EiGwFchxUnA2N28HBSTgpCocCOR5M7ekpodtWLAaffAJvv93IO++kSCZXk0w+RVPT\nn1X1c7/LM5vPQjeHicgQCJ4KpWdBYhgcBZxWCAcBEb/L6149NXTTqcI338Drr8d58cU4yeQiYrE7\nSaX+oqqL/C7PbBoL3RzjDemaCBWXQnxXOFng1ALYDwj7XV7m5EPopkul4NNP4cUXG5g0KUg4/CF1\ndXcCz6pqg9/lmc6z0M0RIjIUIt+HwA9gVBh+HIUTyMb9DzIi30I3XSwGkybB88/XMm1aiHD4GRoa\n7gHeVtWU3+WZ9bPQzWJeq/YoqPgZJHaGM4NwYRBG+12a//I5dNMtWwavvpriuefqWbmyiWTyPuLx\ne1R1lt+lmfZZ6GYhERmW1qoNwU+icDx526ptj4Xu2lTd3nEvvtjESy8pweAr1Ndfraof+V2aWZuF\nbpYQkQBQDRW/hMROcJbXqt3O79Kyk4Vux+rr4YUXUjz2WCPJ5GTq66/Ejf+1L3sWsND1mRe2x0D5\nTTCwP/wq4vpqe/jog81lobth8Ti8/jo89FAdq1cvoaHhSuBJ2wXZXxa6PnG75MoxUH4LDOgPvy+A\nI+kROy5kgoVu56nCBx+48J01q4nm5t+RSt2rqrV+l5aPLHQzzAvbaii/FfoPgOsLoBoL241kobtp\npk+HRx6p58MPQeQumppuUNUl3fmUIlIOnKaqd3Xn8+QKC90M8cL2SBe2fQe6sD2anJz3IBtY6G6e\nBQvg8cebeOWVJKnUtcTjN6pqY3c8lRvuyPOqOrY7Hj/X2De+m4mIiAQmQsU3MPz/4L6hMK0AjsXe\nfuObgQPh0ksLuf/+Inba6TIikbkicrI3E11XuxYYLiIfi8j13fD4OcVaut1IJLA3lD8AlYPhugI3\n7MuCtktYS7drffop3HRTHcuXz6Sh4Xuq+n5XPbTbXZ3nVXX7rnrMXJYXCSAiQ0RkcuaeL9xLpPez\n0OsNuGMEfF0AJ5Inb7fJRTvsAA8+WMIll4yltPQNKS5+WkQG+11WT5RPKdDtTXrXlVB+MUQWwHET\nYWYITiO/3maTswIBOOww4cknoxx77JEUFk6TwsJrRaRkw3c2nZVPaRAWkUdF5EsReUpEunQgrEj5\nKOgzBfrfDC9H4N5gThyBwZi2olE4//wwDz8cZY89fkgkMk8CgbM2o7+3FijtyhJzWT6FbhVwu6pu\nh/sQXNgVDypSGhbpcwskJ8NPqmBKyB2RwZgc168fXHVVETfdVMFWW91BUdGbIrLlxj6Mqq4A3hGR\nz21DWp5sSPM68t9S1aHe+QOAS1T1uM173IqDIfAYjK2AB0IwvAuqNZ1iG9IyK5GARx+N88QTzcTj\nPySVesB2K940+dTSbfsB2eQPjEhpL5F+L0DgJbi9D7xpgWt6tlAIzj47zO23FzNo0K0UFb0uIoP8\nLisX5VPoDhGR3bzTpwGTNuVBRPodB/ItHHEYfBN0D2V7k5k8MWIE/PnPxRx//N4UFk4Vkc1aW8xH\n+RS604CLRORLoALYqF0SRfaNigx6DOJPwRNF8GAQKrulUGOyWigE554b4qabSqmsfESKih4WkWK/\ny8oVedGnu7lExoyFZS/AoIHw9xBs5XdJxvp0s0N9PfzhDzHee28Zsdgxqvqx3yVlu3xq6W40kWoR\nGX4BzPsQTt8S3rPANSZdcTFccUWUSy/dkmh0koRC5/ldUraz0O2ASHUUPnsIlt8BfymEPwTy6sCP\nxmyMAw8U7r47Sq9et0o0erM3T7Rph70x7RDZbwh8/D6EToWPgm6eW2PMeg0eDPfdV8SQId+lqOgF\nESnyu6RsZKHbhshO+8PUj2DstvBpCEb4XZIxuaO8HG69tZjddz+AaPQjERnod0nZxkLXI1IdEBl7\nMcz4J1xQAf8I2Z6LxmyCggL49a8jnHrqCCKRz0VkB79LyiYWurT03867BmbfBH8qhN8G7a0xZjOI\nwBlnhPjlLyuJRCaJyFF+l5Qt8j5ZRKorYP5t8M2l8EgYTvG7JGN6jv33F266qZjS0iekoODn3TRJ\nek7J69AVqa6Eb2+Fr86CJ8JwjN8lGdPzbLst3HNPEf36XUkkcku+B2/ehq5IdR+Yewd8fSr8Xwgm\n+l2SMT3XgAFw113F9OlzLoWF1/hdjp/yMnRFqvvCnDth5gnwbAgO8bskY3q+0lI3sqG8/EdSUHC5\n3+X4Je9CV6S6P8z+E8w61o1QmOB3Scbkj1694Pbbiygt/bWEwxf7XY4f8ip0Raq3gJn3wOyj4F8h\n2MfvkozJP337wm23FVFU9HsJBs/2u5xMy5vQFakeBLP+BPOOgNfs6A7G+GngQLjttijR6J0SCJzg\ndzmZlBehK1K9FSy/HmYf5lq4u/pdkjFm8GD44x+jRKMPi0jebMnu8aErUj0AGq+AL4+GP4Zgd79L\nMsa0GDECbrghSjT6lHcYrR6vR4euSHURpH4Ikw+Hk6NwXl6PDzQmK223Hfzv/xZRWPisiAz2u5zu\n1mNDV6Q6AJwLX06Ewf3gtqDfNRljOrDTTnDGGVGKip4XkR49h2qPDV3gCJh1PNRvC8+HbS5cY7Lc\nqaeGGDVqBJHIH/wupTuF/C6gO4hUj4Nl34VZe8FbYejnd0kmny1dCtdeCytWQCAARx4Jxx0HDz0E\nL7zgxq4CnH8+jB+/7v3r6uDGG2HWLDeRzC9+4VbJ77kH3n8fttkGLrvM3faVV6CmBo4/PnOvr6sE\nAnDFFUWcffZ5IvKqqj7nd0ndoceFrkj1QIj9BL48EO4Iwc5+l2TyXTAIF17oNhrFYnDBBbDLLu66\nE0+Ek05a//1vvx122w2uugqSSWhsdMcm+/pruP/+NYE8aBD8619w/fXd/pK6TXm569+99NJHRWR7\nVZ3td0ldrUd1L4hUl0DqRzD5EDgjAmfl0Iazb3F7x40GxgK3eZefAuzkLcO8v+0ZCowDdgTSW0uX\neZefnXbZX4Bbu6Zss2GVlS5wAaJRN1Rq6dLO3be+HiZPhsMPd+eDQXdcskDABTC4EA6F4Mkn4dhj\n3W1y2ejRcPbZRd7RJwr8Lqer9ZjQFakOAufB9P1gSB+4Occ+eSHgJmAK8C5wO+6o8U8AH3vL8cBx\nHdw/ALwJfAJ84F1W453/DNenPQVoBB4ELur6l2A2bNEi+OYb1z0A8MwzrlvhhhtcN0JbCxdCWZlr\nvX7ve65V29Tkwnv8ePjud90RkYuLYdo02GuvzL6e7nLyyUFGjx5GJHKz36V0tR4TusBRULcXLNgJ\nHg7nXs/JAKBlgv0SYFtgfpvbPAWc2sH9FUi1uSwAxL3TDbjgvRG4BMix36SeIBaDK6+Eiy92oXn0\n0fDYY3Dffa41fOed694nmXTdCMcc4/pwIxF3H4BTToF774Xvfx/+/Gc45xz4xz/g6qvh0Ucz+9q6\nmgj85jdFRKNni8ixfpfTlXpE6IpUDwWOhunbw0VBqPK7pM00G/gU2C3tsrdxwbx1B/cR4GDc3nb3\nepeVAIfjuhwGAWW4VnB1l1dsNiCZdIF78MGw997usooKFy4AEyfC9Onr3q9vX+jXD6q8z/R++7kQ\nTtdyfsst4a233PPMn++WXFZa2jJ+9wERqfC7nK6S86HrdSucCfMqIDkcrsrx11QHnADcggvNFo/T\ncSsX4B1cF8SLwB3AJO/yn+O6GH4P/Ab4f8D9wMnA77qycLM+118PQ4bACWnTDKxYseb022/D0KHr\n3q+y0gXvvHnu/Mcfu8dJ98ADcO65kEiAqrssEHDdELluu+1gwoRCIpHf+11KV8m1dfD27AnJbWD2\nBHgwDLl81OcELnDPAI5OuzwJPI0L1Y5s4f3tCxyLa9HunXb9J97fkbiNa/8EzgVm0HHr2XSJyZPh\ntddg2DDXByvi+nFffRVmzHDnBwyAn/7U3X75ctd3e+217vwll8A117hQHTjQDRlrMWmSawVXVrrz\nW28N553n/g4fntnX2V0uuCDCW299R0TuUtVPNnyH7Cba8suYg0Sqy4FrYfIY2Hp3eCXsVrNz1ZlA\nH9wGtXT/BK4H3ujgfg24/twSoB43KfuVrD05+1G4bococCLwMnA+8CPcaIkcUxiFRx9xG5FMz/fC\nC8pdd02moWFHVW278SKn5PiqOMfC6kpYujvck+OB+w5uKNfruD7YnXBhC/Ak63YtLASO9E4vxrVq\nd8RN6HMUawfus7i+3gFAOW4I2fZAEzkZuCb/HHGE0L//cOA0v0vZXDnb0hWpHgH8Gj7cBy4Ymvt9\nuWajWEs3/3z2GVx22VIaGweraqPf5WyqnAwqkeoQcBbM6gXBreCynHwdxpiNMG4cjB5dRDB4id+l\nbI5cDat9QbeCBfvAXWGI+F2PMSYTLr64mFDoChHp5XcpmyrnQlekuhdwMsyOQK/iNf2axpgeb+hQ\n2G+/EAUFP/e7lE2Vc6GLm6AgBEv3gt8U5PbGM2PMRjv11AgiF+bqvLs5Fboi1cXAobBIId5//TsL\nGGN6pKFDYciQAGsPZs8ZORW6uPFQIZi/J/wsBD1uAiJjTGecdFIpJSU52cWQM6ErUh0Gqt1cdytH\nwgXWr2BMvtpnH0iltheRbfwuZWPlTOjiRvSXwzfbwjEKlX7XY4zxS0EBTJwYpLAw5+YozYnQFakW\n4AjQ1VC7K1zcE+aMMMZsjqOPDqN6nojk1JjRnAhdYEtgGCwOQzgKe/hdjzHGb4MGwciRipvdP2fk\nSuiOB5KwaCycFrBhYsYYAE48Mec2qGV96IpUB4B9QJdB7Q5wuh3ywBjj7LknJBJVIjLI71I6K+tD\nF3c0xjJYVAHRAju6rzGmVSgEY8fGgf38LqWzciF0dwaSsHgbODZoXQvGmLWMH19KNHqo32V0VlaH\nrte1sDewDBJVcIh1LRhj1rbDDiBykN9ldFZWhy5u1u0Sd5ySFQNgX7/rMcZkm2HDIJnsLSJbbPjG\n/sv20B3m/iwYBEMTkLOzuRljukswCGPGNJMjrbJsD90dgQZYPgQOtR0ijDHtGz++JFf6dbM2dL1D\nq48GVgNbwW7Wn2uMad+4cUIgcLDfZXRG1oYu0B83jVgcGvrDGL/rMcZkqxEjIB7vJyL9/S5lQ7I9\ndAXiQagtgSq/6zHGZKtgEIYMieHWjrNaNofuFoDC8j4wMAGFftdjjMlmffsGcI21rJbNobs10ACr\n+8C2uXmceGNM5vTrV4hrrGW1bA7doUA9NJbAENuIZoxZv759CwiHt/S7jA3JytD1jhJRCTRCogQG\n2XAxY8z6VVZCJDLU7zI2JCtDFygBUu5koMLtmGaMMetRWQki1tLdRMWA14+rZTnQN26M8VtlJaRS\nWR8W2Rq6JWtOJiO2+68xZoN694bm5qw/eGK2hm4xrXM4agBsO5oxZgPKyiAeLxKRAr9LWZ9s3UBV\nyJqJc8VC1zgJYDrwuTv7ySeudRMOuyUUckeJDYXavyyQrW0M0yWCQQiFksTjEaDZ73I6kq2hK6xp\n6Ur2NshN11kCfAp8CXwFzIbgHCW4WJE6IdUsJNR9YotBmgvRW24hAIRBk7iNAKoqmkq5DQKp1NqL\niPtipi+hUMuirSHtAtudLyhoWYSCAqWgQCgslNbL175P++c7c1ko5OoxmyeRCAExv8tYn2wOXY9a\nSzenNQNfAJOBacAsYDaE56cIrBS0EZJJQYEoUCJKGUqlCuUIZQil0LqE3aNqqgnebCLyNuyucDfI\nNuupIgXEVKlLJNwC1AENQL1bJOadjrlFYkDj2os0AS1LowiNgYA2BgI0i2jc/SXhLUmQpAgpVVJe\nDaoqqoqqtv+jEAisCeCWH4Vg0P0ItPxAuJa7rhX8hYVCONzyo9ByfuN/DNZ3m2z/UUgmW04l/Cxj\nQ3IgdANJ95E32SUFfItb1f8S+BqYA8E5KUJLgXohFXet0wKgCKVUlF4KvQhQRoBSoAwXplG8/7qm\nreWsRwCYAA17wltPwbiZcAFwtfeQ7d282Fu6bPO2qvvBcF/2zTqOVApoVqUmmaQ+maQW9wPQQOuP\ngzR452Pe+ZYfhRitPwTSSNqPAtAYDLb+KDQHAsRFiIuQFCHR9kdBFQVRVVBFUyn3GlMpF2gtPwpt\nfxBa1hTaBnZBwZq1hZa/a9YS3I9DR63+9rqJNvSD0dwMgUBCE4ms3oM1B0I3FIMV/lWSlxqAz3Ct\n06+BmSBzlPB8RWoEbRQSKfdfigKlopSj9NIA5QRag7RlcQ0k8QK1a0UgeSYSWwR3P4Y+WIPcCJxD\nbnVKBYCIt3SpLvpRAGhSpbbNj0IdrWsH0vIjEaPjtYW11hSAWCCgTcFg649CmzWFtdYWFEh5awq0\nt7agihQWpjb3dXa3bA3dOK3jdKXeQrerpIAZuNbpVOAbYA6E5qUILgPqhWTCfdoLgWJvVb+XCr0Q\nSr1V/ZZQbU2ITrZOu9MAaPwp0vgp/Oh59A9JuB9kD1+L6lkKvaVPVz5oKiWkUtAFn58vgD2bmlZu\ndk3dLFtDN0brHmnUWeh2xirchqgvcK3T2RCYrYQWea3TZtc6DQBFeK1TVXqxbuu0BK+ZmAVhurF2\ngPqxyNQX4cCP4AjQW0AG+V2X6XargXAOhEW2hm7DmpOpOlii5NqXv8skcBugJuNapzOAuRCelyKw\nQiBGa+s0QsuGKLy+03Vbp60jGHMwUDsrCBwFsf3hucfhpQXwS9BfgHT56rvJGssAgeV+17Eh2Rq6\naUM+oqthepy0uOg5lgCfAFNwq/qz1zdMym2IqvBW9cvatE6LSG+dGoBSiH8Pic+C65+CO2JwF3As\nPfXXJr/NB+Iwx+86NiSbQ9f7XpQugy+zemvkutoOk2ppnbYOk3IbN9obJlXRpnVaQsswqe7ZEJUP\nhkHDL5GGd+DMV2GMoveCjPW7LtOlPoRYDXzgdx0bkq2hu4rWHSQql8N/Qi6h/M6clmFSn7L2hqg5\nKYIdDJMqE6UibZhUeut0Y4dJmc2zF9SPhw+eht2mwumg14H09rsu0yU+dK2dz/yuY0Pc8IssJFJ9\nvXcyBq9dDt8UdO+k8HW4lulk3K6ms0DmesOkVgvatGaYVBGuddoyTKqCNUHa0jrN8nHkeW85FD5G\nKrycwDWgF4JkawvEbFgSKIJ4M/RT1VV+17M+2fw5+xYYCcSgbBV82W/TQrdlmNRnrNkQNVsJzVOC\ny2l3mFQ5eK1Tt6qf3jrNpmFSZtP1hqZLCDR9Cb96Bm6Ow33AgX7XZTbJDKAAVjVleeBCdofubGAH\nYDkE58AEqacHAAAWc0lEQVT7feHANiGXPkzqK2AWBOYqoYWK1HqtU113mFQlAcrTdi8tw+2qZBui\n8s92UD8KqX8Nqt+BfUDvBBnud11mo3wGFLggyHrZHLoLaG1JFs2BP+xM6E9KcKUbJpVICCnWDJMq\nV6UCabd1mg/DpMymCwAHQ8Pe8OqTMGY2XAR6JUjJhu5rssLHkKyBd/yuozOyOXTn07pXWt/58HWA\nw1asGW9ahtsQlauD+E32iULybCQ2H+54Ar2/FrkZOIPc2qU4H70L9Qk3/jLrZfNnaQllcyrp99mR\nbDHrQCKSZEtcL+8WpHUHGNPFBkHsUgIrj4SLgug40GwYh3QebrKe7dMuWwkcAlQBh+L2ymrPzcAY\n776ns2ay2cuAccDZabf9C3BrVxWdATHgfbfO+2+/a+mMrI0t1edS9Jn+AaULvqFoycuE9Qtm+V2V\nySu7QP2vkC92QPYHTgFd6GM55wD/anPZdcBBuPE2E4Br27nfAuA24GPcrBsJ4AmgBtc0/Aw3FHwK\nbhKaB4GLurz67vMKEIUpqrrM71o6I2tDF4BQ0zsEEw0Ek7UUMJ0pNPldkskzQeAYiP0Enh4AI4Br\nQf34IO7NukcLfBY4yzt9FvD3Du6bxM0GlsDtYz8Q9+WPe9c34IL3RuAScmvE41MQWwUP+V1HZ2V3\n6JLWtq3gaxYRpNbHakz+Kof495GGM+CaCAwDnqd1o4NvlrBmfuAB3vm2BgKXAoOBQUAFrnVcAhwO\n7OhdXobbnau6e0vuUgngWRDt+Pcm62R76M7HrQVFCJIgytdM8bskk9e2hvrLkIUT4FRxQ8y+9Lum\nNO1tTV6FaxHPwXU11AGPedf9HNfF8HvgN8D/A+4HTgZ+193FdoFJQAi+VdWsn3OhRVaHrk7XFK5z\n3E3hWcgnfGJdDCYL7Av1l8O7I2EX4Aegfkzk2h9Y7J1eBPRr5zavAsOBSly3wXHAf9rcpmWz/0jg\nr8CTuJ3cZ3RxvV3tb9BcD4/6XcfGyOrQ9XxESxdTBTNYgZD10xSbvFAAqdOQ2EXwYC90KG4Ws+QG\n7rY5lLW7NKpxG77AdWoe3c59BgPv4TaSKfAasG2b21wB/BbXx9sykXWAteZYzToKPAmJOPyf37Vs\njFwI3bm4OTKLCZAiwlS+8L0rzZg1+kLjjwjUHA8/D6NVwFvd8DSnAXvi9r0cDDyAG/L1Cm7I2Gve\neYCFwJHe6fHACbi+23G4sPpe2uM+C+yK6xMu926zPe6QOtk8E9ubQJPLhpzqdMzaCW/SSZVMxK0V\nzWM1g4lxOj+mICd+Mkx+SQEvQ9F7cADoHSBD/K6phzoQ6t6AS1Oq9/hdy8bIldj6hJZay5hLghq+\n8rcgY9oVAA6Dhp/Dv7Zyq/G/As3m1fRcNBV41x2r+BG/a9lYuRK6C3ET4PTyjkD7Bm+27lRjTPYp\nhsR5SOxcuKUYHYzbISH71ytzw3UQS8Etqhrb8K2zS06Erk5XxXU9lQNQyVRWEmeur2UZs2GDoeHn\nBJYfDucH0J1BP/a7phy3GHgKpCm39lZulROh65kMLAVKEJQob1lr1+SM3dwuxZ+OdXuWnQHa3o4M\nZsNug0QInlLVpX7XsilyJnR1uiZxe524o6v05hPm4WLYmFwQAj0eif0Inurrxs7eAGoth85rAG6F\nRF1u7LvRrpwJXc9/ce97yx5q/+G11t3HjckNvaD5IqT+NLi6ELYGXvK7phxxN6QC8I6qTve7lk2V\nE0PG0kmVHAacBMwlQZhv+QmnE8XG5ZhcpMCbUPRv2FXRe0BG+l1TlloODIVYHeyqqjk1NjddrrV0\nwc0O3wRECBGnhBd5nubW3WiMySUCHAANl8HbI9yOCT8E7Whe3Hz2a/e9fyyXAxdysKULIFVyAG4m\nu9koMJ8LOJD+7GxHjzA5bjFEHkMLVyM3AOeSW9MsdpepwM5QH4OhuTJvbkdysaULbnKhhUAFApTz\nHK+QpNHnqozZXP2h8SfI6mPgJyF0NDly4K9upMC57nA8v8n1wIUcDV2drnHcnii9AKGUhYSZyhvd\nOteIMZmzgxtiNn1XdzieY0G/9bsmnzwKOgXmx90BMDaKiAwRkaki8oCITBeRR0XkQBGZ5J3fpTtq\nXp+cDF3PVNwMZG4O50r+xcckyddPpul5AsBEaPgZvDDITbt4JeTeLlibYQVwCTTWwhmqmtjEh9ka\nuEFVq4BRwKmqujduOuH/6aJSOy1nQ9fbS+1J3FFGwhRSTxnP8hRx22XC9CglkPguEjsHbiyCIcDf\nyI9dii+ExqTbeLY5xwadpdo61/wU3IRs4Ha4yvi4p5wNXQCdrotxO0wMAqA3X5LiK15kU38Rjcle\nQ6DhF8jSQ+DsAOwG+rnfNXWjB0D/AYvr4Eeb+VDpBz5IpZ1PAaHNfOyNltOh6/knMJOWSfP78TxT\naWKarzUZ0332dEet+O9o2B04F3J/61IbXwIXuzG5E1W1fjMfbn2jmjI+4innQ9fbqHYPrpshQogm\nKniKv5Owg1iaHisMeiISuwQe6wNDgZtBe8LumQ3AUdDQBD/sojG52sHp9s53u5wcp9seqZK9cRPi\nzwKURRxEJeM5k3Du/7QYswHToPhptE8zci9wsN/1bIYzoOlZeKEWTtSeElBpelIcvYM7gvRAAPrx\nOktYxqs2jMzkgVHuKMVz9oZjBA4GzfaDSrbnEdC/w5JaOLsnBi70oJYugFRJOe74egmghmaKWMgP\nOJxidrC91UyeaITgk2jBLOQHoFeBlPpdUydMB3aGhnrYQ7XnbiPsSS1ddLquBm7H7TRRSAEN9OFh\nXrQJz00eiUDyLCT2PbirdM1BLLN5epIlwEGuH/fHPTlwoYe1dFtIlewFXADMAZKsZAS1nMwFhOjl\nc3HGZNrHUPwPdGgS7gfZze962qgFdofYbLi1XvWyDd0+1/XU0BXgeOAo3LHVlCXsjjCBCwgT8bU8\nYzIvCbwA0U9gIugtIAP9rgk3YPZAaPoMnqzrwf246Xpk6AJIlQSBC4EdgHkosIhqKhjDmYQJ+1uf\nMb6ogfDjaHghchnoz0H8aoMkgeOg+Q14rRaOUtW82OjdY0MXQKqkCLgcd4ifJSjCQk6kNyP4jgVv\npyVwnYJJXMfgdsD+3nXvAx/itg5sQ/tjlZ4FvgKKcT+DLV4BvgEGAMd6l32OG6i5e1e+ALOOmVD0\nV7Q0htwNHE1m9xJQ4AKIPwGf1sK+qpo3cwT26NAFkCrpg5vUIgwsI0WARZxEX4ZzOuHM7wSYo5qB\nAlzo3g8cDsSBt4HTcZO+1uOCta053n2fYU3oNgJ/Bc4AnsOFbC/gceA79LBNvFnsbSh+HbZX9F6Q\n0Rl62isgeTPM9I4CkVdztvf4j7ZO12XA73Fx0ZsAKQbwFEuYzePEbZaGTirw/iZw76Tgjli3N2tm\n2W4vcMFNKRJtc5lA6wjqOO6T+B9gPHnwqcwi+0D9r+C9UbAr8D3QFd38lH+A1E2wtM61cPMqcCFP\nPt46XRcC13lnXfBuwRMsYi5PELfdJzohBdwN3IibKG8Q7qBVc4B7gQeB+RvxeIW47oi7gVLv/Hzc\nxHsms8KgpyCxi+CRSnQocDts8jyKHVHgMkheBYu8sbiLuvgpckKP715IJ1WyJXAZLkJWkCLIQk5j\nIIM5iZD18XZCI25CzcNx8wsO807Px3UX/LiD+60CHmPtPt10z+GaWguBGbhZkvftsqrNxpgCxX9H\n+8eR+4ADuuAhE8D5kHwGZtTA3qq6tAseNiflRUu3hU7Xb4HrcSvEvQiQZACPsZAZ/Jk4DT4XmAsi\nuNlVvgHKgG29ywfhugw25T1c6P3tjZvt9ETc7NXdvZ5r2jca6i9HZu4JRwKHg87ajIdrBI6GxDPw\ncQ3sks+BC3kWugA6Xefh+njDQG+CJNmCJ6nnv9xN3L7o7aiH1uPPxXEt0T64roCWb+My3PpDUQeP\nsb4VqjdwzalU2u3Eey7jjwBwCDT8Al4dAqOBX8JGz7G4Gtgf4pPglRrYR1Xzfu6/vOpeSCdVMgj4\nKW7zj+tbWsquxDiE7xBiSz+ryzKLcSMP1FvG4Fb9k7jhYItw6w6H4lrBtbjugtO9+/8Nt4tKDPdu\nHwDs6F03zbv//t75l1kzjOy47npBZqN9C9En0GgdcitwGhseYub9W5sXwF9q4bv5Mg53Q/I2dAGk\nSnoBP8Ttnj4PgBWMpIYTOI6wbdQxpo0PofgldOuU26W4o6M6TgMOguaVcGMD/Dof9jTrrLwOXQCp\nkihuHt6dcNviU9QykOWcwX4UsCcBm5/MmDRJ4DmIfgbHgN4EMiDt6meAM6G5CS5uVr3XpyqzVt6H\nLoBUSQg4Fbc/1TwgToxylvEdBlPOcYQp9LdGY7LOaih4DA0tRq4A/SHIlZC6C+rq4aCU6od+l5iN\nLHQ93iQ5h+HCdzFQT5IQSziSANtxOmHvKGzGmHRfQ/Ff0WQzEoEptTAhobrE77KylYVuG1Il43Cj\nSZO4aT5hKTtQzxEcRogdEetuMCbNbJAnSQSbeTqR5HTVLt+vokex0G2HVEl/4GJgS1x3Q4p6+rKc\n0xhOCUcTsu4Gk/dSwFuovENzYYJTY6rP+F1SLrDQ7YBUSQQ4BZiAG74fI0mYJUwEtuN4wgz1s0Jj\nfLQU+BsJaphJMwdpQuf5XVKusNBdD6+fd1fgfNyejK67YQVV1HAMYwhxqLV6TR5JAu+Q4m1SBPkd\njVytqtl8JKCsY6HbCVIlA4Dv44b+zwfixImwjImkqOJYwozwtURjut8SXOu2jvkI1VrXs49l1l0s\ndDtJqiSMG1J2PG6nWLf/+Eq2pobjGEkBRxBaZwpDY3JdEphEikmkCHMTAS7XWmvdbioL3Y3k7T58\nDm5iwgVAEwkKWMphJBjDUYTZlsxOw29Md5kLPEeCBuYR4Bittdbt5rLQ3QTe8df2x43pTdAyd8Mq\nhlDDMfShmImE2cK/Go3ZLKuAf5JgJklCXEOE/9XlFhZdwUJ3M3hDy87GHTVsMdBAigDL2JkGDmQU\nQQ4mRKmvZRrTec3A26R4DyXCa0Q5RxfrAr/L6kksdDeTVEkA2APX6i3CDS9zG9qWsz+N7MxeBNiT\ngE2SbrKWAl8AL5EkwCwifJ9KXtfpFhBdzUK3i3hHHj4EOAq36WEhoDRQySqOQBnMYYQZTR7OYmyy\nluKm3fwXCVZTRyFX0o87dbrtVdZdLHS7mFRJX9wIh91xM8suB2AVQ6ljIoWUMYECC1/jK8VNQP8q\ncVYQp4CHqOBynZN/B4rMNAvdbiJVsg1urufhuOMq1KLAKkZQz8GE6cUEwoxhzdF0jelubcM2wrP0\n4nKdqXP8Li1fWOh2I2+Uw064o371wx31qwYFVjOMOg4mSB8OIMw4LHxN92kJ21eIs5I4Uf5BL64m\nxJfWb5tZFroZ4M3XOw4XvgNwA3JWAW6YWT0HI/RjH0LsgNhuxabLJIAvgUnEqSFOlBeo5CqCTLOw\n9YeFbgZ5Ld/tcX2+W+KO27cSgNVsRT370sRQxiHsRpC+/tVqclwt8CEpPiBFiFUU8hIVXGth6z8L\nXR94w8zG4MJ3CO7A5UsBpZEyVrErMXZlAMJeFDAS2+hmOudb4D/E+YoARcyihOcp4U/AVxa22cFC\n10de+FbhhprtgJuhdDHQTJIgK9mORvZBqGB3QuyEUOxnxSYrNeG6EN6lmdWkKOIzyniIQv4BzLOw\nzS4WullCqqQfsDduUp0Irs/XDd+pZSB17EmMKrYixU4UUAUU+Fau8VsSmAl8TJyvCRBlCRH+Szl3\nEOBdna41fpdo2mehm2W8ydPHAUfi+n2bcV0PCRIUsJJRxNmFJgYykhQ7EWYY1v2QDxQ3y8cnJPkc\nJUgdBUyjjNco5G/AF7ZTQ/az0M1S3gTqw4H9cDtahIF63LCzFE2UsIoxxNmFFOVsjzCOIAOxGc56\nmqXANJSPidNAighfUcpkingG+AD41roQcoeFbg6QKokC2+ICeKx38Wpauh/q6UMt42j2BpyNQtiW\nEMPA5nvIQUncBrGpJPmSJI0oEeZRxDRKeR5hEm7DmLVqc5CFbo6RKinHDTs7EDfyQXHDzmoBqKc3\ntVSRZCxN9GUwSUZ7IyBstrPs1QTMAKYQ52uEEPWEmU0JcyniHYQ3cN0HDT5XajaThW4O86aW3AnY\nC1o7FupwIZwiTpTVjCDOGGIMo4IU21LAUIStsA1xfkrgpsCfhfI1zSwiRJTlhJhJGXMp5BPgP8AU\nna7L/S3WdCUL3R5CqqQ3MBLX/7sdbqfiBG7CnSZSBKhhMDFGkGIEjfShNwlGEGYoAQbjxkyY7tE2\nZBcSopAawiygkAWUMJ8Q7wEf4roO6vwt2HQXC90eyBsBMQLYERiPm+dXcDthrAbiJAlRy5bEGIoy\nkhj9qCDB1oQY4m2Qq8A2ym2qWlzIfosya62Q/ZZCFlPCYkIsBd4FPgNm6nSN+1qzyQgL3R7O2wFj\nEDAM1xc8GtexEGBNCLudMeoY6IXwcJoYAIToT5KtCLMFAQYAvbHhaenc5EVuKNcilHk0s4AAcSDK\nUgIsppDlXsguwwXsF7hZbJfZqIP8Y6GbZ7wQ3gK3Ea4lhItxbdo4ro1WT8uwtHq2oIlBwFbE6U+c\nKL2IM4AA/QhTCa1LT+6eiOMG663AddgsIc5SUiwnCCQpZCnCIgqopYhlFFKHUIOFrGnDQjfPeeOB\n+wNb4cYFj/JOi7ekcBvn6nA7aBRSTz+a6EuS3ggDSFBJI6WEUSpI0ocAfQlRiVAKlHhLhOzsrlDc\n6IE63E9Oy99lJFhCkhUEaCRIIfWEWAEsIUQthTQSoYYCGlizj9g0XMDOx0LWtMNC16xDqiSMC+IB\nuBbxCGAoUIiLKPH+xloXJUUTpTRRSROVJOkN9EMpI0kRcaKkCBIhQTEpSoAyApQRJorr8CjAjSsu\naOd8qJ1CtZ3Tcdw+fM24IG17ugloIMlqktSSoo4AMUKAEqaREHUEqEVZTYDVFNJAhEYKqUVI4TZQ\n1uNmp/0KmIfrXFiq0zW56e+6yRcWuqZTvBZxBa4joTdrWscDvdMB1gRygLXjzy1JAjRTQpxiEpSQ\noIQkpUARQgQoQCkECkhRgBImSZgUIVLrTPHuPrjS5rIASYIkCNCMtC6NQCNKI0qMAM2EiRGiiQKa\nCZMkRArXqm95DQncce7mA3Nw+4Utx3Uw1FsL1mwqC12z2bx+4jLW9O6WAX1w4VwJ9ALKca1EZU27\ntCUyA97pZDvLOk/XzmUB77FblpYfgJbnSX++lkBdyZoe2mXe3zrcZrHlQI0Fq+kOFromI7yWciFr\neniLWbszIexdFm2zpEsP0fQPbhK3yt+Q9re9DoZ6XG9tkwWq8YuFrkFETgd+iAu+94EL1T4YxnQL\nG3GZ50RkFHAysKeq7oTr1zzd36qM6bna2yZs8suBuPkbPhQRwQ3sWuxvScb0XBa6RoCHVPV//C7E\nmHxg3QvmNeAEEekLICK9RGSwzzUZ02NZ6OY5VZ0K/Bp4WUQ+A17G7RRhjOkGNnrBGGMyyFq6xhiT\nQRa6xhiTQRa6xhiTQRa6xhiTQRa6xhiTQRa6xhiTQRa6xhiTQRa6xhiTQRa6xhiTQRa6xhiTQRa6\nxhiTQRa6xhiTQRa6xhiTQRa6xhiTQRa6xhiTQRa6xhiTQRa6xhiTQRa6xhiTQRa6xhiTQRa6xhiT\nQf8feN4p2xejFEMAAAAASUVORK5CYII=\n",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"categories_pie_plot(cont2,\"Categories % in training set\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": false
},
"source": [
"## Train and test the classifier"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In order to train and test the classifier, the first step should be to tokenize and count the number of occurrence of each word that appears into the news'titles using for that [CountVectorizer class](http://scikit-learn.org/stable/modules/feature_extraction.html#common-vectorizer-usage). Each term found is assigned a unique integer index.\n",
"Then the counters will be transformed to a TF-IDF representation using [TfidfTransformer class](http://scikit-learn.org/stable/modules/feature_extraction.html#tfidf-term-weighting). The last step creates the [Multinomial Naive Bayes classifier](http://scikit-learn.org/stable/modules/naive_bayes.html#multinomial-naive-bayes). \n",
"In order to make the training process easier, scikit-learn provides a [Pipeline class](http://scikit-learn.org/stable/modules/pipeline.html) that behaves like a compound classifier. \n",
"The [metrics module](http://scikit-learn.org/stable/modules/classes.html) allows to calculate score functions, performance metrics and pairwise metrics and distance computations. F1-score can be interpreted as a weighted average of the [precision and recall](https://en.wikipedia.org/wiki/Precision_and_recall)."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"from sklearn.feature_extraction.text import CountVectorizer \n",
"from sklearn.feature_extraction.text import TfidfTransformer \n",
"from sklearn.naive_bayes import MultinomialNB \n",
"from sklearn.pipeline import Pipeline \n",
"from sklearn import metrics \n",
"import numpy as np\n",
"import pprint\n",
"\n",
"# lmats = [] # list of confussion matrix \n",
"nrows = nlabels\n",
"ncols = nlabels\n",
"# conf_mat_sum = np.zeros((nrows, ncols))\n",
"# f1_acum = [] # list of f1-score\n",
"\n",
"def train_test():\n",
" global X_train, y_train, X_test, y_test, labels \n",
" #lmats, \\\n",
" # conf_mat_sum, f1_acum, ncategories\n",
" text_clf = Pipeline([('vect', CountVectorizer()),\n",
" ('tfidf', TfidfTransformer()),\n",
" ('clf', MultinomialNB()),\n",
" ])\n",
" text_clf = text_clf.fit(X_train, y_train)\n",
" predicted = text_clf.predict(X_test)\n",
" return predicted"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Wall time: 27.1 s\n"
]
}
],
"source": [
"%time predicted = train_test()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To compare the predicted labels to the corresponding set of true labels we will use the [method accuracy_score from scikit-learn](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html), which gives an accuracy over 0.92"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"0.92380411281031516"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"metrics.accuracy_score(y_test, predicted)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To show the main classification metrics we will use the [classification_report method](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html) from scikit-learn."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" precision recall f1-score support\n",
"\n",
" b 0.90 0.91 0.90 34729\n",
" e 0.95 0.97 0.96 45625\n",
" m 0.97 0.85 0.90 13709\n",
" t 0.90 0.90 0.90 32663\n",
"\n",
"avg / total 0.92 0.92 0.92 126726\n",
"\n"
]
}
],
"source": [
"print(metrics.classification_report(y_test, predicted, target_names=labels))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that, although the metrics (precision, recall and f1-score) in average give us 0.92, the results for category e (entertainment) are even better."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[Confusion matrix](http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html) allows to detect if a classification algorithm is confusing two or more classes if you have an unequal number of observations in each class as in this case. An ideal classifier with 100% accuracy would produce a pure diagonal matrix which would have all the points predicted in their correct class. In case of class imbalance, confusion matrix normalization by class support size (number of elements in each class) can be interesting in order to have a visual interpretation of which class is being misclassified."
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"array([[ 0.91004636, 0.02361139, 0.00552852, 0.06081373],\n",
" [ 0.01176986, 0.97192329, 0.00192877, 0.01437808],\n",
" [ 0.06200306, 0.06193012, 0.84601357, 0.03005325],\n",
" [ 0.06778312, 0.02510486, 0.00324526, 0.90386676]])"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"mat = metrics.confusion_matrix(y_test, predicted,labels=labels)\n",
"cm = mat.astype('float') / mat.sum(axis=1)[:, np.newaxis]\n",
"cm"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's print a plot for the confussion matrix."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import itertools\n",
"import matplotlib.pyplot as plt\n",
"\n",
"def plot_confusion_matrix(cm, classes,\n",
" title='Confusion matrix',\n",
" cmap=plt.cm.Blues):\n",
" \"\"\"\n",
" This function prints and plots the confusion matrix.\n",
" \"\"\"\n",
" plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
" plt.title(title)\n",
" plt.colorbar()\n",
" tick_marks = np.arange(len(classes))\n",
" plt.xticks(tick_marks, classes, rotation=45)\n",
" plt.yticks(tick_marks, classes)\n",
"\n",
" thresh = cm.max() / 2.\n",
" for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
" plt.text(j, i, '{:5.2f}'.format(cm[i, j]),\n",
" horizontalalignment=\"center\",\n",
" color=\"white\" if cm[i, j] > thresh else \"black\")\n",
"\n",
" plt.tight_layout()\n",
" plt.ylabel('True label')\n",
" plt.xlabel('Predicted label')\n",
" plt.colorbar()\n",
" plt.show()"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEpCAYAAACA6BUXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd8VFXawPHfkwQUpQiCQMIShFAVpCX2roBSFRSxrn1B\nsCu7ilhWF1FXEXFdUV/X1UUQEWkioK4KiiYYigWkypIMoCgqnQDP+8dcwpDGkCn3kHm+fuZj7twz\nc8557nCfe89toqoYY4wxB5LkdwOMMcYcGixhGGOMCYslDGOMMWGxhGGMMSYsljCMMcaExRKGMcaY\nsFjCSFAicriITBGRX0VkXATfc7mIvB/NtvlFRE4TkcV+t8MYV4ldh+E2EbkcuANoAfwOLAD+pqqf\nRfi9VwIDgZM1AX4EIrIHyFDVlX63xZhDle1hOExE7gSeBh4FjgEaAs8D3aPw9enA0kRIFp4y+yki\nyfFqiDGHKksYjhKR6sDDwABVnaSq21R1t6q+p6p/9spUFpERIpIvInki8oyIVPLmnSkia0TkThFZ\n75W5xpv3EDAUuExEfheRa0XkQRF5PaT+dBHZIyJJ3vQfRWSFV36FiPTz3r9GRGaHfO4UEckWkY0i\n8qWInBwy778i8oiIzPG+530RqVVK//e2/56Q9vcUkQtE5HsR2SAifwkpnykin3v15ovIcyKS4s37\nBBBgkVfvJSHff6+IrAX+b+973mcai8jPItLWm04VkR9F5IxIl60xhypLGO46GTgMeLeMMkOALKAN\ncIL395CQ+fWAakAqcAPwDxGpoaoPAX8DxqpqdVV91StfdCtcAUTkCOBZoLOqVgdOITg0VrRcTWAq\nMAI4GngGmOa9v1c/4Bqgjte/u8voXz2gstf+B4GXgCuAdsAZwAMiku6V3Q3cDtQiGLtzgAEAqnqm\nV6a119/xId9/FME9t5tC++INXd0LvCEiVYBXgVdV9dMy2mtMhWYJw11HAxtUdU8ZZS4HHlbVn1X1\nZ4J7JFeFzN8J/NXbM5kObAaal7M9u4HWInK4qq5X1ZIODnclOMw1RlX3qOpYYAn7D6G9qqorVHUH\n8BbQtow6dxI8XrMbGAvUBkao6lZV/Q74jmCiRFVzVTVbg/4HjAbOLPJ9UkKfHlTVAq89+1HVV4Dl\nwJdAXfZPxsYkHEsY7voZqL13SKgUqcD/QqZXe+8VfkeRhLMVqHqwDVHVrUBfoD+w1ju7qqTEk+q1\nIdRqIC1ket1BtOfnkGMs27z//xgyf9vez4tIU69da0XkV+AxggmmLD+pasEByrwMHAc8F0ZZYyo0\nSxjumgvsAHqVUSaf4MHrvdKBQDnr2wIcETJdP3Smqs5S1U4Eh3G+J7gFX1QAaFTkvYZeO2PtBWAx\n0ERVjwLup/geRVEHOhB+JMHhtVeAh0TkqGg01JhDlSUMR6nq7wTH7Z/3DvZWEZEU76Dv416xscAQ\nEaktIrWBB4DXS/vOA1gAnCEifxCRGsCf984QkWNEpId3LKOA4NBWSUNl7wFNReQyEUkWkb5AS2BK\nOdt0MKoBv6vqVhFpQXBvKNQ6oPFBfudIIFtVbyLYtxcjb6Yxhy5LGA5T1aeBOwmOnf9IcPhpAPsO\nhD8KzAMWAQu9vx8r6yvLqOsDYJz3XTnsv5JP8tqRD2wgeMC56AoZVf0F6EbwQPYG7/9dVXXjgeoP\nU4kH5T13A1eIyO8EV+xji5R9CPi3iPwiIn0OVJGI9AA64R04J9j/dnvPDjMmEdmFe8YYY8JiexjG\nGGPCYgnDGGNMWCxhGGOMCYslDGOMMWFJ8bsBRYmIHYU35hChqge61qVUUrm6UrDpYD6yWlUblbe+\nQ4XLcXHuLCkR0cPP+mtc6yxY9RGVjj0nrnUC/PT+fXGv829/fZj7Hngw7vWmJMd/Z/bRRx5iyNCH\n4l7v5u274l7nE397hHvvGxrXOutUqxRZwhDRw9sNCrv89vnPRVTfocLluDi3h2GMSSBS4df/5eNo\nXCxhGGP8U+at0hKYo3GxhAEkHXWs302Im9PPKHoD14rrjDPP8rsJcXPq6YfocnV0S9p3jsbFEgaQ\nXDOBEkYCrUQtYRwCHN2S9p2jcbGEYYzxj6Nb0r5zNC6WMIwx/nF0S9p3jsbFEoYxxj+Obkn7ztG4\nWMIwxvgnKdnvFrjJ0bhYwjDG+MfRoRffORoXSxjGGP84OvTiO0fjYgnDGOMfR7ekfedoXNxslTEm\nMUhS+K/SvkKki4gsEZGlIjK4hPlHicg7IrJQRL4QkVYx7VM0OBoXSxjGGP8kSfivEohIEjAK6Awc\nB/QTkRZFit0HzFfVE4BrgJEx7FF0OBoXSxjGGP9EviWdBSxT1dWqWgCMBXoWKdMK+AhAVb8HGolI\nnVh1KSocjYslDGOMf0TCf5UsDVgTMp3nvRdqIXBxsDrJAhoCDaLck+hyNC520NsY458yxuB3b1zJ\nno0ro1HL48CzIpILfA3MB3ZH44tjxtG4WMIwxvinjNNHk2s1IblWk8Lp3as+LKlYPsEt470aeO8V\nUtVNwHX7qpRVQFTWuDHjaFxsSMoY45/Ix+pzgAwRSReRysBlwOT9qhCpISKVvL9vBD5R1c2x7FbE\nHI2L7WEYY/wT4QVqqrpbRAYCMwluAL+iqotF5ObgbB0NtAReE5E9wLfA9RG2OvYcjYslDGOMf6Jw\nzyRVfR9oXuS9F0P+/qLofOc5GpcKPyTVtmk9sl++kUX/7s+Tt5xfYpmU5CT+eW83sl++kbmjb+C0\nE/YN/T143VksHTuI9VPviVOLy2/B/FxO6tiWtse3YPDdd5Ra7qknHqftcc3pcMJxfDhrJgDbtm2j\nz0Xd6XDCcZzY4QQeeuD+eDW7XObn5pLZrg2tWzXj7jtvL7Xck8OHcXzLprRt3ZIPvL4CPDR0CE0b\nN+SYWtXj0dyILFyQy5kntePEtq24f/CdpZYb8dRwstq25JQOrfnvh7MK3y8oKOCuW/tzUrvjOLVj\nG6ZNfjcezQ5PFC5Qq5AcjUuFXwoj77iA/k9Opc3VL5DR4GjOy2xcrMx13dqhClk3vET3e8bweP/z\nCudN+3wpp/X/v3g2udzuuPUWnv/nSyz4ZgnLly3lg1kzipVZsmQxEyeMZ97Cb3ln0jTuvG0gqgrA\n7XfczVcLv+WzL79i7ueflfh5V9w6sD8vjH6Fr79byvJlS5k1s4S+Ll7MhPFvseDrxUyaMp3bBg0o\n7GvXbj2YMzcn3s0ul3vvGMiI50fz5YLvWLF8GR99MLNYmaVLFjNp4ng+n/c1Y9+Zwr13Dirs6zNP\nDqPOMXX5Yv63fDZvEaecdka8u1C6yE8frZgcjUuFThh1ax5J1SqV+er7tQCMmbmI7qcW3wNrmV6b\nT+b/AMCG37by2+YdtG9WH4B5SwL8uHFL3NpcXuvXrWPTpk106JgJQL8rrmLq5EnFyk2bMpnel/Ql\nJSWF9EaNaJKRwbycbKpUqcJp3vO+U1JSaNuuHfl5eXHtQ7jWrVvH5s2b6JgZ7OvlV17NlEnFt5qn\nTJlEn76XFfY1I6MpOdnZAGRmZVG3bt24trs81q9fx+ZNm2nXIdjXvv2uZPrU4st1+rQpXNQ7uFwb\npjeicZMMcucF+zrm9X9x21377gxRs1at+DQ+HI5uSfvO0bjErTbvaP3X8aoPILVONfJ/2lQ4nf/T\nJlJrVytWbtGKH+l6SlOSkoT0ekfRrlk9Ghzj/lBFqEAgn7S0fdflpKY1IBDIL1ZubSCfBg32XZtT\nPzWtWLlff/2V6dOmctbZ58auwREI5OeTlravD2ml9DWQn0+DBn8onE4toa+uWxfIJzVkudZPTWNt\nIFCs3Nq1+aSFLtf6qawNBPj9t98AGPbIUM49PYsbrrmcDT/9FPuGh8vRFaPvHI1LvJeCxrm+sLw2\nfQGBDZuY88J1PDHgPOZ+k8fuPXv8bpYvdu/ezfXXXMGAQbeR3qiR380xEdq1axeB/DxOPPlUPpyd\nTcfMLB6836HjcY4OvfjO0bjE+yypSiLyBtAe+Aa4WlW3x6qywE+b9ttTSKtTjcCGTcXK7dmjDP7H\nB4XTHz13DcvW/BKrZsVEamoaeSFDSIH8PFJTi94JILiFWla5QQNupmmz5vxpwMDYNjgCqWlp5OXt\nu+tBfil9Dbecy+qlpu03NLg2kE/91NRi5erX379cwCtX6+ijOeLII+naoxcAPS7qw5jX/xXzdoct\n0fYcwuVoXOLdqubAKFVtBWwCBsSysvUbt/D7lu10bBH8B3Z5pzZM/ez7YuUOr5xClcOCufOcDsdS\nsGsPS9f8vF8Z1zdw6tarR/UaNZiXk42q8uZ/Xqdr9x7FynXt1p0J48exc+dOfli1ipUrVtAxMwuA\nRx58gE2bfufxJ5+Od/MPSr169ahevQY52cG+jnnj33TrUfS+atCtWw/eHje2sK8rViwnMytrvzJ7\nDwy7qm7delSvXp3cecG+jnvzDbp0Lb5cu3TtxsQJweW6+odVrFq5gvYdg33tfEE35nz6MQCf/PdD\nmrVw6O7ejm5J+87RuMR7D+N/3rm/AG8Ag4Bia6eCVR8V/p101LEk1zy23BXe/uz7jB7cg8MrpzDj\ny+XMygle+X7hyU1p16w+j732KcfUPJLJT/Rj9x4l8NMmrv/bvoOKj950Dn3PPZ4qh1Vi6dhBvPre\nAob9e3a52xNLT494jj/deB07tm+nU5cLOL9TFwDemzaFBbm53PfAg7Ro2YqLel9CZtvjSalUiaef\nHYWIEMjP56knhtG8RUtOPbEDIsJNfxrA1X+87gC1+mPEyOe56fo/sn3Hdjp3uZBOnYN9nTZ1CvNz\nv2LI0Ido2aoVvS+5lHZtWlGpUiWefe4fiPcP7P6/DGbc2DFs27aNpo0bcu11N3DfkKE+9qh0w58e\nyaA/Xc/2HTs4r1MXzj2/MwAz3pvKwgW53HvfUJq3aEXPiy7h1Mw2VEqpxBNPP1fY1wcefowBN/6R\nIX++i9q16zDyhZfL1Y7PZn/CZ7M/iVq/AGe3pH3naFwkXltYIpIOfKyqx3rTZwMDVbV3kXJ6+Fl/\njUub/PbT+/f53YS4SUl28x9ALGzevsvvJsRFnWqVUNVyb+KKiB5+UfjJa/vEGyKq71DhclzivYeR\nLiInquqXwOXAnDjXb4xxiCTaUFOYXI1LvDf7lgC3iMh3wFHAC3Gu3xjjEEmSsF+JxNW4xG0PQ1VX\nE3zCkzHGAO5uSfvN1bjYzQeNMb5xdcXoN1fjYgnDGOMbV1eMfnM1LpYwjDG+cXXF6DdX45I45zoa\nY9wjB/Eq7StEuojIEhFZKiKDS5hfXUQmi8gCEflaRP4Y9X5Em6NxsYRhjPGNiIT9KuXzScAooDNw\nHNBPRFoUKXYL8K2qtgXOBv4uIk6PrrgaF0sYxhjfRLpiBLKAZaq6WlULgLFA0fvEKLD3NtXVgJ9V\n1emrK12Ni9NZ1hhTsUVhrD4NWBMynUdwZRlqFDBZRAJAVaBvpJXGmqtxsYRhjPFNWSvGgnXfUbD+\nu2hU0xmYr6rniEgTYJaItFHVzdH48lhwNS6WMIwx/iljQ7pS/VZUqr/vWt/ti94pqVg+0DBkuoH3\nXqhrgWEAqrpCRFYBLYB55WlyXDgaFzuGYYzxTVJSUtivUuQAGd4TPSsDlwGTi5RZDZwHICJ1gWbA\nyhh1KSpcjYvtYRhjfBPpWL2q7haRgcBMghvAr6jqYhG5OThbRwOPAv8SkUXex+5VVaefkOZqXCxh\nGGP8E4Xr01T1fYIPZwt978WQv9cSHK8/dDgaF0sYxhjfuHpFs99cjYslDGOMb1xdMfrN1bhYwjDG\n+MbVFaPfXI2LJQxjjG9cXTH6zdW4WMIwxvjHzfWi/xyNiyUMY4xvXN2S9purcbGEYYzxjasrRr+5\nGhdLGMYY37i6YvSbq3GxhGGM8Y0kubli9JurcbGEYYzxjatb0n5zNS6WMIwxvnF1xeg3V+NiCcMY\n4xtXV4x+czUuTiaMn2fc73cT4uLoEwf53YS42Zgzyu8mxE2Vysl+N+HQ4eZ60X+OxsXJhGGMSQyu\nbkn7zdW4WMIwxvjG1RWj31yNiz1xzxjjG5HwX6V/h3QRkSUislREBpcw/24RmS8iuSLytYjsEpGj\nYtmvSLkaF0sYxhjfiEjYr1I+nwSMIvggoOOAfiLSIrSMqj6lqu1UtT3wF+BjVf01xl2LiKtxsYRh\njPFNFLaks4BlqrpaVQuAsUDPMqrsB7wZ3V5En6txsYRhjPFNpFvSQBqwJmQ6z3uvpLqqAF2ACVHt\nRAy4Ghc76G2M8U2cj+12B+a4PhwF7sbFEoYxxjfJyaWvGbesXsiW1YsO9BX5QMOQ6QbeeyW5jENg\nOArcjYslDGOMb8o6fbRqo7ZUbdS2cHrDnDdKKpYDZIhIOrCW4MqvXwn11ADOBK6IrMXx4WpcLGEY\nY3wT6dCLqu4WkYHATILHZF9R1cUicnNwto72ivYCZqjqtshqjA9X42IJwxjjm2hcoKaq7wPNi7z3\nYpHp14DXIq4sTlyNiyUMY4xvXL2i2W+uxsUShjHGN46uF33nalwsYRhjfOPqlrTfXI2LJQxjjG8c\nXS/6ztW4WMIwxvjG1S1pv7kaF0sYxhjfOLpe9J2rcbGEYYzxjatb0n5zNS6WMIwxvklKcnPF6DdX\n42IJwxjjG0c3pH3nalwsYRhjfOPq0IvfXI1LhX8exvz5uWR1OIETjmvOPXfdXmq5J58YRptWzWjf\nphUfzJpZ+P7DDw6heUY69WrXiEdzI9K2RQOyx/2FRe8O5cm7e5dYJiUliX8+eAXZ4/7C3DcHc1qH\nDACOrFKZuW8O5vMxg5n75mD+99Ewht91cTybf1Dm5+aS2a4NrVs14+47y1iuw4dxfMumtG3dcr/l\nGu7nXbBgfi4ndjiBtsc1594yfsNPPTGME7zf8IdFfsMtMtKp7+BvOBqPIq2IXI1LhU8Ytw8awAsv\nvszCb79n+bJlzJo5o1iZJYsX887b48ld9B0TJ7/HHbfegqoC0LVbDz79LDvezS6XkfddRv+H/0Ob\nXo+QkX4M553csliZ6y46FUXJ6juM7gOe5/E7g0lhy7adnNxvOKdcPpyT+w3nf2s38u4HC+LdhbDd\nOrA/L4x+ha+/W8ryZUtLXa4Txr/Fgq8XM2nKdG4bNKBwuYbzeVfcPmgA/3jxZRZ4v+EPyvgNf1XK\nb/gTR3/DUXhQUIXkalwqdMJYt24dmzdtokPHTAAuv/Iqpk5+t1i5qVMn0efSvqSkpJDeqBFNMpoy\nLyf4D6xjZhZ169aNa7vLo+7R1ah65GF89d3/ABgzNZvuZ7cpVq5l43p8kr0UgA0bN/Pbpq20b9Vw\nvzIZDY+hTs2qzF24MvYNL4d169axefMmOmbuXa5XM2VS8eU6Zcok+vS9rHC5ZmQ0JSc7O+zPu2D9\nunVsCvkN97vyKqaU8Buedoj+hl3dkvabq3Gp0AljbSCf1LQGhdNpaQ0IBALFy+Xn06DBHwqnU9NS\nCQRKe9aIm1KPOYr89fsemJW/fiOpxxxVrNyipfl0PbM1SUlCeurRtGvZkAZ19y/Xp3N73p7xVczb\nXF6B/HzSii3X4ssrUHS5pqYRCOSH/XkXBALF27q2hN9wsb4eIr9hV7ek/eZqXOygd4J5bdJcWjSu\nx5w37mXN2l+Yu2Alu/fofmUu6dyB6+4/ZO4EbQ5hiZYIwuVqXOK6hyEiV4jIlyKSKyIvSIyjUj81\njfy8fc9Bz8/PIzU1tXi5tDTyQsvl5ZOaWuLz0p0V+PFXGtStWTidVrcmgR+LP6J3zx5l8N/f4ZTL\nh9P3rpeoWb0Ky1b/WDj/+KapJCcJC7/Pi0u7yyO16PLKzytxeZVWLtzPuyA1tXhb65fwGy7Wp0Pk\nNxyNoRcR6SIiS0RkqYgMLqXMWSIyX0S+EZH/xqo/0eJqXOKWMESkBdAXOEVV2wN7iPHjEuvVq0f1\nGjWYl5ONqjLmjdfp2r1nsXJdu/Xg7bfGsXPnTn5YtYqVK5bTMTNrvzJ7DyC6av3Pm/h98zY6HpcO\nwOXdspj6cfHn/h5+WCWqHF4JgHNObEHBrj0s/WF94fxLu3TkrffdHY4Cb7lWr0FO9t7l+m+69Si+\nXLt168Hb48YWLtcVK5aTmZUV9uddULdePWqE/IbffON1upXwG77wEP0NRzr0IiJJwCigM3Ac0M9b\n14SWqQE8D3RT1eOBS2Lbq8i5Gpd47mGcC7QHckRkPnAO0DjWlT7z7Cj633wDJxzXnCYZGXTq3AWA\n96ZO4bG/PgRAy5atuLjPJXQ44Tgu7tmVZ0Y+X7gghtw3mGZNGrJt2zaaZ6Qz7LFHYt3kcrv98bd4\n4cEr+HrSUFb870dmfb4YgAvPOJ77b74QgGNqVWXumMF89fb93HHNuVw/ZP+hp4vPa+d8wgAYMfJ5\n+t90Pa1bNaNJRtPC5Tpt6hQefeQhAFq2akXvSy6lXZtW9OpxIc8+94/C5Vra51309LOjGHDzDbT1\nfsPnl/Eb7ljCb/iB+wbT3PsNt3DsNxyFLeksYJmqrlbVAmAsUDSjXg5MUNV8AFXdEKv+RIurcZF4\nbXV4z5etr6r3H6Cc3jdkaOH06WecxRlnnhXj1vnj6BMH+d2EuNmYM8rvJsRN0WNCFcWnn3zM7E8/\nLpwe9ugjqGq5h5VFRM8Z+XnY5T+69ZRi9YlIb6Czqt7kTV8JZKnqrSFlngEqEdzSrgqMVNXXy9vu\nWHM5LvE86P0h8K6IjFDVn0SkJlBNVf9XtOD9DzwUx2YZY8Jxxpn7b7wNezTyPZWk+BzcTSE4unEO\ncCQwV0TmquryeFReHq7GJW4JQ1UXi8gQYKY3vrYTuAUoljCMMYmhrPXiL0tz2bgs90BfkQ+EXkjU\nwHsvVB6wQVW3A9tF5FPgBMDZhOFqXOJ6Wq2qjgfGx7NOY4y7yjpR8ujmHTi6eYfC6ZXTXympWA6Q\nISLpwFrgMqBfkTKTgOdEJBk4DDgReDqihseYq3Gx6zCMMb6J9C7eqrrbOz46k+BJPK94oxk3B2fr\naFVdIiIzgEXAbmC0qn4XYdNjytW4WMIwxvgmGpdiqer7QPMi771YZPop4KmIK4sTV+NSasIQkeoH\naMzv4VZijDElcfSCZt+5Gpey9jC+BRQIbfreaWX/AyrGGHPQBEfXjD5zNS6lJgxV/UNp84wxJhoc\nfRKp71yNS1jHMETkMqCxqv5NRBoAdVXV/cuBjTFOc/Ume35zNS4HvDWIiIwCzgau8t7aCvwzlo0y\nxiQGV5/74DdX4xLOHsYpqtreu/8TqvqLiFSOcbuMMQkg2dWxF5+5GpdwEkaBd2W2AojI0QTvNGuM\nMRFxdejFb67GJZyE8TwwAagjIg8DlwIPx7RVxpiE4Oh60XeuxuWACUNV/y0iXwHneW9doqrfxLZZ\nxphEEKeb7B1yXI1LuFd6JwMFBIelKvRzwI0x8ePmatF/rsYlnLOk7gfeBFIJ3vFwjIj8JdYNM8ZU\nfJE+Wa6icjUu4exhXA20U9WtACLyGDAfGBbLhhljKj5HTwbynatxCSdhrC1SLsV7zxhjIpJoew7h\ncjUuZd188BmCxyx+Ab71boOrQCeC91o3xpiIOLpe9J2rcSlrD2PvmVDfAtNC3v8ids0xxiQSV7ek\n/eZqXMq6+WCJj3EyxphocXWs3m+uxiWcs6SaiMhYEVkkIkv3vuLROGNMxRaNs4FEpIuILPHWTYNL\nmH+miPwqIrnea0hMOxUFrsYlnIPe/wIeJfhUpguAa/FuE2KMMZFIjnDoxbtt0SjgXCAA5IjIJFVd\nUqTop6raI6LK4sjVuIRzEd4RqjoDQFVXqOoQgonDGGMiEoW7smYBy1R1taoWAGOBniVVFaMuxISr\ncQknYezwstUKEfmTiHQHqh1MJcYYU5IoDL2kAWtCpvO894o6WUQWiMg0EWkV7X5Em6txCWdI6g7g\nSOBW4DGgBnBdGJ8zxpgylTXyEvg2m8C3UTmD/yugoapuFZELgHeBZtH44lhxNS7h3HzwS+/PTex7\niJIxxkSsrJvsNTj+RBocf2LhdO74F0oqlg80DP2Y914hVd0c8vd0EfmHiNRS1V/K2eyYczUuZV24\nN5EyDm6r6sWlzTPGmHBE4XKDHCBDRNIJ3oHiMqDf/nVIXVVd7/2dBYjLyQLcjUtZexijImtv+W3d\nuduvquNqzewRfjchbmr2TZzLetb/51q/m3DIiPQCNVXdLSIDgZkEj8m+oqqLReTm4GwdDfQRkf4E\n77i9DegbYbNjztW4lHXh3ocRtdgYYw4gGs9KUNX3geZF3nsx5O/nCT4I7pDhalzCfR6GMcZEnau3\nwPCbq3GxhGGM8Y2rt8Dwm6txCTthiMhhqrojlo0xxiQWV1eMfnM1LuHcSypLRL4GlnnTJ4jIczFv\nmTGmwktOkrBficTVuIRzbGUk0A34GUBVFwJnx7JRxpjEEIVbYFRIrsYlnCGpJFVdXeQgTGKc92qM\niamyLlBLZK7GJZyEsca7qENFJBkYBNjtzY0xEYvG6aMVkatxCSdh9Cc4LNUQWA984L1njDERcXRD\n2neuxiWce0n9SPCycmOMiSpXh1785mpcDpgwROQlSrinlKreFJMWGWMShqPrRd+5GpdwhqQ+CPn7\ncOAi9r/PujHGlEuCnS0bNlfjEs6Q1LjQaRF5HZgTsxYZYxKGq0MvfnM1LuW5NcixQN1oN8QYk3gc\nXS/6ztW4hHMMYyP7jmEkAb8Af45lo4wxicHVoRe/uRqXMhOGBK/WO4F9T2rao6qlPlTJGGMORrKr\nm9I+czUuZV4f4iWH91R1t/eyZGGMiZokCf9VGhHpIiJLRGSpiAwuo1ymiBSIiPNPC3U1LuFcULhA\nRNqFUc4YYw6KiIT9KuXzSQSfDtoZOA7oJyItSin3ODAjht2JGlfjUtYzvVNUdRfQDsgRkRXAFkAI\n7ny0D6cCY4wpTRTG6rOAZaq6GkBExgI9gSVFyg0C3gYyI64xDlyNS1nHMLKB9kCPg26qMcaEIQpD\n9Wnsf11YHsGVZUgdkgr0UtWzvfviOc/VuJSVMARAVVccZEONMSYsZV1vsHz+Fyyf/0U0qhkBhI7h\nu3lEOYSE3nqhAAAa9UlEQVSrcSkrYdQRkTtLm6mqTx9Ew3yzcEEut/7perZv38F5nbvw2PCSmz3i\nqeGMeeNfpCSn8NgTT3P2uecDUFBQwJ/vupXPZn9KcnIy9w19hK49esWzC2FLpL62PfZoRg86g8Mr\nJTMjdw33vPplsTLJScILA06jbePaJCcJb36ynKcmLgLg/YcvoF7NI9i2YzeK0v2R9/l5k5sPlFww\nP5cBN13H9u3b6dT5Ah5/6pkSy/39ycd547VXSUlJ4fGnnuHc8zoB0Lvnhaxfv57du3Zx8qmn8fcR\no5x5ZnRZQy/N2p9Es/YnFU7P+NfIkorlE7wx6l4N2HdW514dgbHeWZ+1gQtEpEBVJ5ev1bHnalzK\nShjJQFUOgWxclnvvGMiI50fTrkMml/XuzkcfzOQc7x/SXkuXLGbSxPF8Pu9rAvl59O7RhewFixER\nnnlyGHWOqcsX878FYOMvv/jRjbAkUl9H3nQK/Z+fzVcrNjDx/k6c1zaNDxbs/++h9ynHUjklmaw7\nJ3J45WTmj+jNuNkrWLNhCwDXPPMxC1f97EfzD8qdt97Ccy+8RIeOmfTp1ZUPZs3gvPM771fm+yWL\nmThhPDkLviU/P4+eF3Zi/jffIyK89p+3qFq1KgBX9buEiRPGc3GfS/3oSjFRyFs5QIaIpANrCd4o\ntV9oAVVtvK8+eRWY4nKyAHfjUlbCWKuqj5S/vf5bv34dmzdtpl2H4PGcvv2uZPrUScVWotOnTeGi\n3n1JSUmhYXojGjfJIHdeNh0yT2TM6/9ibu63hWVr1qoV1z6EK5H6WveoKlStUomvVmwAYMzHy+me\nlV4sYajCEYelkJQkHFE5hR27dvP71oLC+a5eHBVq/bp1bN68iQ4dg8v1ssuvYtqUScUSxrQpk+nd\nJ7hc09Mb0SQjg69ysumYdWJhsigoKKCgYKczexcASRFuj6rqbhEZCMwkeNbnK6q6WERuDs7W0UU/\nElGFceJqXA54DONQti6QT2paWuF0/dQ01gYCxcqtXZtPZta+Xbz69VNZGwjw+2+/ATDskaF8NucT\njm2cweNPPUvtOnVi3/iDlEh9Ta11BPk/by2czv95C6m1jihW7p25q+iW1ZBVL/ejSuVk7n31S37b\nurNw/uiBZ1Cwew+TvljN8AkL4tL2gxUoslzT0hqwNlB0ZAHWBvLJOjFkuaamEQgpd3GPC8j9ah7n\nd+pCr4v7xLbRByEauUtV3weaF3nvxVLKXhd5jbHnalzKug7j3PCbVjHt2rWLQH4eJ558Kh/OzqZj\nZhYP3n+P382KiYrY18ymddi1W2l0/RhaDRjP7T1b07BOcGv7j898TOadEzlvyDRObVWXy85o4nNr\nY+udydNZuiqfHTt28MnHH/ndnELRuECtInI1LqUmDFWNeABbRNJFZLGIvCoi34vIGyJyrojM8aY7\nRlpHWeqlppGfl1c4vTaQT/3U1GLl6tffv1zAK1fr6KM54sgjCw/89rioD18vdHNLNJH6GvhlKw1q\nH1k4nXb0kQR+2VqsXN/TmzBrfh6qsOH37cxdsp4OTWoDsO7XbQBs3bGLcbNXkJnh3p4UQGqR5Zqf\nn0f91LRi5eqnppEXulzz80gtUq5y5cpc2K07701xZ/g+OUnCfiUSV+MSj0fHNgGeVNXmQAugn6qe\nBtwD3B/LiuvWrUf16tXJnZeNqjLuzTfo0rX4ZSVdunZj4oRx7Ny5k9U/rGLVyhW07xg8LbnzBd2Y\n8+nHAHzy3w9p1qJVLJtcbonU1/W/buP3rTvpmBFc+V9+VgZTs1cXK7fmp82c2bo+EDyWkdXsGL7P\n/42kJKFW1cMASEkWLuzQkG/XbIxfBw5C3Xr1qF69Bl/lBJfr2DGv07Vb8eV6YbfuTHg7uFx/+GEV\nK1esoENmFlu2bGH9unVAcC9yxvT3aNq82AW/vkkSCfuVSFyNS3lub36wVqnqd97f3wIfen9/DaTH\nuvLhT49k0J+uZ/uOHZzXqQvnegcLZ7w3lYULcrn3vqE0b9GKnhddwqmZbaiUUoknnn6u8MDgAw8/\nxoAb/8iQP99F7dp1GPnCy7FucrklUl9vf+lzRg88g8MrJzMjN49Z3gHvCzv+gXaNa/PYW/P55/uL\nGT3wdOY9E7xFzmsfLuW7NRupUjmZyQ90JiU5ieQk4b+LAvzfrO/97E6Znhrx3H6n1Z7XqQsQPIFh\nwfxc/jLkQVq0bMVFvS8hq93xVKpUib8/Gzx1duuWLVzWpxcFBTvZs2cPp59xFtffeLPPPdonwfJA\n2FyNi8TyfoLeKV1TVLWNN7331K13is4L+Yze85cHCqdPPf1MTj39zJi10cTHH655ze8mxM36/1zr\ndxNiYvanHzPn008Kpx9/7BFUtdyrNhHRV0vYMyzNtVnpEdV3qHA5LvHYwyirIyXOu/e+oTFqijGm\nvE4/4yxOP+OswunHH4v8rHuXTvF1iatxiUfC0FL+LmnaGJNA3Fwt+s/VuMQ0YXh3SmwTMn1dafOM\nMYkn0Q5mh8vVuMRjD8MYY0rk5mrRf67GxRKGMcY3jm5I+87VuFjCMMb4xtWDu35zNS6WMIwxvonH\nlcOHIlfjYgnDGOMbV7ek/eZqXCxhGGN84+rZQH5zNS6WMIwxvnF16MVvrsbF1XYZYxKAiIT9KuM7\nuojIEhFZKiKDS5jfQ0QWish8EckWkVNj2qkocDUutodhjPFNpAMvIpIEjCL4/J4AkCMik1R1SUix\nD/Y+elREWgNvAS0jrDqmXI2L7WEYY3wjEv6rFFnAMlVdraoFwFigZ2gBVQ19WEpVYE8s+hJNrsbF\n9jCMMb6J9NnVQBqwJmQ6j+DKcj8i0gsYBtQBukZaaay5GhfbwzDG+CYKW9JhUdV3VbUl0At4NBpt\njyVX42J7GMYY30gZW9KLcj7j65zPD/QV+UDDkOkG3nslUtU5ItJYRGpF4zHUseJqXGL6AKXyEBH9\naVOB380wUWYPUKp4alRJjvgBStO+WR92+a7H1y1Wn4gkA98TPLi7Fsgm+BjoxSFlmqjqCu/v9sAk\nVf1Dedsday7HxfYwjDG+iXSsXlV3i8hAYCbBIfZXVHWxiNwcnK2jgd4icjWwE9gGXBphs2PO1bhY\nwjDG+CYaFzSr6vtA8yLvvRjy9xPAE5HXFD+uxsUShjHGN67eAsNvrsbFEoYxxjdJbq4XfedqXCxh\nGGN8U9bZQInM1bhYwjDG+MbRkRffuRoXSxjGGN+4uiXtN1fjYgnDGOMbV8fq/eZqXCxhGGN84+qW\ntN9cjYslDGOMb1wdq/ebq3GxhGGM8Y2j60XfuRoXJxPGHsfubxUrVSol+92EuNk47nq/mxA3NTv9\nze8mHDJcvUDNb67GxcmEYYxJDG6uFv3nalwsYRhjfFPWM6kTmatxsYRhjPGNo+tF37kaF0sYxhjf\nOLpe9J2rcbGEYYzxj6trRr85Ghd7prcxxjdyEP+V+h0iXURkiYgsFZHBJcy/XEQWeq85ItI6pp2K\nAlfjYnsYxhjfRDpWLyJJwCiCjyINADkiMklVl4QUWwmcoaq/iUgX4CXgpMhqji1X42J7GMYY38hB\nvEqRBSxT1dWqWgCMBXqGFlDVL1T1N2/yCyAtqp2IAVfjYgnDGOOfyNeMacCakOk8yl7x3QBMj6TJ\nceFoXGxIyhjjm3jeZE9EzgauBU6LW6Xl5GpcLGEYY3xT1lj9vLmzmffF7AN9RT7QMGS6gfdekXqk\nDTAa6KKqGw+6oXHmalxEHbtvk4jo+t93+t2MuEike0lVSkmc0c9EuZfU9v/ej6qWe1NYRHTB6t/D\nLt82vXqx+kQkGfie4MHdtUA20E9VF4eUaQh8CFylql+Ut73x4nJcbA/DGOOfCEdeVHW3iAwEZhI8\nJvuKqi4WkZuDs3U08ABQC/iHBO+5UaCqWZHVHGOOxsX2MHxkexgVk+1hhEdEdNGaTWGXb/OHahHV\nd6hwOS62h2GM8U2FX/uXk6txsYRhjPGPq2tGvzkaF0sYxhjfuPrsar+5GhdLGMYY37h6G2+/uRoX\nSxjGGN84ul70natxsYRhjPGPq2tGvzkaF0sYxhjfuDpW7zdX41LhT45ftGA+Z53cnpPbtWLI4LtK\nLffs34dzUttWnNaxNR9/OAuALZs3c+5pmZx3ehbnnpZJq2NTGfqXe+LV9IO2YH4uJ2e2pV3rFgy+\n+45Sy/39ycdpe3xzOrY9jg8/mFn4/sU9L+S0kzpwUscTuOO2W3DtGp1Q83NzyWzXhtatmnH3nbeX\nWu7J4cM4vmVT2rZuyQezZh70513Qtmldsl+6gUWv3cyTt5xXYpmU5CT+eU9Xsl+6gbkvXsdpbRoe\n1Of9IhL+K5G4GpcKnzDuvWMgz4wazdz537Fy+TL+G7KC3Gvp94uZPPFt5sxbxJgJUxh8562oKkdW\nrcqHc3L4YHY2H87JoUHDhnTreZEPvQjPHbfewqgXXmL+10tYvnwpH8yaUazM90sWM3HCeOYt+JYJ\nk6Zx520DCxPDv//zFnO++Iov5i1kw48/MnHC+Hh3IWy3DuzPC6Nf4evvlrJ82VJmzSze1yWLFzNh\n/Fss+Hoxk6ZM57ZBAwr7Gs7nXTHy9i70f2oaba55kYwGtTiv47HFylzXtS2qStaNL9P93rE83v/c\ng/q8X6JwG+8KydW4VOiE8eP6dWzZvIl2HToCcEm/K5g+bXKxcu9Pm0Kv3peSkpJCw/RGNG6SQe68\nnP3KrFi2lJ83bCDrpFPi0vaDtX7dOjZv3kSHjpkA9Lv8KqZNmVSs3LQpk+ndpy8pKSmkpzeiSUYG\n83KyAahatSoABQUF7CzYiTi6WbfO62vHzGBfL7/yaqZMerdYuSlTJtGn72XBvjZqREZGU3Kys8P+\nvAvq1jySqlUO46vv1wIwZuY3dD+tebFyLRvV5pP5qwHY8NtWftu8nfbN6oX9ed+4umb0m6NxiXvC\nEJEaItI/HnWtDQSon9qgcDo1rQFrA8Vu2MjaQIDUtH3l6qWmsm7t/uUmvTOenhf3iV1jIxQI5JOa\ntu9292lpDQiU0NdAIJ+0BiExSU3bLyYX9biAjEapVKtWnV6O9jeQn09ayPIqta/5+TRo8IfC6dTU\nNAKB/LA/74LU2tXI37DvRnT5P/1Oau2qxcotWvEjXU9pSlKSkF6vBu2a1aPBMdXD/rxfkkTCfiUS\nV+Pixx5GTWCAD/VG5N0Jb3FRn75+NyPmJk6ezrJV+ezcsYNPPv7I7+aYML02fSGBDZuY88K1PDHg\nPOZ+k8fuPe4eg9rL0Q1p37kaFz/OkhoGNBaRXGCWqhZ7OHm01E9NJZC/76FTgfw86qcWf+hUsFxe\n4fTa/Hzq1d9X7ttvFrF7925an9AuVk2NWGpqGvl5+/qQn59Hagl9Lalc0ZhUrlyZC7p1Z9qUyZx1\n9rlFv8J3qWlp5OXtW66l9rWUcuF+3gWBDZtoUKd64XRaneoENmwuVm7PHmXwCx8WTn808iqWrfmF\n3zZvD+vzvkm0TBAuR+Pixx7Gn4EVqto+lskC4Ji69ahWvQa583JQVca/+R+6XNi9WLnOF3bj3Qlv\nsXPnTlb/sIpVK1fQ3jsWADDx7XHO713UrVeP6tVrMC8nG1XlzTGvc2G3HsXKXditOxPeHsfOnTv5\n4YdVrFyxgo6ZWWzZsoX169YBsGvXLmZMf49mzVvEuxthqef1NSc72Ncxb/ybbj16FivXrVsP3h43\nNtjXVatYsWI5mVlZYX/eBes3buH3LTvo2KI+AJd3Op6pny0tVu7wyilUOSy4/XdOh0YU7NrD0jU/\nh/15v8hB/JdIXI1Lhb8O4/G/j+TW/tezY/sOzu3UhXPO7wzAjOlTWTQ/l3vuG0rzFq3ocVEfTs88\ngUqVUhj+9Mj9DvhOmfgOY94ufgDZNX8f8Rz9b7qO7du306nzBZzfqQsA06dNYf78XO4b8iAtWrbi\not6XkNnueCpVqsTTz45CRNi6ZQt9+/SioGAne/bs4fQzzuL6G2/2uUelGzHyeW66/o9s37Gdzl0u\npFPnYF+nTZ3C/NyvGDL0IVq2akXvSy6lXZtWVKpUiWef+0fhci3t8y66/dkZjB7cjcMrpzDjyxXM\nylkJwIUnZ9CuWX0ee202x9Q8gsnD+7F7zx4CGzZx/bDJB/y8CxLs0ETYXI1L3J+HISLpwBRVbVPK\nfL37z0MKp085/UxOPf3MeDUvrux5GBVTRX0exu6NK9nz66p90z98FPHzMFb9tC3s8sfWqZIwz8Nw\nNS5+7GFsAqqVVeCe+4bGqSnGmHAl12xMcs3GhdO7f4jCSRFRWM2JSBdgBPueLDe8yPzmwKtAe+A+\nVX068lpjzNG4xH2zT1V/AT4TkUUiMvyAHzDGVFiRjtWLSBIwCugMHAf0E5GiB99+BgYBT8ayL9Hk\nalx8OYahqlf6Ua8xxi1RGKvPApap6urg98lYoCewZG8BVd0AbBCRbhHXFieuxiVxBpaNMc6JwvUG\nacCakOk8771DmqtxqfBnSRlj3FXWlvTcOZ8wd86n8WuMQ1yNiyUMY4xvyrpf2Smnn8Upp59VOP3M\nE4+VVCwfaBgy3cB775DmalxsSMoY45soDL3kABkiki4ilYHLgOJ3GN2/Sue5GhfbwzDG+CbSg7uq\nultEBgIz2Xf66GIRuTk4W0eLSF1gHsHT+feIyG1AK1V16B4p+3M1LpYwjDG+icatLVT1faB5kfde\nDPl7PfCHop9zmatxsYRhjPHPITFA5ANH42IJwxjjG0fXi75zNS6WMIwxvnH1Jnt+czUuljCMMb5J\ntNuWh8vVuFjCMMb4x831ov8cjYslDGOMbxxdL/rO1bhYwjDG+MbVsXq/uRoXSxjGGN8kubpm9Jmr\ncbFbgxhjjAmL7WEYY3zj6Ia071yNiyUMY4xvXD191G+uxsUShjHGN65uSfvN1bhYwjDG+MbR9aLv\nXI2LJQxjjH9cXTP6zdG4WMIwxvjG1bF6v7kaFzutFvhs9id+NyFuZn/6sd9NiJtPP/nY7ybEze6N\nK/1uQrmIhP9KJK7GxRIG8HlCJYzE6WsiJYw9v67yuwnlEoVHkVZIrsbFhqSMMf5JtEwQLkfjYgnD\nGOMbV8fq/eZqXERV/W7DfkTErQYZY0qlquVes4nID0D6QXxktao2Km99hwqX4+JcwjDGGOMmO+ht\njDEmLJYwjDHGhMUShjHGmLAkbMIQkSTv/26ejhBlidJPY0zsJGzCUNU93p/pIpKSACvUVAARsVOp\nD3EiUllEWnl/nysi9f1uk0kMCbfyEJFTgIaqOlZEBgHXAt8Ai0VkeEgiqTBEZCDQWUS+BQIi8qKq\n7vC7XbEkIj2BLcAXqrrZ7/ZEWUNghIisB2oBV/vcHpMgEnEPoyYwTET+CpwMXAL8C6gCPL53qKqi\nEJFewKXAVcCJQLMESBaXAf8EzgdmiEgtn5sUVaq6HFgE9ASmq+rPIpKcAHvJxmcVauUYDlWdBtwE\nXBSc1BXAHOB1oC7QysfmxUINYATQCygA7gQQkWZ+NipWRKQhoMBpqjoYeAGYWdGSBsGEOAC4TkSu\nUNXdqqoiUtXvhpmKK+GGpABUdZaI3A/8S0T6quo4YJmIVANaEhyiqih+AP4PCKjq6QAicivQQETu\nV9UCPxsXTV6/rgCqAU+LSL6qvuHdPSBHRDqq6kZ/Wxkd3l7GchH5DXhURH4luEFwmog8oqq7/G2h\nqYgSMmEAqOokEbkKGCkiLYEFQFMg19+WRd1XwCRgj4icRXD8+xrgmgqWLHoBHQkOvd0AtAZOEpE5\nqvofEdkFHAVUiISxl6pOEZEC4AlgJ3C1JQsTKwl/axBvRTMBmArcoaqH5oMFyuCdRdPDe/0MPKmq\nX/vbqugRkTRgLjBLVa8XkcOB+wkmiMnAfyv6SlRE6gCo6k9+t8VUXAmfMABE5EyCN/D6we+2xJKI\nVAKoSHsWe4nIxcAo4C5VfdM7ffgJYA8wVFW3+tpAYyoASximwhCRrsAwYFhI0qhpW93GREfCHsMw\nFY+qThORPcBoEdmlquMBSxbGRIntYZgKR0TOB1ZUxONRxvjJEoYxxpiwJNyFe8YYY8rHEoYxxpiw\nWMIwxhgTFksYxhhjwmIJwxhjTFgsYVRAIrJbRHJF5GsRGefdKqO833WmiEzx/u4uIveWUbaGiPQv\nRx0Pisid4b5fpMyr3lXe4daVLiIV5rYoxsSTJYyKaYuqtlfV1gTvYPqnogUO8tkJCsEb3anqE2WU\nq0nwltuus3PJjSkHSxgV32wgw9uyXiIir3lb2A1E5HwR+VxE5nl7IkcAiEgXEVksIvOAwq13EblG\nRJ7z/j5GRN4RkQUiMl9ETiJ4W44m3t7NcK/c3SKS7ZV7MOS77heR70XkU6D5gTohIjd43zNfRMYX\n2Ws6X0RyvP519conicgTIvKlV/eNEUfSmARnCaNiEih8fvcFwN4hmKbAKG/PYyswBDhXVTsSvA36\nnSJyGDAa6Oq9X6/Id+/dOh8JfKyqbYH2wLfAn4Hl3t7NYO+K66aqmgW0AzqKyGki0p7gUwDbAF2B\nzDD6NEFVs1S1HbAEuD5kXrqqZgLdgH+KSGVv/q+qeiKQBdwkIulh1GOMKYXdS6piqiIie5/rMRt4\nBUgDflDVHO/9kwg+XfAzb3iqEsFbhLcAVobcVuMNoKSt83MIPnsCDd4uYFMJT7XrRHDrP5dgEjuS\nYNKqDkz0HhW7Q0Qmh9GnNt5jdY/yvmdGyLy3vHYsF5EVXh86Aa1F5BKvTHWv7mVh1GWMKYEljIpp\nq6q2D33DO2SxJfQtYKaqXlGk3AnevAMJ5ziAELxz7EtF6rgtjM8W9SrQQ1W/EZFrgDNLaYt40wIM\nUtVZReq2vQxjysmGpCqm0lb4oe9/AZwqIk0AROQIEWlKcLgnXUSO9cr1K+W7PsQ7wO0dL6gObCL4\neNS9ZhB85vSRXrlU70E/nwK9ROQw77G43cPoU1VgnfdMjyuKzLtEgpoAxwLfe3UP8IblEJGmIlKl\nhDgYY8JkexgVU2lb/4Xvq+oGEfkj8KZ33EKBIaq6TERuBt4TkS0Eh7SqlvBdtxO8jfj1wC6gv6p+\n6R1EXwRM945jtATmens4m4ArVXW+iLwFLALWA9lh9GmoV+5H4Ev2T0z/8+ZVA25W1Z0i8jLQCMj1\nhtx+BHodID7GmDLY3WqNMcaExYakjDHGhMUShjHGmLBYwjDGGBMWSxjGGGPCYgnDGGNMWCxhGGOM\nCYslDGOMMWGxhGGMMSYs/w+7wUtuUMzsmwAAAABJRU5ErkJggg==\n",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plot_confusion_matrix(cm, labels, title='Confusion matrix')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Confussion matrix columns represent the instances in a predicted class while rows represent the instances in an actual class. The diagonal elements represent the number of points for which the predicted label is equal to the true label, while off-diagonal elements are those that are mislabeled by the classifier. The higher the diagonal values of the confusion matrix the better, indicating many correct predictions."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, let's see the relation between [f1-score](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html) and the percentage by category"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def resume_data(labels,y_train,f1s):\n",
" c = Counter(y_train)\n",
" cont = dict(c)\n",
" tot = sum(list(cont.values()))\n",
" nlabels = len(labels)\n",
" d = {\n",
" \"category\" : [labels[i] for i in range(nlabels)],\n",
" \"percent\" : [cont[labels[i]]/tot for i in range(nlabels)],\n",
" \"f1-score\" : [f1s[i] for i in range(nlabels)]\n",
" }\n",
" \n",
" print(pd.DataFrame(d)) \n",
" print(\"total \\t\",tot) \n",
" return cont"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" category f1-score percent\n",
"0 b 0.903839 0.274738\n",
"1 e 0.959225 0.361334\n",
"2 m 0.902814 0.107984\n",
"3 t 0.903314 0.255945\n",
"total \t 295693\n"
]
}
],
"source": [
"f1s = metrics.f1_score(y_test, predicted, labels=labels, average=None)\n",
"cont3 = resume_data(labels,y_train,f1s)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Results analysis"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As a resume, results show a good accuracy (0.9238) with a good average level for precision, recall and f1-score (0.92) Analyzing these results by category, results are even better for the entertainment category ('e') with 0.96 for f1-score, 0.97 for recall and 0.95 for precision. I would like to highlight that best result for prediction corresponds to health category ('m') with 0.97, but with a recall of 0.85. Other categories show a more even behavior. \n",
"Analyzing confusion matrix results, the higher index of points predicted in their correct class for category 'e' with 0.9719. This category presents a misclassification index of 0.014 for technology category ('t') and lower indexes for the other categories.\n",
"On the contrary, category 'm' presents the worst hit rate with an indicator of 0.846, which has misclassification indexes of 0.062 with business category ('b'), of 0.0619 with category 'e' and of 0.03 with category 't'.\n",
"Analyzing the number of news by category, category 'e' presents the higher percentage, 36%, with 45625 news. On the other hand, category 'm' presents the lower percentage, 10.79%, with just 13709 news. Thus, category 'e' is more than three times bigger than category 'm'. Categories 'b' and 't' present similar number of news and percentage: 'b' has 34729 news with 27%, and 't' has 32663 news with a 25%. Both categories, 'b' and 't', are more than two times bigger than category 'm'. According to these, better results seem to correspond with higher percentage categories. In future experiments, I would try to confirm this hypothesis.\n",
"In this experience, we just trained the classification algorithm with one set of data, so we just have one set of results. Although the training set and the test set were selected by random, it is just a sample of the possible results. In future experiments, I would try to test which is the confidence of the experiment results. "
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.2"
}
},
"nbformat": 4,
"nbformat_minor": 1
}