# Cricpy takes a swing at the ODIs

No computer has ever been designed that is ever aware of what it’s doing; but most of the time, we aren’t either.” Marvin Minksy

“The competent programmer is fully aware of the limited size of his own skull. He therefore approaches his task with full humility, and avoids clever tricks like the plague” Edgser Djikstra

# Introduction

In this post, cricpy, the Python avatar of my R package cricketr, learns some new tricks to be able to handle ODI matches. To know more about my R package cricketr see Re-introducing cricketr! : An R package to analyze performances of cricketers

Cricpy uses the statistics info available in ESPN Cricinfo Statsguru. The current version of this package supports only Test cricket

You should be able to install the package using pip install cricpy and use the many functions available in the package. Please mindful of the ESPN Cricinfo Terms of Use

To know how to use cricpy see Introducing cricpy:A python package to analyze performances of cricketers. To the original version of cricpy, I have added 3 new functions for ODI. The earlier functions work for Test and ODI.

This post is also hosted on Rpubs at Cricpy takes a swing at the ODIs. You can also down the pdf version of this post at cricpy-odi.pdf

You can fork/clone the package at Github cricpy

Note: If you would like to do a similar analysis for a different set of batsman and bowlers, you can clone/download my skeleton cricpy-template from Github (which is the R Markdown file I have used for the analysis below). You will only need to make appropriate changes for the players you are interested in. The functions can be executed in RStudio or in a IPython notebook.

# The cricpy package

The data for a particular player in ODI can be obtained with the getPlayerDataOD() function. To do you will need to go to ESPN CricInfo Player and type in the name of the player for e.g Virat Kohli, Virendar Sehwag, Chris Gayle etc. This will bring up a page which have the profile number for the player e.g. for Virat Kohli this would be http://www.espncricinfo.com/india/content/player/253802.html. Hence, Kohli’s profile is 253802. This can be used to get the data for Virat Kohlis shown below

The cricpy package is a clone of my R package cricketr. The signature of all the python functions are identical with that of its clone ‘cricketr’, with only the necessary variations between Python and R. It may be useful to look at my post R vs Python: Different similarities and similar differences. In fact if you are familar with one of the lanuguages you can look up the package in the other and you will notice the parallel constructs.

You can fork/clone the package at Github cricpy

Note: The charts are self-explanatory and I have not added much of my owy interpretation to it. Do look at the plots closely and check out the performances for yourself.

## 1 Importing cricpy – Python

# Install the package
# Do a pip install cricpy
# Import cricpy
import cricpy.analytics as ca 

## 2. Invoking functions with Python package crlcpy

import cricpy.analytics as ca
ca.batsman4s("./kohli.csv","Virat Kohli")


# 3. Getting help from cricpy – Python

import cricpy.analytics as ca
help(ca.getPlayerDataOD)
## Help on function getPlayerDataOD in module cricpy.analytics:
##
## getPlayerDataOD(profile, opposition='', host='', dir='./data', file='player001.csv', type='batting', homeOrAway=[1, 2, 3], result=[1, 2, 3, 5], create=True)
##     Get the One day player data from ESPN Cricinfo based on specific inputs and store in a file in a given directory
##
##     Description
##
##     Get the player data given the profile of the batsman. The allowed inputs are home,away or both and won,lost or draw of matches. The data is stored in a <player>.csv file in a directory specified. This function also returns a data frame of the player
##
##     Usage
##
##     getPlayerDataOD(profile, opposition="",host="",dir = "../", file = "player001.csv",
##     type = "batting", homeOrAway = c(1, 2, 3), result = c(1, 2, 3,5))
##     Arguments
##
##     profile
##     This is the profile number of the player to get data. This can be obtained from http://www.espncricinfo.com/ci/content/player/index.html. Type the name of the player and click search. This will display the details of the player. Make a note of the profile ID. For e.g For Virender Sehwag this turns out to be http://www.espncricinfo.com/india/content/player/35263.html. Hence the profile for Sehwag is 35263
##     opposition      The numerical value of the opposition country e.g.Australia,India, England etc. The values are Australia:2,Bangladesh:25,Bermuda:12, England:1,Hong Kong:19,India:6,Ireland:29, Netherlands:15,New Zealand:5,Pakistan:7,Scotland:30,South Africa:3,Sri Lanka:8,United Arab Emirates:27, West Indies:4, Zimbabwe:9; Africa XI:405 Note: If no value is entered for opposition then all teams are considered
##     host            The numerical value of the host country e.g.Australia,India, England etc. The values are Australia:2,Bangladesh:25,England:1,India:6,Ireland:29,Malaysia:16,New Zealand:5,Pakistan:7, Scotland:30,South Africa:3,Sri Lanka:8,United Arab Emirates:27,West Indies:4, Zimbabwe:9 Note: If no value is entered for host then all host countries are considered
##     dir
##     Name of the directory to store the player data into. If not specified the data is stored in a default directory "../data". Default="../data"
##     file
##     Name of the file to store the data into for e.g. tendulkar.csv. This can be used for subsequent functions. Default="player001.csv"
##     type
##     type of data required. This can be "batting" or "bowling"
##     homeOrAway
##     This is vector with either or all 1,2, 3. 1 is for home 2 is for away, 3 is for neutral venue
##     result
##     This is a vector that can take values 1,2,3,5. 1 - won match 2- lost match 3-tied 5- no result
##     Details
##
##     More details can be found in my short video tutorial in Youtube https://www.youtube.com/watch?v=q9uMPFVsXsI
##
##     Value
##
##     Returns the player's dataframe
##
##     Note
##
##     Maintainer: Tinniam V Ganesh <tvganesh.85@gmail.com>
##
##     Author(s)
##
##     Tinniam V Ganesh
##
##     References
##
##     http://www.espncricinfo.com/ci/content/stats/index.html
##
##
##     getPlayerDataSp getPlayerData
##
##     Examples
##
##
##     ## Not run:
##     # Both home and away. Result = won,lost and drawn
##     sehwag =getPlayerDataOD(35263,dir="../cricketr/data", file="sehwag1.csv",
##     type="batting", homeOrAway=[1,2],result=[1,2,3,4])
##
##     # Only away. Get data only for won and lost innings
##     sehwag = getPlayerDataOD(35263,dir="../cricketr/data", file="sehwag2.csv",
##     type="batting",homeOrAway=[2],result=[1,2])
##
##     # Get bowling data and store in file for future
##     malinga = getPlayerData(49758,dir="../cricketr/data",file="malinga1.csv",
##     type="bowling")
##
##     # Get Dhoni's ODI record in Australia against Australua
##     dhoni = getPlayerDataOD(28081,opposition = 2,host=2,dir=".",
##     file="dhoniVsAusinAusOD",type="batting")
##
##     ## End(Not run)

The details below will introduce the different functions that are available in cricpy.

## 4. Get the ODI player data for a player using the function getPlayerDataOD()

Important Note This needs to be done only once for a player. This function stores the player’s data in the specified CSV file (for e.g. kohli.csv as above) which can then be reused for all other functions). Once we have the data for the players many analyses can be done. This post will use the stored CSV file obtained with a prior getPlayerDataOD for all subsequent analyses

import cricpy.analytics as ca
#sehwag=ca.getPlayerDataOD(35263,dir=".",file="sehwag.csv",type="batting")
#kohli=ca.getPlayerDataOD(253802,dir=".",file="kohli.csv",type="batting")
#jayasuriya=ca.getPlayerDataOD(49209,dir=".",file="jayasuriya.csv",type="batting")
#gayle=ca.getPlayerDataOD(51880,dir=".",file="gayle.csv",type="batting")

Included below are some of the functions that can be used for ODI batsmen and bowlers. For this I have chosen, Virat Kohli, ‘the run machine’ who is on-track for breaking many of the Test & ODI records

## 5 Virat Kohli’s performance – Basic Analyses

The 3 plots below provide the following for Virat Kohli

1. Frequency percentage of runs in each run range over the whole career
2. Mean Strike Rate for runs scored in the given range
3. A histogram of runs frequency percentages in runs ranges
import cricpy.analytics as ca
import matplotlib.pyplot as plt
ca.batsmanRunsFreqPerf("./kohli.csv","Virat Kohli")

ca.batsmanMeanStrikeRate("./kohli.csv","Virat Kohli")

ca.batsmanRunsRanges("./kohli.csv","Virat Kohli")

## 6. More analyses

import cricpy.analytics as ca
ca.batsman4s("./kohli.csv","Virat Kohli")

ca.batsman6s("./kohli.csv","Virat Kohli")

ca.batsmanDismissals("./kohli.csv","Virat Kohli")

ca.batsmanScoringRateODTT("./kohli.csv","Virat Kohli")

## 7. 3D scatter plot and prediction plane

The plots below show the 3D scatter plot of Kohli’s Runs versus Balls Faced and Minutes at crease. A linear regression plane is then fitted between Runs and Balls Faced + Minutes at crease

import cricpy.analytics as ca
ca.battingPerf3d("./kohli.csv","Virat Kohli")

## Average runs at different venues

The plot below gives the average runs scored by Kohli at different grounds. The plot also the number of innings at each ground as a label at x-axis.

import cricpy.analytics as ca
ca.batsmanAvgRunsGround("./kohli.csv","Virat Kohli")

## 9. Average runs against different opposing teams

This plot computes the average runs scored by Kohli against different countries.

import cricpy.analytics as ca
ca.batsmanAvgRunsOpposition("./kohli.csv","Virat Kohli")

## 10 . Highest Runs Likelihood

The plot below shows the Runs Likelihood for a batsman. For this the performance of Kohli is plotted as a 3D scatter plot with Runs versus Balls Faced + Minutes at crease. K-Means. The centroids of 3 clusters are computed and plotted. In this plot Kohli’s highest tendencies are computed and plotted using K-Means

import cricpy.analytics as ca
ca.batsmanRunsLikelihood("./kohli.csv","Virat Kohli")

# A look at the Top 4 batsman – Kohli, Jayasuriya, Sehwag and Gayle

The following batsmen have been very prolific in ODI cricket and will be used for the analyses

1. Virat Kohli: Runs – 10232, Average:59.83 ,Strike rate-92.88
2. Sanath Jayasuriya : Runs – 13430, Average:32.36 ,Strike rate-91.2
3. Virendar Sehwag :Runs – 8273, Average:35.05 ,Strike rate-104.33
4. Chris Gayle : Runs – 9727, Average:37.12 ,Strike rate-85.82

The following plots take a closer at their performances. The box plots show the median the 1st and 3rd quartile of the runs

## 12. Box Histogram Plot

This plot shows a combined boxplot of the Runs ranges and a histogram of the Runs Frequency

import cricpy.analytics as ca
ca.batsmanPerfBoxHist("./kohli.csv","Virat Kohli")

ca.batsmanPerfBoxHist("./jayasuriya.csv","Sanath jayasuriya")

ca.batsmanPerfBoxHist("./gayle.csv","Chris Gayle")

ca.batsmanPerfBoxHist("./sehwag.csv","Virendar Sehwag")

## 13 Moving Average of runs in career

Take a look at the Moving Average across the career of the Top 4 (ignore the dip at the end of all plots. Need to check why this is so!). Kohli’s performance has been steadily improving over the years, so has Sehwag. Gayle seems to be on the way down

import cricpy.analytics as ca
ca.batsmanMovingAverage("./kohli.csv","Virat Kohli")

ca.batsmanMovingAverage("./jayasuriya.csv","Sanath jayasuriya")

ca.batsmanMovingAverage("./gayle.csv","Chris Gayle")

ca.batsmanMovingAverage("./sehwag.csv","Virendar Sehwag")

## 14 Cumulative Average runs of batsman in career

This function provides the cumulative average runs of the batsman over the career. Kohli seems to be getting better with time and reaches a cumulative average of 45+. Sehwag improves with time and reaches around 35+. Chris Gayle drops from 42 to 35

import cricpy.analytics as ca
ca.batsmanCumulativeAverageRuns("./kohli.csv","Virat Kohli")

ca.batsmanCumulativeAverageRuns("./jayasuriya.csv","Sanath jayasuriya")

ca.batsmanCumulativeAverageRuns("./gayle.csv","Chris Gayle")

ca.batsmanCumulativeAverageRuns("./sehwag.csv","Virendar Sehwag")

## 15 Cumulative Average strike rate of batsman in career

Sehwag has the best strike rate of almost 90. Kohli and Jayasuriya have a cumulative strike rate of 75.

import cricpy.analytics as ca
ca.batsmanCumulativeStrikeRate("./kohli.csv","Virat Kohli")

ca.batsmanCumulativeStrikeRate("./jayasuriya.csv","Sanath jayasuriya")

ca.batsmanCumulativeStrikeRate("./gayle.csv","Chris Gayle")

ca.batsmanCumulativeStrikeRate("./sehwag.csv","Virendar Sehwag")

## 16 Relative Batsman Cumulative Average Runs

The plot below compares the Relative cumulative average runs of the batsman . It can be seen that Virat Kohli towers above all others in the runs. He is followed by Chris Gayle and then Sehwag

import cricpy.analytics as ca
frames = ["./sehwag.csv","./gayle.csv","./jayasuriya.csv","./kohli.csv"]
names = ["Sehwag","Gayle","Jayasuriya","Kohli"]
ca.relativeBatsmanCumulativeAvgRuns(frames,names)

## Relative Batsman Strike Rate

The plot below gives the relative Runs Frequency Percentages for each 10 run bucket. The plot below show Sehwag has the best strike rate, followed by Jayasuriya

import cricpy.analytics as ca
frames = ["./sehwag.csv","./gayle.csv","./jayasuriya.csv","./kohli.csv"]
names = ["Sehwag","Gayle","Jayasuriya","Kohli"]
ca.relativeBatsmanCumulativeStrikeRate(frames,names)

## 18. 3D plot of Runs vs Balls Faced and Minutes at Crease

The plot is a scatter plot of Runs vs Balls faced and Minutes at Crease. A 3D prediction plane is fitted

import cricpy.analytics as ca
ca.battingPerf3d("./kohli.csv","Virat Kohli")

ca.battingPerf3d("./jayasuriya.csv","Sanath jayasuriya")

ca.battingPerf3d("./gayle.csv","Chris Gayle")

ca.battingPerf3d("./sehwag.csv","Virendar Sehwag")

## 3D plot of Runs vs Balls Faced and Minutes at Crease

From the plot below it can be seen that Sehwag has more runs by way of 4s than 1’s,2’s or 3s. Gayle and Jayasuriya have large number of 6s

import cricpy.analytics as ca
frames = ["./sehwag.csv","./kohli.csv","./gayle.csv","./jayasuriya.csv"]
names = ["Sehwag","Kohli","Gayle","Jayasuriya"]
ca.batsman4s6s(frames,names)

## 20. Predicting Runs given Balls Faced and Minutes at Crease

A multi-variate regression plane is fitted between Runs and Balls faced +Minutes at crease.

import cricpy.analytics as ca
import numpy as np
import pandas as pd
BF = np.linspace( 10, 400,15)
Mins = np.linspace( 30,600,15)
newDF= pd.DataFrame({'BF':BF,'Mins':Mins})
kohli= ca.batsmanRunsPredict("./kohli.csv",newDF,"Kohli")
print(kohli)
##             BF        Mins        Runs
## 0    10.000000   30.000000    6.807407
## 1    37.857143   70.714286   36.034833
## 2    65.714286  111.428571   65.262259
## 3    93.571429  152.142857   94.489686
## 4   121.428571  192.857143  123.717112
## 5   149.285714  233.571429  152.944538
## 6   177.142857  274.285714  182.171965
## 7   205.000000  315.000000  211.399391
## 8   232.857143  355.714286  240.626817
## 9   260.714286  396.428571  269.854244
## 10  288.571429  437.142857  299.081670
## 11  316.428571  477.857143  328.309096
## 12  344.285714  518.571429  357.536523
## 13  372.142857  559.285714  386.763949
## 14  400.000000  600.000000  415.991375

The fitted model is then used to predict the runs that the batsmen will score for a given Balls faced and Minutes at crease.

## 21 Analysis of Top Bowlers

The following 4 bowlers have had an excellent career and will be used for the analysis

1. Muthiah Muralitharan:Wickets: 534, Average = 23.08, Economy Rate – 3.93
2. Wasim Akram : Wickets: 502, Average = 23.52, Economy Rate – 3.89
3. Shaun Pollock: Wickets: 393, Average = 24.50, Economy Rate – 3.67
4. Javagal Srinath : Wickets:315, Average – 28.08, Economy Rate – 4.44

How do Muralitharan, Akram, Pollock and Srinath compare with one another with respect to wickets taken and the Economy Rate. The next set of plots compute and plot precisely these analyses.

## 22. Get the bowler’s data

This plot below computes the percentage frequency of number of wickets taken for e.g 1 wicket x%, 2 wickets y% etc and plots them as a continuous line

import cricpy.analytics as ca
#akram=ca.getPlayerDataOD(43547,dir=".",file="akram.csv",type="bowling")
#murali=ca.getPlayerDataOD(49636,dir=".",file="murali.csv",type="bowling")
#pollock=ca.getPlayerDataOD(46774,dir=".",file="pollock.csv",type="bowling")
#srinath=ca.getPlayerDataOD(34105,dir=".",file="srinath.csv",type="bowling")

## 23. Wicket Frequency Plot

This plot below plots the frequency of wickets taken for each of the bowlers

import cricpy.analytics as ca
ca.bowlerWktsFreqPercent("./murali.csv","M Muralitharan")

ca.bowlerWktsFreqPercent("./akram.csv","Wasim Akram")

ca.bowlerWktsFreqPercent("./pollock.csv","Shaun Pollock")

ca.bowlerWktsFreqPercent("./srinath.csv","J Srinath")

## 24. Wickets Runs plot

The plot below create a box plot showing the 1st and 3rd quartile of runs conceded versus the number of wickets taken. Murali’s median runs for wickets ia around 40 while Akram, Pollock and Srinath it is around 32+ runs. The spread around the median is larger for these 3 bowlers in comparison to Murali

import cricpy.analytics as ca
ca.bowlerWktsRunsPlot("./murali.csv","M Muralitharan")

ca.bowlerWktsRunsPlot("./akram.csv","Wasim Akram")

ca.bowlerWktsRunsPlot("./pollock.csv","Shaun Pollock")

ca.bowlerWktsRunsPlot("./srinath.csv","J Srinath")

## 25 Average wickets at different venues

The plot gives the average wickets taken by Muralitharan at different venues. McGrath best performances are at Centurion, Lord’s and Port of Spain averaging about 4 wickets. Kapil Dev’s does good at Kingston and Wellington. Anderson averages 4 wickets at Dunedin and Nagpur

import cricpy.analytics as ca
ca.bowlerAvgWktsGround("./murali.csv","M Muralitharan")

ca.bowlerAvgWktsGround("./akram.csv","Wasim Akram")

ca.bowlerAvgWktsGround("./pollock.csv","Shaun Pollock")

ca.bowlerAvgWktsGround("./srinath.csv","J Srinath")

## 26 Average wickets against different opposition

The plot gives the average wickets taken by Muralitharan against different countries. The x-axis also includes the number of innings against each team

import cricpy.analytics as ca
ca.bowlerAvgWktsOpposition("./murali.csv","M Muralitharan")

ca.bowlerAvgWktsOpposition("./akram.csv","Wasim Akram")

ca.bowlerAvgWktsOpposition("./pollock.csv","Shaun Pollock")

ca.bowlerAvgWktsOpposition("./srinath.csv","J Srinath")

## 27 Wickets taken moving average

From the plot below it can be see James Anderson has had a solid performance over the years averaging about wickets

import cricpy.analytics as ca
ca.bowlerMovingAverage("./murali.csv","M Muralitharan")

ca.bowlerMovingAverage("./akram.csv","Wasim Akram")

ca.bowlerMovingAverage("./pollock.csv","Shaun Pollock")

ca.bowlerMovingAverage("./srinath.csv","J Srinath")

## 28 Cumulative average wickets taken

The plots below give the cumulative average wickets taken by the bowlers. Muralitharan has consistently taken wickets at an average of 1.6 wickets per game. Shaun Pollock has an average of 1.5

import cricpy.analytics as ca
ca.bowlerCumulativeAvgWickets("./murali.csv","M Muralitharan")

ca.bowlerCumulativeAvgWickets("./akram.csv","Wasim Akram")

ca.bowlerCumulativeAvgWickets("./pollock.csv","Shaun Pollock")

ca.bowlerCumulativeAvgWickets("./srinath.csv","J Srinath")

## 29 Cumulative average economy rate

The plots below give the cumulative average economy rate of the bowlers. Pollock is the most economical, followed by Akram and then Murali

import cricpy.analytics as ca
ca.bowlerCumulativeAvgEconRate("./murali.csv","M Muralitharan")

ca.bowlerCumulativeAvgEconRate("./akram.csv","Wasim Akram")

ca.bowlerCumulativeAvgEconRate("./pollock.csv","Shaun Pollock")

ca.bowlerCumulativeAvgEconRate("./srinath.csv","J Srinath")

## 30 Relative cumulative average economy rate of bowlers

The Relative cumulative economy rate shows that Pollock is the most economical of the 4 bowlers. He is followed by Akram and then Murali

import cricpy.analytics as ca
frames = ["./srinath.csv","./akram.csv","./murali.csv","pollock.csv"]
names = ["J Srinath","Wasim Akram","M Muralitharan", "S Pollock"]
ca.relativeBowlerCumulativeAvgEconRate(frames,names)

## 31 Relative Economy Rate against wickets taken

Pollock is most economical vs number of wickets taken. Murali has the best figures for 4 wickets taken.

import cricpy.analytics as ca
frames = ["./srinath.csv","./akram.csv","./murali.csv","pollock.csv"]
names = ["J Srinath","Wasim Akram","M Muralitharan", "S Pollock"]
ca.relativeBowlingER(frames,names)

## 32 Relative cumulative average wickets of bowlers in career

The plot below shows that McGrath has the best overall cumulative average wickets. While the bowlers are neck to neck around 130 innings, you can see Muralitharan is most consistent and leads the pack after 150 innings in the number of wickets taken.

import cricpy.analytics as ca
frames = ["./srinath.csv","./akram.csv","./murali.csv","pollock.csv"]
names = ["J Srinath","Wasim Akram","M Muralitharan", "S Pollock"]
ca.relativeBowlerCumulativeAvgWickets(frames,names)

# 33. Key Findings

The plots above capture some of the capabilities and features of my cricpy package. Feel free to install the package and try it out. Please do keep in mind ESPN Cricinfo’s Terms of Use.

Here are the main findings from the analysis above

## Analysis of Top 4 batsman

The analysis of the Top 4 test batsman Tendulkar, Kallis, Ponting and Sangakkara show the folliwing

1. Kohli is a mean run machine and has been consistently piling on runs. Clearly records will lay shattered in days to come for Kohli
2. Virendar Sehwag has the best strike rate of the 4, followed by Jayasuriya and then Kohli
3. Shaun Pollock is the most economical of the bowlers followed by Wasim Akram
4. Muralitharan is the most consistent wicket of the lot.

Important note: Do check out my other posts using cricpy at cricpy-posts

To see all posts click Index of Posts

# Introducing cricpy:A python package to analyze performances of cricketers

Full many a gem of purest ray serene,
The dark unfathomed caves of ocean bear;
Full many a flower is born to blush unseen,
And waste its sweetness on the desert air.

            Thomas Gray, An Elegy Written In A Country Churchyard


# Introduction

It is finally here! cricpy, the python avatar , of my R package cricketr is now ready to rock-n-roll! My R package cricketr had its genesis about 3 and some years ago and went through a couple of enhancements. During this time I have always thought about creating an equivalent python package like cricketr. Now I have finally done it.

So here it is. My python package ‘cricpy!!!’

This package uses the statistics info available in ESPN Cricinfo Statsguru. The current version of this package supports only Test cricket

You should be able to install the package using pip install cricpy and use the many functions available in the package. Please mindful of the ESPN Cricinfo Terms of Use

Do check out my post on R package cricketr at Re-introducing cricketr! : An R package to analyze performances of cricketers

If you are passionate about cricket, and love analyzing cricket performances, then check out my 2 racy books on cricket! In my books, I perform detailed yet compact analysis of performances of both batsmen, bowlers besides evaluating team & match performances in Tests , ODIs, T20s & IPL. You can buy my books on cricket from Amazon at $12.99 for the paperback and$4.99/$6.99 respectively for the kindle versions. The books can be accessed at Cricket analytics with cricketr and Beaten by sheer pace-Cricket analytics with yorkr A must read for any cricket lover! Check it out!! This package uses the statistics info available in ESPN Cricinfo Statsguru. Note: If you would like to do a similar analysis for a different set of batsman and bowlers, you can clone/download my skeleton cricpy-template from Github (which is the R Markdown file I have used for the analysis below). You will only need to make appropriate changes for the players you are interested in. The functions can be executed in RStudio or in a IPython notebook. # The cricpy package The cricpy package has several functions that perform several different analyses on both batsman and bowlers. The package has functions that plot percentage frequency runs or wickets, runs likelihood for a batsman, relative run/strike rates of batsman and relative performance/economy rate for bowlers are available. Other interesting functions include batting performance moving average, forecasting, performance of a player against different oppositions, contribution to wins and losses etc. The data for a particular player can be obtained with the getPlayerData() function. To do this you will need to go to ESPN CricInfo Player and type in the name of the player for e.g Rahul Dravid, Virat Kohli, Alastair Cook etc. This will bring up a page which have the profile number for the player e.g. for Rahul Dravid this would be http://www.espncricinfo.com/india/content/player/28114.html. Hence, Dravid’s profile is 28114. This can be used to get the data for Rahul Dravid as shown below The cricpy package is almost a clone of my R package cricketr. The signature of all the python functions are identical with that of its R avatar namely ‘cricketr’, with only the necessary variations between Python and R. It may be useful to look at my post R vs Python: Different similarities and similar differences. In fact if you are familiar with one of the languages you can look up the package in the other and you will notice the parallel constructs. You can fork/clone the cricpy package at Github cricpy The following 2 examples show the similarity between cricketr and cricpy packages ## 1a.Importing cricketr – R Importing cricketr in R #install.packages("cricketr") library(cricketr) ## 2a. Importing cricpy – Python # Install the package # Do a pip install cricpy # Import cricpy import cricpy # You could either do #1. import cricpy.analytics as ca #ca.batsman4s("../dravid.csv","Rahul Dravid") # Or #2. from cricpy.analytics import * #batsman4s("../dravid.csv","Rahul Dravid")  I would recommend using option 1 namely ca.batsman4s() as I may add an advanced analytics module in the future to cricpy. ## 2 Invoking functions You can seen how the 2 calls are identical for both the R package cricketr and the Python package cricpy ## 2a. Invoking functions with R package ‘cricketr’ library(cricketr) batsman4s("../dravid.csv","Rahul Dravid") ## 2b. Invoking functions with Python package ‘cricpy’ import cricpy.analytics as ca ca.batsman4s("../dravid.csv","Rahul Dravid") # 3a. Getting help from cricketr – R #help("getPlayerData") # 3b. Getting help from cricpy – Python help(ca.getPlayerData) ## Help on function getPlayerData in module cricpy.analytics: ## ## getPlayerData(profile, opposition='', host='', dir='./data', file='player001.csv', type='batting', homeOrAway=[1, 2], result=[1, 2, 4], create=True) ## Get the player data from ESPN Cricinfo based on specific inputs and store in a file in a given directory ## ## Description ## ## Get the player data given the profile of the batsman. The allowed inputs are home,away or both and won,lost or draw of matches. The data is stored in a <player>.csv file in a directory specified. This function also returns a data frame of the player ## ## Usage ## ## getPlayerData(profile,opposition="",host="",dir="./data",file="player001.csv", ## type="batting", homeOrAway=c(1,2),result=c(1,2,4)) ## Arguments ## ## profile ## This is the profile number of the player to get data. This can be obtained from http://www.espncricinfo.com/ci/content/player/index.html. Type the name of the player and click search. This will display the details of the player. Make a note of the profile ID. For e.g For Sachin Tendulkar this turns out to be http://www.espncricinfo.com/india/content/player/35320.html. Hence the profile for Sachin is 35320 ## opposition ## The numerical value of the opposition country e.g.Australia,India, England etc. The values are Australia:2,Bangladesh:25,England:1,India:6,New Zealand:5,Pakistan:7,South Africa:3,Sri Lanka:8, West Indies:4, Zimbabwe:9 ## host ## The numerical value of the host country e.g.Australia,India, England etc. The values are Australia:2,Bangladesh:25,England:1,India:6,New Zealand:5,Pakistan:7,South Africa:3,Sri Lanka:8, West Indies:4, Zimbabwe:9 ## dir ## Name of the directory to store the player data into. If not specified the data is stored in a default directory "./data". Default="./data" ## file ## Name of the file to store the data into for e.g. tendulkar.csv. This can be used for subsequent functions. Default="player001.csv" ## type ## type of data required. This can be "batting" or "bowling" ## homeOrAway ## This is a list with either 1,2 or both. 1 is for home 2 is for away ## result ## This is a list that can take values 1,2,4. 1 - won match 2- lost match 4- draw ## Details ## ## More details can be found in my short video tutorial in Youtube https://www.youtube.com/watch?v=q9uMPFVsXsI ## ## Value ## ## Returns the player's dataframe ## ## Note ## ## Maintainer: Tinniam V Ganesh <tvganesh.85@gmail.com> ## ## Author(s) ## ## Tinniam V Ganesh ## ## References ## ## http://www.espncricinfo.com/ci/content/stats/index.html ## https://gigadom.wordpress.com/ ## ## See Also ## ## getPlayerDataSp ## ## Examples ## ## ## Not run: ## # Both home and away. Result = won,lost and drawn ## tendulkar = getPlayerData(35320,dir=".", file="tendulkar1.csv", ## type="batting", homeOrAway=[1,2],result=[1,2,4]) ## ## # Only away. Get data only for won and lost innings ## tendulkar = getPlayerData(35320,dir=".", file="tendulkar2.csv", ## type="batting",homeOrAway=[2],result=[1,2]) ## ## # Get bowling data and store in file for future ## kumble = getPlayerData(30176,dir=".",file="kumble1.csv", ## type="bowling",homeOrAway=[1],result=[1,2]) ## ## #Get the Tendulkar's Performance against Australia in Australia ## tendulkar = getPlayerData(35320, opposition = 2,host=2,dir=".", ## file="tendulkarVsAusInAus.csv",type="batting") The details below will introduce the different functions that are available in cricpy. ## 3. Get the player data for a player using the function getPlayerData() Important Note This needs to be done only once for a player. This function stores the player’s data in the specified CSV file (for e.g. dravid.csv as above) which can then be reused for all other functions). Once we have the data for the players many analyses can be done. This post will use the stored CSV file obtained with a prior getPlayerData for all subsequent analyses import cricpy.analytics as ca #dravid =ca.getPlayerData(28114,dir="..",file="dravid.csv",type="batting",homeOrAway=[1,2], result=[1,2,4]) #acook =ca.getPlayerData(11728,dir="..",file="acook.csv",type="batting",homeOrAway=[1,2], result=[1,2,4]) import cricpy.analytics as ca #lara =ca.getPlayerData(52337,dir="..",file="lara.csv",type="batting",homeOrAway=[1,2], result=[1,2,4])253802 #kohli =ca.getPlayerData(253802,dir="..",file="kohli.csv",type="batting",homeOrAway=[1,2], result=[1,2,4]) ## 4 Rahul Dravid’s performance – Basic Analyses The 3 plots below provide the following for Rahul Dravid 1. Frequency percentage of runs in each run range over the whole career 2. Mean Strike Rate for runs scored in the given range 3. A histogram of runs frequency percentages in runs ranges import cricpy.analytics as ca import matplotlib.pyplot as plt ca.batsmanRunsFreqPerf("../dravid.csv","Rahul Dravid") ca.batsmanMeanStrikeRate("../dravid.csv","Rahul Dravid") ca.batsmanRunsRanges("../dravid.csv","Rahul Dravid")  ## 5. More analyses import cricpy.analytics as ca ca.batsman4s("../dravid.csv","Rahul Dravid") ca.batsman6s("../dravid.csv","Rahul Dravid")  ca.batsmanDismissals("../dravid.csv","Rahul Dravid") ## 6. 3D scatter plot and prediction plane The plots below show the 3D scatter plot of Dravid Runs versus Balls Faced and Minutes at crease. A linear regression plane is then fitted between Runs and Balls Faced + Minutes at crease import cricpy.analytics as ca ca.battingPerf3d("../dravid.csv","Rahul Dravid") ## 7. Average runs at different venues The plot below gives the average runs scored by Dravid at different grounds. The plot also the number of innings at each ground as a label at x-axis. It can be seen Dravid did great in Rawalpindi, Leeds, Georgetown overseas and , Mohali and Bangalore at home import cricpy.analytics as ca ca.batsmanAvgRunsGround("../dravid.csv","Rahul Dravid") ## 8. Average runs against different opposing teams This plot computes the average runs scored by Dravid against different countries. Dravid has an average of 50+ in England, New Zealand, West Indies and Zimbabwe. import cricpy.analytics as ca ca.batsmanAvgRunsOpposition("../dravid.csv","Rahul Dravid") ## 9 . Highest Runs Likelihood The plot below shows the Runs Likelihood for a batsman. For this the performance of Sachin is plotted as a 3D scatter plot with Runs versus Balls Faced + Minutes at crease. K-Means. The centroids of 3 clusters are computed and plotted. In this plot Dravid’s highest tendencies are computed and plotted using K-Means import cricpy.analytics as ca ca.batsmanRunsLikelihood("../dravid.csv","Rahul Dravid") ## 10. A look at the Top 4 batsman – Rahul Dravid, Alastair Cook, Brian Lara and Virat Kohli The following batsmen have been very prolific in test cricket and will be used for teh analyses 1. Rahul Dravid :Average:52.31,100’s – 36, 50’s – 63 2. Alastair Cook : Average: 45.35, 100’s – 33, 50’s – 57 3. Brian Lara : Average: 52.88, 100’s – 34 , 50’s – 48 4. Virat Kohli: Average: 54.57 ,100’s – 24 , 50’s – 19 The following plots take a closer at their performances. The box plots show the median the 1st and 3rd quartile of the runs ## 11. Box Histogram Plot This plot shows a combined boxplot of the Runs ranges and a histogram of the Runs Frequency import cricpy.analytics as ca ca.batsmanPerfBoxHist("../dravid.csv","Rahul Dravid") ca.batsmanPerfBoxHist("../acook.csv","Alastair Cook") ca.batsmanPerfBoxHist("../lara.csv","Brian Lara") ca.batsmanPerfBoxHist("../kohli.csv","Virat Kohli") ## 12. Contribution to won and lost matches The plot below shows the contribution of Dravid, Cook, Lara and Kohli in matches won and lost. It can be seen that in matches where India has won Dravid and Kohli have scored more and must have been instrumental in the win For the 2 functions below you will have to use the getPlayerDataSp() function as shown below. I have commented this as I already have these files import cricpy.analytics as ca #dravidsp = ca.getPlayerDataSp(28114,tdir=".",tfile="dravidsp.csv",ttype="batting") #acooksp = ca.getPlayerDataSp(11728,tdir=".",tfile="acooksp.csv",ttype="batting") #larasp = ca.getPlayerDataSp(52337,tdir=".",tfile="larasp.csv",ttype="batting") #kohlisp = ca.getPlayerDataSp(253802,tdir=".",tfile="kohlisp.csv",ttype="batting") import cricpy.analytics as ca ca.batsmanContributionWonLost("../dravidsp.csv","Rahul Dravid") ca.batsmanContributionWonLost("../acooksp.csv","Alastair Cook") ca.batsmanContributionWonLost("../larasp.csv","Brian Lara") ca.batsmanContributionWonLost("../kohlisp.csv","Virat Kohli") ## 13. Performance at home and overseas From the plot below it can be seen Dravid has a higher median overseas than at home.Cook, Lara and Kohli have a lower median of runs overseas than at home. This function also requires the use of getPlayerDataSp() as shown above import cricpy.analytics as ca ca.batsmanPerfHomeAway("../dravidsp.csv","Rahul Dravid") ca.batsmanPerfHomeAway("../acooksp.csv","Alastair Cook") ca.batsmanPerfHomeAway("../larasp.csv","Brian Lara") ca.batsmanPerfHomeAway("../kohlisp.csv","Virat Kohli") ## 14 Moving Average of runs in career Take a look at the Moving Average across the career of the Top 4 (ignore the dip at the end of all plots. Need to check why this is so!). Lara’s performance seems to have been quite good before his retirement(wonder why retired so early!). Kohli’s performance has been steadily improving over the years import cricpy.analytics as ca ca.batsmanMovingAverage("../dravid.csv","Rahul Dravid") ca.batsmanMovingAverage("../acook.csv","Alastair Cook") ca.batsmanMovingAverage("../lara.csv","Brian Lara") ca.batsmanMovingAverage("../kohli.csv","Virat Kohli") ## 15 Cumulative Average runs of batsman in career This function provides the cumulative average runs of the batsman over the career. Dravid averages around 48, Cook around 44, Lara around 50 and Kohli shows a steady improvement in his cumulative average. Kohli seems to be getting better with time. import cricpy.analytics as ca ca.batsmanCumulativeAverageRuns("../dravid.csv","Rahul Dravid") ca.batsmanCumulativeAverageRuns("../acook.csv","Alastair Cook") ca.batsmanCumulativeAverageRuns("../lara.csv","Brian Lara") ca.batsmanCumulativeAverageRuns("../kohli.csv","Virat Kohli") ## 16 Cumulative Average strike rate of batsman in career Lara has a terrific strike rate of 52+. Cook has a better strike rate over Dravid. Kohli’s strike rate has improved over the years. import cricpy.analytics as ca ca.batsmanCumulativeStrikeRate("../dravid.csv","Rahul Dravid") ca.batsmanCumulativeStrikeRate("../acook.csv","Alastair Cook") ca.batsmanCumulativeStrikeRate("../lara.csv","Brian Lara") ca.batsmanCumulativeStrikeRate("../kohli.csv","Virat Kohli") ## 17 Future Runs forecast Here are plots that forecast how the batsman will perform in future. Currently ARIMA has been used for the forecast. (To do: Perform Holt-Winters forecast!) import cricpy.analytics as ca ca.batsmanPerfForecast("../dravid.csv","Rahul Dravid") ## ARIMA Model Results ## ============================================================================== ## Dep. Variable: D.runs No. Observations: 284 ## Model: ARIMA(5, 1, 0) Log Likelihood -1522.837 ## Method: css-mle S.D. of innovations 51.488 ## Date: Sun, 28 Oct 2018 AIC 3059.673 ## Time: 09:47:39 BIC 3085.216 ## Sample: 07-04-1996 HQIC 3069.914 ## - 01-24-2012 ## ================================================================================ ## coef std err z P>|z| [0.025 0.975] ## -------------------------------------------------------------------------------- ## const -0.1336 0.884 -0.151 0.880 -1.867 1.599 ## ar.L1.D.runs -0.7729 0.058 -13.322 0.000 -0.887 -0.659 ## ar.L2.D.runs -0.6234 0.071 -8.753 0.000 -0.763 -0.484 ## ar.L3.D.runs -0.5199 0.074 -7.038 0.000 -0.665 -0.375 ## ar.L4.D.runs -0.3490 0.071 -4.927 0.000 -0.488 -0.210 ## ar.L5.D.runs -0.2116 0.058 -3.665 0.000 -0.325 -0.098 ## Roots ## ============================================================================= ## Real Imaginary Modulus Frequency ## ----------------------------------------------------------------------------- ## AR.1 0.5789 -1.1743j 1.3093 -0.1771 ## AR.2 0.5789 +1.1743j 1.3093 0.1771 ## AR.3 -1.3617 -0.0000j 1.3617 -0.5000 ## AR.4 -0.7227 -1.2257j 1.4230 -0.3348 ## AR.5 -0.7227 +1.2257j 1.4230 0.3348 ## ----------------------------------------------------------------------------- ## 0 ## count 284.000000 ## mean -0.306769 ## std 51.632947 ## min -106.653589 ## 25% -33.835148 ## 50% -8.954253 ## 75% 21.024763 ## max 223.152901 ## ## C:\Users\Ganesh\ANACON~1\lib\site-packages\statsmodels\tsa\kalmanf\kalmanfilter.py:646: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. ## if issubdtype(paramsdtype, float): ## C:\Users\Ganesh\ANACON~1\lib\site-packages\statsmodels\tsa\kalmanf\kalmanfilter.py:650: FutureWarning: Conversion of the second argument of issubdtype from complex to np.complexfloating is deprecated. In future, it will be treated as np.complex128 == np.dtype(complex).type. ## elif issubdtype(paramsdtype, complex): ## C:\Users\Ganesh\ANACON~1\lib\site-packages\statsmodels\tsa\kalmanf\kalmanfilter.py:577: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. ## if issubdtype(paramsdtype, float): ## 18 Relative Batsman Cumulative Average Runs The plot below compares the Relative cumulative average runs of the batsman for each of the runs ranges of 10 and plots them. The plot indicate the following Range 30 – 100 innings – Lara leads followed by Dravid Range 100+ innings – Kohli races ahead of the rest import cricpy.analytics as ca frames = ["../dravid.csv","../acook.csv","../lara.csv","../kohli.csv"] names = ["Dravid","A Cook","Brian Lara","V Kohli"] ca.relativeBatsmanCumulativeAvgRuns(frames,names) ## 19. Relative Batsman Strike Rate The plot below gives the relative Runs Frequency Percetages for each 10 run bucket. The plot below show Brian Lara towers over the Dravid, Cook and Kohli. However you will notice that Kohli’s strike rate is going up import cricpy.analytics as ca frames = ["../dravid.csv","../acook.csv","../lara.csv","../kohli.csv"] names = ["Dravid","A Cook","Brian Lara","V Kohli"] ca.relativeBatsmanCumulativeStrikeRate(frames,names) ## 20. 3D plot of Runs vs Balls Faced and Minutes at Crease The plot is a scatter plot of Runs vs Balls faced and Minutes at Crease. A prediction plane is fitted import cricpy.analytics as ca ca.battingPerf3d("../dravid.csv","Rahul Dravid") ca.battingPerf3d("../acook.csv","Alastair Cook") ca.battingPerf3d("../lara.csv","Brian Lara") ca.battingPerf3d("../kohli.csv","Virat Kohli") ## 21. Predicting Runs given Balls Faced and Minutes at Crease A multi-variate regression plane is fitted between Runs and Balls faced +Minutes at crease. import cricpy.analytics as ca import numpy as np import pandas as pd BF = np.linspace( 10, 400,15) Mins = np.linspace( 30,600,15) newDF= pd.DataFrame({'BF':BF,'Mins':Mins}) dravid = ca.batsmanRunsPredict("../dravid.csv",newDF,"Dravid") print(dravid) ## BF Mins Runs ## 0 10.000000 30.000000 0.519667 ## 1 37.857143 70.714286 13.821794 ## 2 65.714286 111.428571 27.123920 ## 3 93.571429 152.142857 40.426046 ## 4 121.428571 192.857143 53.728173 ## 5 149.285714 233.571429 67.030299 ## 6 177.142857 274.285714 80.332425 ## 7 205.000000 315.000000 93.634552 ## 8 232.857143 355.714286 106.936678 ## 9 260.714286 396.428571 120.238805 ## 10 288.571429 437.142857 133.540931 ## 11 316.428571 477.857143 146.843057 ## 12 344.285714 518.571429 160.145184 ## 13 372.142857 559.285714 173.447310 ## 14 400.000000 600.000000 186.749436 The fitted model is then used to predict the runs that the batsmen will score for a given Balls faced and Minutes at crease. ## 22 Analysis of Top 3 wicket takers The following 3 bowlers have had an excellent career and will be used for the analysis 1. Glenn McGrath:Wickets: 563, Average = 21.64, Economy Rate – 2.49 2. Kapil Dev : Wickets: 434, Average = 29.64, Economy Rate – 2.78 3. James Anderson: Wickets: 564, Average = 28.64, Economy Rate – 2.88 How do Glenn McGrath, Kapil Dev and James Anderson compare with one another with respect to wickets taken and the Economy Rate. The next set of plots compute and plot precisely these analyses. ## 23. Get the bowler’s data This plot below computes the percentage frequency of number of wickets taken for e.g 1 wicket x%, 2 wickets y% etc and plots them as a continuous line import cricpy.analytics as ca #mcgrath =ca.getPlayerData(6565,dir=".",file="mcgrath.csv",type="bowling",homeOrAway=[1,2], result=[1,2,4]) #kapil =ca.getPlayerData(30028,dir=".",file="kapil.csv",type="bowling",homeOrAway=[1,2], result=[1,2,4]) #anderson =ca.getPlayerData(8608,dir=".",file="anderson.csv",type="bowling",homeOrAway=[1,2], result=[1,2,4]) ## 24. Wicket Frequency Plot This plot below plots the frequency of wickets taken for each of the bowlers import cricpy.analytics as ca ca.bowlerWktsFreqPercent("../mcgrath.csv","Glenn McGrath") ca.bowlerWktsFreqPercent("../kapil.csv","Kapil Dev") ca.bowlerWktsFreqPercent("../anderson.csv","James Anderson") ## 25. Wickets Runs plot The plot below create a box plot showing the 1st and 3rd quartile of runs conceded versus the number of wickets taken import cricpy.analytics as ca ca.bowlerWktsRunsPlot("../mcgrath.csv","Glenn McGrath") ca.bowlerWktsRunsPlot("../kapil.csv","Kapil Dev") ca.bowlerWktsRunsPlot("../anderson.csv","James Anderson") ## 26 Average wickets at different venues The plot gives the average wickets taken by Muralitharan at different venues. McGrath best performances are at Centurion, Lord’s and Port of Spain averaging about 4 wickets. Kapil Dev’s does good at Kingston and Wellington. Anderson averages 4 wickets at Dunedin and Nagpur import cricpy.analytics as ca ca.bowlerAvgWktsGround("../mcgrath.csv","Glenn McGrath") ca.bowlerAvgWktsGround("../kapil.csv","Kapil Dev") ca.bowlerAvgWktsGround("../anderson.csv","James Anderson") ## 27 Average wickets against different opposition The plot gives the average wickets taken by Muralitharan against different countries. The x-axis also includes the number of innings against each team import cricpy.analytics as ca ca.bowlerAvgWktsOpposition("../mcgrath.csv","Glenn McGrath") ca.bowlerAvgWktsOpposition("../kapil.csv","Kapil Dev") ca.bowlerAvgWktsOpposition("../anderson.csv","James Anderson") ## 28 Wickets taken moving average From the plot below it can be see James Anderson has had a solid performance over the years averaging about wickets import cricpy.analytics as ca ca.bowlerMovingAverage("../mcgrath.csv","Glenn McGrath") ca.bowlerMovingAverage("../kapil.csv","Kapil Dev") ca.bowlerMovingAverage("../anderson.csv","James Anderson") ## 29 Cumulative average wickets taken The plots below give the cumulative average wickets taken by the bowlers. mcGrath plateaus around 2.4 wickets, Kapil Dev’s performance deteriorates over the years. Anderson holds on rock steady around 2 wickets import cricpy.analytics as ca ca.bowlerCumulativeAvgWickets("../mcgrath.csv","Glenn McGrath") ca.bowlerCumulativeAvgWickets("../kapil.csv","Kapil Dev") ca.bowlerCumulativeAvgWickets("../anderson.csv","James Anderson") ## 30 Cumulative average economy rate The plots below give the cumulative average economy rate of the bowlers. McGrath’s was very expensive early in his career conceding about 2.8 runs per over which drops to around 2.5 runs towards the end. Kapil Dev’s economy rate drops from 3.6 to 2.8. Anderson is probably more expensive than the other 2. import cricpy.analytics as ca ca.bowlerCumulativeAvgEconRate("../mcgrath.csv","Glenn McGrath") ca.bowlerCumulativeAvgEconRate("../kapil.csv","Kapil Dev") ca.bowlerCumulativeAvgEconRate("../anderson.csv","James Anderson") ## 31 Future Wickets forecast import cricpy.analytics as ca ca.bowlerPerfForecast("../mcgrath.csv","Glenn McGrath") ## ARIMA Model Results ## ============================================================================== ## Dep. Variable: D.Wickets No. Observations: 236 ## Model: ARIMA(5, 1, 0) Log Likelihood -480.815 ## Method: css-mle S.D. of innovations 1.851 ## Date: Sun, 28 Oct 2018 AIC 975.630 ## Time: 09:28:32 BIC 999.877 ## Sample: 11-12-1993 HQIC 985.404 ## - 01-02-2007 ## =================================================================================== ## coef std err z P>|z| [0.025 0.975] ## ----------------------------------------------------------------------------------- ## const 0.0037 0.033 0.113 0.910 -0.061 0.068 ## ar.L1.D.Wickets -0.9432 0.064 -14.708 0.000 -1.069 -0.818 ## ar.L2.D.Wickets -0.7254 0.086 -8.469 0.000 -0.893 -0.558 ## ar.L3.D.Wickets -0.4827 0.093 -5.217 0.000 -0.664 -0.301 ## ar.L4.D.Wickets -0.3690 0.085 -4.324 0.000 -0.536 -0.202 ## ar.L5.D.Wickets -0.1709 0.064 -2.678 0.008 -0.296 -0.046 ## Roots ## ============================================================================= ## Real Imaginary Modulus Frequency ## ----------------------------------------------------------------------------- ## AR.1 0.5630 -1.2761j 1.3948 -0.1839 ## AR.2 0.5630 +1.2761j 1.3948 0.1839 ## AR.3 -0.8433 -1.0820j 1.3718 -0.3554 ## AR.4 -0.8433 +1.0820j 1.3718 0.3554 ## AR.5 -1.5981 -0.0000j 1.5981 -0.5000 ## ----------------------------------------------------------------------------- ## 0 ## count 236.000000 ## mean -0.005142 ## std 1.856961 ## min -3.457002 ## 25% -1.433391 ## 50% -0.080237 ## 75% 1.446149 ## max 5.840050 ## 32 Get player data special As discussed above the next 2 charts require the use of getPlayerDataSp() import cricpy.analytics as ca #mcgrathsp =ca.getPlayerDataSp(6565,tdir=".",tfile="mcgrathsp.csv",ttype="bowling") #kapilsp =ca.getPlayerDataSp(30028,tdir=".",tfile="kapilsp.csv",ttype="bowling") #andersonsp =ca.getPlayerDataSp(8608,tdir=".",tfile="andersonsp.csv",ttype="bowling") ## 33 Contribution to matches won and lost The plot below is extremely interesting Glenn McGrath has been more instrumental in Australia winning than Kapil and Anderson as seems to have taken more wickets when Australia won. import cricpy.analytics as ca ca.bowlerContributionWonLost("../mcgrathsp.csv","Glenn McGrath") ca.bowlerContributionWonLost("../kapilsp.csv","Kapil Dev") ca.bowlerContributionWonLost("../andersonsp.csv","James Anderson") ## 34 Performance home and overseas McGrath and Kapil Dev have performed better overseas than at home. Anderson has performed about the same home and overseas import cricpy.analytics as ca ca.bowlerPerfHomeAway("../mcgrathsp.csv","Glenn McGrath") ca.bowlerPerfHomeAway("../kapilsp.csv","Kapil Dev") ca.bowlerPerfHomeAway("../andersonsp.csv","James Anderson") ## 35 Relative cumulative average economy rate of bowlers The Relative cumulative economy rate shows that McGrath has the best economy rate followed by Kapil Dev and then Anderson. import cricpy.analytics as ca frames = ["../mcgrath.csv","../kapil.csv","../anderson.csv"] names = ["Glenn McGrath","Kapil Dev","James Anderson"] ca.relativeBowlerCumulativeAvgEconRate(frames,names) ## 36 Relative Economy Rate against wickets taken McGrath has been economical regardless of the number of wickets taken. Kapil Dev has been slightly more expensive when he takes more wickets import cricpy.analytics as ca frames = ["../mcgrath.csv","../kapil.csv","../anderson.csv"] names = ["Glenn McGrath","Kapil Dev","James Anderson"] ca.relativeBowlingER(frames,names) ## 37 Relative cumulative average wickets of bowlers in career The plot below shows that McGrath has the best overall cumulative average wickets. Kapil’s leads Anderson till about 150 innings after which Anderson takes over import cricpy.analytics as ca frames = ["../mcgrath.csv","../kapil.csv","../anderson.csv"] names = ["Glenn McGrath","Kapil Dev","James Anderson"] ca.relativeBowlerCumulativeAvgWickets(frames,names) # Key Findings The plots above capture some of the capabilities and features of my cricpy package. Feel free to install the package and try it out. Please do keep in mind ESPN Cricinfo’s Terms of Use. Here are the main findings from the analysis above ## Key insights 1. Brian Lara is head and shoulders above the rest in the overall strike rate 2. Kohli performance has been steadily improving over the years and with the way he is going he will shatter all records. 3. Kohli and Dravid have scored more in matches where India has won than the other two. 4. Dravid has performed very well overseas 5. The cumulative average runs has Kohli just edging out the other 3. Kohli is probably midway in his career but considering that his moving average is improving strongly, we can expect great things of him with the way he is going. 6. McGrath has had some great performances overseas 7. Mcgrath has the best economy rate and has contributed significantly to Australia’s wins. 8.In the cumulative average wickets race McGrath leads the pack. Kapil leads Anderson till about 150 matches after which Anderson takes over. The code for cricpy can be accessed at Github at cricpy Do let me know if you run into issues. ## Conclusion I have long wanted to make a python equivalent of cricketr and I have been able to make it. cricpy is still work in progress. I have add the necessary functions for ODI and Twenty20. Go ahead give ‘cricpy’ a spin!! Stay tuned! Important note: Do check out my other posts using cricpy at cricpy-posts # Practical Machine Learning with R and Python – Part 5 This is the 5th and probably penultimate part of my series on ‘Practical Machine Learning with R and Python’. The earlier parts of this series included 1. Practical Machine Learning with R and Python – Part 1 In this initial post, I touch upon univariate, multivariate, polynomial regression and KNN regression in R and Python 2.Practical Machine Learning with R and Python – Part 2 In this post, I discuss Logistic Regression, KNN classification and cross validation error for both LOOCV and K-Fold in both R and Python 3.Practical Machine Learning with R and Python – Part 3 This post covered ‘feature selection’ in Machine Learning. Specifically I touch best fit, forward fit, backward fit, ridge(L2 regularization) & lasso (L1 regularization). The post includes equivalent code in R and Python. 4.Practical Machine Learning with R and Python – Part 4 In this part I discussed SVMs, Decision Trees, validation, precision recall, and roc curves This post ‘Practical Machine Learning with R and Python – Part 5’ discusses regression with B-splines, natural splines, smoothing splines, generalized additive models (GAMS), bagging, random forest and boosting As with my previous posts in this series, this post is largely based on the following 2 MOOC courses 1. Statistical Learning, Prof Trevor Hastie & Prof Robert Tibesherani, Online Stanford 2. Applied Machine Learning in Python Prof Kevyn-Collin Thomson, University Of Michigan, Coursera You can download this R Markdown file and associated data files from Github at MachineLearning-RandPython-Part5 Note: Please listen to my video presentations Machine Learning in youtube 1. Machine Learning in plain English-Part 1 2. Machine Learning in plain English-Part 2 3. Machine Learning in plain English-Part 3 Check out my compact and minimal book “Practical Machine Learning with R and Python:Third edition- Machine Learning in stereo” available in Amazon in paperback($12.99) and kindle($8.99) versions. My book includes implementations of key ML algorithms and associated measures and metrics. The book is ideal for anybody who is familiar with the concepts and would like a quick reference to the different ML algorithms that can be applied to problems and how to select the best model. Pick your copy today!! For this part I have used the data sets from UCI Machine Learning repository(Communities and Crime and Auto MPG) ## 1. Splines When performing regression (continuous or logistic) between a target variable and a feature (or a set of features), a single polynomial for the entire range of the data set usually does not perform a good fit.Rather we would need to provide we could fit regression curves for different section of the data set. There are several techniques which do this for e.g. piecewise-constant functions, piecewise-linear functions, piecewise-quadratic/cubic/4th order polynomial functions etc. One such set of functions are the cubic splines which fit cubic polynomials to successive sections of the dataset. The points where the cubic splines join, are called ‘knots’. Since each section has a different cubic spline, there could be discontinuities (or breaks) at these knots. To prevent these discontinuities ‘natural splines’ and ‘smoothing splines’ ensure that the seperate cubic functions have 2nd order continuity at these knots with the adjacent splines. 2nd order continuity implies that the value, 1st order derivative and 2nd order derivative at these knots are equal. A cubic spline with knots $\alpha_{k}$ , k=1,2,3,..K is a piece-wise cubic polynomial with continuous derivative up to order 2 at each knot. We can write $y_{i} = \beta_{0} +\beta_{1}b_{1}(x_{i}) +\beta_{2}b_{2}(x_{i}) + .. + \beta_{K+3}b_{K+3}(x_{i}) + \epsilon_{i}$. For each ($x{i},y{i}$), $b_{i}$ are called ‘basis’ functions, where $b_{1}(x_{i})=x_{i}$$b_{2}(x_{i})=x_{i}^2$, $b_{3}(x_{i})=x_{i}^3$, $b_{k+3}(x_{i})=(x_{i} -\alpha_{k})^3$ where k=1,2,3… K The 1st and 2nd derivatives of cubic splines are continuous at the knots. Hence splines provide a smooth continuous fit to the data by fitting different splines to different sections of the data ## 1.1a Fit a 4th degree polynomial – R code In the code below a non-linear function (a 4th order polynomial) is used to fit the data. Usually when we fit a single polynomial to the entire data set the tails of the fit tend to vary a lot particularly if there are fewer points at the ends. Splines help in reducing this variation at the extremities library(dplyr) library(ggplot2) source('RFunctions-1.R') # Read the data df=read.csv("auto_mpg.csv",stringsAsFactors = FALSE) # Data from UCI df1 <- as.data.frame(sapply(df,as.numeric)) #Select specific columns df2 <- df1 %>% dplyr::select(cylinder,displacement, horsepower,weight, acceleration, year,mpg) auto <- df2[complete.cases(df2),] # Fit a 4th degree polynomial fit=lm(mpg~poly(horsepower,4),data=auto) #Display a summary of fit summary(fit) ## ## Call: ## lm(formula = mpg ~ poly(horsepower, 4), data = auto) ## ## Residuals: ## Min 1Q Median 3Q Max ## -14.8820 -2.5802 -0.1682 2.2100 16.1434 ## ## Coefficients: ## Estimate Std. Error t value Pr(>|t|) ## (Intercept) 23.4459 0.2209 106.161 <2e-16 *** ## poly(horsepower, 4)1 -120.1377 4.3727 -27.475 <2e-16 *** ## poly(horsepower, 4)2 44.0895 4.3727 10.083 <2e-16 *** ## poly(horsepower, 4)3 -3.9488 4.3727 -0.903 0.367 ## poly(horsepower, 4)4 -5.1878 4.3727 -1.186 0.236 ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 ## ## Residual standard error: 4.373 on 387 degrees of freedom ## Multiple R-squared: 0.6893, Adjusted R-squared: 0.6861 ## F-statistic: 214.7 on 4 and 387 DF, p-value: < 2.2e-16 #Get the range of horsepower hp <- range(auto$horsepower)
#Create a sequence to be used for plotting
hpGrid <- seq(hp[1],hp[2],by=10)
#Predict for these values of horsepower. Set Standard error as TRUE
pred=predict(fit,newdata=list(horsepower=hpGrid),se=TRUE)
#Compute bands on either side that is 2xSE
seBands=cbind(pred$fit+2*pred$se.fit,pred$fit-2*pred$se.fit)
#Plot the fit with Standard Error bands
plot(auto$horsepower,auto$mpg,xlim=hp,cex=.5,col="black",xlab="Horsepower",
ylab="MPG", main="Polynomial of degree 4")
lines(hpGrid,pred$fit,lwd=2,col="blue") matlines(hpGrid,seBands,lwd=2,col="blue",lty=3) ## 1.1b Fit a 4th degree polynomial – Python code import numpy as np import pandas as pd import os import matplotlib.pyplot as plt from sklearn.model_selection import train_test_split from sklearn.preprocessing import PolynomialFeatures from sklearn.linear_model import LinearRegression #Read the auto data autoDF =pd.read_csv("auto_mpg.csv",encoding="ISO-8859-1") # Select columns autoDF1=autoDF[['mpg','cylinder','displacement','horsepower','weight','acceleration','year']] # Convert all columns to numeric autoDF2 = autoDF1.apply(pd.to_numeric, errors='coerce') #Drop NAs autoDF3=autoDF2.dropna() autoDF3.shape X=autoDF3[['horsepower']] y=autoDF3['mpg'] #Create a polynomial of degree 4 poly = PolynomialFeatures(degree=4) X_poly = poly.fit_transform(X) # Fit a polynomial regression line linreg = LinearRegression().fit(X_poly, y) # Create a range of values hpGrid = np.arange(np.min(X),np.max(X),10) hp=hpGrid.reshape(-1,1) # Transform to 4th degree poly = PolynomialFeatures(degree=4) hp_poly = poly.fit_transform(hp) #Create a scatter plot plt.scatter(X,y) # Fit the prediction ypred=linreg.predict(hp_poly) plt.title("Poylnomial of degree 4") fig2=plt.xlabel("Horsepower") fig2=plt.ylabel("MPG") # Draw the regression curve plt.plot(hp,ypred,c="red") plt.savefig('fig1.png', bbox_inches='tight') ## 1.1c Fit a B-Spline – R Code In the code below a B- Spline is fit to data. The B-spline requires the manual selection of knots #Splines library(splines) # Fit a B-spline to the data. Select knots at 60,75,100,150 fit=lm(mpg~bs(horsepower,df=6,knots=c(60,75,100,150)),data=auto) # Use the fitted regresion to predict pred=predict(fit,newdata=list(horsepower=hpGrid),se=T) # Create a scatter plot plot(auto$horsepower,auto$mpg,xlim=hp,cex=.5,col="black",xlab="Horsepower", ylab="MPG", main="B-Spline with 4 knots") #Draw lines with 2 Standard Errors on either side lines(hpGrid,pred$fit,lwd=2)
lines(hpGrid,pred$fit+2*pred$se,lty="dashed")
lines(hpGrid,pred$fit-2*pred$se,lty="dashed")
abline(v=c(60,75,100,150),lty=2,col="darkgreen")

## 1.1d Fit a Natural Spline – R Code

Here a ‘Natural Spline’ is used to fit .The Natural Spline extrapolates beyond the boundary knots and the ends of the function are much more constrained than a regular spline or a global polynomoial where the ends can wag a lot more. Natural splines do not require the explicit selection of knots

# There is no need to select the knots here. There is a smoothing parameter which
# can be specified by the degrees of freedom 'df' parameter. The natural spline

fit2=lm(mpg~ns(horsepower,df=4),data=auto)
pred=predict(fit2,newdata=list(horsepower=hpGrid),se=T)
plot(auto$horsepower,auto$mpg,xlim=hp,cex=.5,col="black",xlab="Horsepower",
ylab="MPG", main="Natural Splines")
lines(hpGrid,pred$fit,lwd=2) lines(hpGrid,pred$fit+2*pred$se,lty="dashed") lines(hpGrid,pred$fit-2*pred$se,lty="dashed") ## 1.1.e Fit a Smoothing Spline – R code Here a smoothing spline is used. Smoothing splines also do not require the explicit setting of knots. We can change the ‘degrees of freedom(df)’ paramater to get the best fit # Smoothing spline has a smoothing parameter, the degrees of freedom # This is too wiggly plot(auto$horsepower,auto$mpg,xlim=hp,cex=.5,col="black",xlab="Horsepower", ylab="MPG", main="Smoothing Splines") # Here df is set to 16. This has a lot of variance fit=smooth.spline(auto$horsepower,auto$mpg,df=16) lines(fit,col="red",lwd=2) # We can use Cross Validation to allow the spline to pick the value of this smpopothing paramter. We do not need to set the degrees of freedom 'df' fit=smooth.spline(auto$horsepower,auto$mpg,cv=TRUE) lines(fit,col="blue",lwd=2) ## 1.1e Splines – Python There isn’t as much treatment of splines in Python and SKLearn. I did find the LSQUnivariate, UnivariateSpline spline. The LSQUnivariate spline requires the explcit setting of knots import numpy as np import pandas as pd import os import matplotlib.pyplot as plt from scipy.interpolate import LSQUnivariateSpline autoDF =pd.read_csv("auto_mpg.csv",encoding="ISO-8859-1") autoDF.shape autoDF.columns autoDF1=autoDF[['mpg','cylinder','displacement','horsepower','weight','acceleration','year']] autoDF2 = autoDF1.apply(pd.to_numeric, errors='coerce') auto=autoDF2.dropna() auto=auto[['horsepower','mpg']].sort_values('horsepower') # Set the knots manually knots=[65,75,100,150] # Create an array for X & y X=np.array(auto['horsepower']) y=np.array(auto['mpg']) # Fit a LSQunivariate spline s = LSQUnivariateSpline(X,y,knots) #Plot the spline xs = np.linspace(40,230,1000) ys = s(xs) plt.scatter(X, y) plt.plot(xs, ys) plt.savefig('fig2.png', bbox_inches='tight')  ## 1.2 Generalized Additiive models (GAMs) Generalized Additive Models (GAMs) is a really powerful ML tool. $y_{i} = \beta_{0} + f_{1}(x_{i1}) + f_{2}(x_{i2}) + .. +f_{p}(x_{ip}) + \epsilon_{i}$ In GAMs we use a different functions for each of the variables. GAMs give a much better fit since we can choose any function for the different sections ## 1.2a Generalized Additive Models (GAMs) – R Code The plot below show the smooth spline that is fit for each of the features horsepower, cylinder, displacement, year and acceleration. We can use any function for example loess, 4rd order polynomial etc. library(gam) # Fit a smoothing spline for horsepower, cyliner, displacement and acceleration gam=gam(mpg~s(horsepower,4)+s(cylinder,5)+s(displacement,4)+s(year,4)+s(acceleration,5),data=auto) # Display the summary of the fit. This give the significance of each of the paramwetr # Also an ANOVA is given for each combination of the features summary(gam) ## ## Call: gam(formula = mpg ~ s(horsepower, 4) + s(cylinder, 5) + s(displacement, ## 4) + s(year, 4) + s(acceleration, 5), data = auto) ## Deviance Residuals: ## Min 1Q Median 3Q Max ## -8.3190 -1.4436 -0.0261 1.2279 12.0873 ## ## (Dispersion Parameter for gaussian family taken to be 6.9943) ## ## Null Deviance: 23818.99 on 391 degrees of freedom ## Residual Deviance: 2587.881 on 370 degrees of freedom ## AIC: 1898.282 ## ## Number of Local Scoring Iterations: 3 ## ## Anova for Parametric Effects ## Df Sum Sq Mean Sq F value Pr(>F) ## s(horsepower, 4) 1 15632.8 15632.8 2235.085 < 2.2e-16 *** ## s(cylinder, 5) 1 508.2 508.2 72.666 3.958e-16 *** ## s(displacement, 4) 1 374.3 374.3 53.514 1.606e-12 *** ## s(year, 4) 1 2263.2 2263.2 323.583 < 2.2e-16 *** ## s(acceleration, 5) 1 372.4 372.4 53.246 1.809e-12 *** ## Residuals 370 2587.9 7.0 ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 ## ## Anova for Nonparametric Effects ## Npar Df Npar F Pr(F) ## (Intercept) ## s(horsepower, 4) 3 13.825 1.453e-08 *** ## s(cylinder, 5) 3 17.668 9.712e-11 *** ## s(displacement, 4) 3 44.573 < 2.2e-16 *** ## s(year, 4) 3 23.364 7.183e-14 *** ## s(acceleration, 5) 4 3.848 0.004453 ** ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 par(mfrow=c(2,3)) plot(gam,se=TRUE) ## 1.2b Generalized Additive Models (GAMs) – Python Code I did not find the equivalent of GAMs in SKlearn in Python. There was an early prototype (2012) in Github. Looks like it is still work in progress or has probably been abandoned. ## 1.3 Tree based Machine Learning Models Tree based Machine Learning are all based on the ‘bootstrapping’ technique. In bootstrapping given a sample of size N, we create datasets of size N by sampling this original dataset with replacement. Machine Learning models are built on the different bootstrapped samples and then averaged. Decision Trees as seen above have the tendency to overfit. There are several techniques that help to avoid this namely a) Bagging b) Random Forests c) Boosting ### Bagging, Random Forest and Gradient Boosting Bagging: Bagging, or Bootstrap Aggregation decreases the variance of predictions, by creating separate Decisiion Tree based ML models on the different samples and then averaging these ML models Random Forests: Bagging is a greedy algorithm and tries to produce splits based on all variables which try to minimize the error. However the different ML models have a high correlation. Random Forests remove this shortcoming, by using a variable and random set of features to split on. Hence the features chosen and the resulting trees are uncorrelated. When these ML models are averaged the performance is much better. Boosting: Gradient Boosted Decision Trees also use an ensemble of trees but they don’t build Machine Learning models with random set of features at each step. Rather small and simple trees are built. Successive trees try to minimize the error from the earlier trees. Out of Bag (OOB) Error: In Random Forest and Gradient Boosting for each bootstrap sample taken from the dataset, there will be samples left out. These are known as Out of Bag samples.Classification accuracy carried out on these OOB samples is known as OOB error ## 1.31a Decision Trees – R Code The code below creates a Decision tree with the cancer training data. The summary of the fit is output. Based on the ML model, the predict function is used on test data and a confusion matrix is output. # Read the cancer data library(tree) library(caret) library(e1071) cancer <- read.csv("cancer.csv",stringsAsFactors = FALSE) cancer <- cancer[,2:32] cancer$target <- as.factor(cancer$target) train_idx <- trainTestSplit(cancer,trainPercent=75,seed=5) train <- cancer[train_idx, ] test <- cancer[-train_idx, ] # Create Decision Tree cancerStatus=tree(target~.,train) summary(cancerStatus) ## ## Classification tree: ## tree(formula = target ~ ., data = train) ## Variables actually used in tree construction: ## [1] "worst.perimeter" "worst.concave.points" "area.error" ## [4] "worst.texture" "mean.texture" "mean.concave.points" ## Number of terminal nodes: 9 ## Residual mean deviance: 0.1218 = 50.8 / 417 ## Misclassification error rate: 0.02347 = 10 / 426 pred <- predict(cancerStatus,newdata=test,type="class") confusionMatrix(pred,test$target)
## Confusion Matrix and Statistics
##
##           Reference
## Prediction  0  1
##          0 49  7
##          1  8 78
##
##                Accuracy : 0.8944
##                  95% CI : (0.8318, 0.9397)
##     No Information Rate : 0.5986
##     P-Value [Acc > NIR] : 4.641e-15
##
##                   Kappa : 0.7795
##  Mcnemar's Test P-Value : 1
##
##             Sensitivity : 0.8596
##             Specificity : 0.9176
##          Pos Pred Value : 0.8750
##          Neg Pred Value : 0.9070
##              Prevalence : 0.4014
##          Detection Rate : 0.3451
##    Detection Prevalence : 0.3944
##       Balanced Accuracy : 0.8886
##
##        'Positive' Class : 0
## 
# Plot decision tree with labels
plot(cancerStatus)
text(cancerStatus,pretty=0)

## 1.31b Decision Trees – Cross Validation – R Code

We can also perform a Cross Validation on the data to identify the Decision Tree which will give the minimum deviance.

library(tree)
cancer <- cancer[,2:32]
cancer$target <- as.factor(cancer$target)
train_idx <- trainTestSplit(cancer,trainPercent=75,seed=5)
train <- cancer[train_idx, ]
test <- cancer[-train_idx, ]

# Create Decision Tree
cancerStatus=tree(target~.,train)

# Execute 10 fold cross validation
cvCancer=cv.tree(cancerStatus)
plot(cvCancer)

# Plot the
plot(cvCancer$size,cvCancer$dev,type='b')

prunedCancer=prune.tree(cancerStatus,best=4)
plot(prunedCancer)
text(prunedCancer,pretty=0)

pred <- predict(prunedCancer,newdata=test,type="class")
confusionMatrix(pred,test$target) ## Confusion Matrix and Statistics ## ## Reference ## Prediction 0 1 ## 0 50 7 ## 1 7 78 ## ## Accuracy : 0.9014 ## 95% CI : (0.8401, 0.945) ## No Information Rate : 0.5986 ## P-Value [Acc > NIR] : 7.988e-16 ## ## Kappa : 0.7948 ## Mcnemar's Test P-Value : 1 ## ## Sensitivity : 0.8772 ## Specificity : 0.9176 ## Pos Pred Value : 0.8772 ## Neg Pred Value : 0.9176 ## Prevalence : 0.4014 ## Detection Rate : 0.3521 ## Detection Prevalence : 0.4014 ## Balanced Accuracy : 0.8974 ## ## 'Positive' Class : 0 ##  ## 1.31c Decision Trees – Python Code Below is the Python code for creating Decision Trees. The accuracy, precision, recall and F1 score is computed on the test data set. import numpy as np import pandas as pd import os import matplotlib.pyplot as plt from sklearn.metrics import confusion_matrix from sklearn import tree from sklearn.datasets import load_breast_cancer from sklearn.model_selection import train_test_split from sklearn.tree import DecisionTreeClassifier from sklearn.datasets import make_classification, make_blobs from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score import graphviz cancer = load_breast_cancer() (X_cancer, y_cancer) = load_breast_cancer(return_X_y = True) X_train, X_test, y_train, y_test = train_test_split(X_cancer, y_cancer, random_state = 0) clf = DecisionTreeClassifier().fit(X_train, y_train) print('Accuracy of Decision Tree classifier on training set: {:.2f}' .format(clf.score(X_train, y_train))) print('Accuracy of Decision Tree classifier on test set: {:.2f}' .format(clf.score(X_test, y_test))) y_predicted=clf.predict(X_test) confusion = confusion_matrix(y_test, y_predicted) print('Accuracy: {:.2f}'.format(accuracy_score(y_test, y_predicted))) print('Precision: {:.2f}'.format(precision_score(y_test, y_predicted))) print('Recall: {:.2f}'.format(recall_score(y_test, y_predicted))) print('F1: {:.2f}'.format(f1_score(y_test, y_predicted))) # Plot the Decision Tree clf = DecisionTreeClassifier(max_depth=2).fit(X_train, y_train) dot_data = tree.export_graphviz(clf, out_file=None, feature_names=cancer.feature_names, class_names=cancer.target_names, filled=True, rounded=True, special_characters=True) graph = graphviz.Source(dot_data) graph ## Accuracy of Decision Tree classifier on training set: 1.00 ## Accuracy of Decision Tree classifier on test set: 0.87 ## Accuracy: 0.87 ## Precision: 0.97 ## Recall: 0.82 ## F1: 0.89 ## 1.31d Decision Trees – Cross Validation – Python Code In the code below 5-fold cross validation is performed for different depths of the tree and the accuracy is computed. The accuracy on the test set seems to plateau when the depth is 8. But it is seen to increase again from 10 to 12. More analysis needs to be done here  import numpy as np import pandas as pd import os import matplotlib.pyplot as plt from sklearn.datasets import load_breast_cancer from sklearn.tree import DecisionTreeClassifier (X_cancer, y_cancer) = load_breast_cancer(return_X_y = True) from sklearn.cross_validation import train_test_split, KFold def computeCVAccuracy(X,y,folds): accuracy=[] foldAcc=[] depth=[1,2,3,4,5,6,7,8,9,10,11,12] nK=len(X)/float(folds) xval_err=0 for i in depth: kf = KFold(len(X),n_folds=folds) for train_index, test_index in kf: X_train, X_test = X.iloc[train_index], X.iloc[test_index] y_train, y_test = y.iloc[train_index], y.iloc[test_index] clf = DecisionTreeClassifier(max_depth = i).fit(X_train, y_train) score=clf.score(X_test, y_test) accuracy.append(score) foldAcc.append(np.mean(accuracy)) return(foldAcc) cvAccuracy=computeCVAccuracy(pd.DataFrame(X_cancer),pd.DataFrame(y_cancer),folds=10) df1=pd.DataFrame(cvAccuracy) df1.columns=['cvAccuracy'] df=df1.reindex([1,2,3,4,5,6,7,8,9,10,11,12]) df.plot() plt.title("Decision Tree - 10-fold Cross Validation Accuracy vs Depth of tree") plt.xlabel("Depth of tree") plt.ylabel("Accuracy") plt.savefig('fig3.png', bbox_inches='tight') ## 1.4a Random Forest – R code A Random Forest is fit using the Boston data. The summary shows that 4 variables were randomly chosen at each split and the resulting ML model explains 88.72% of the test data. Also the variable importance is plotted. It can be seen that ‘rooms’ and ‘status’ are the most influential features in the model library(randomForest) df=read.csv("Boston.csv",stringsAsFactors = FALSE) # Data from MASS - SL # Select specific columns Boston <- df %>% dplyr::select("crimeRate","zone","indus","charles","nox","rooms","age", "distances","highways","tax","teacherRatio","color", "status","medianValue") # Fit a Random Forest on the Boston training data rfBoston=randomForest(medianValue~.,data=Boston) # Display the summatu of the fit. It can be seen that the MSE is 10.88 # and the percentage variance explained is 86.14%. About 4 variables were tried at each # #split for a maximum tree of 500. # The MSE and percent variance is on Out of Bag trees rfBoston ## ## Call: ## randomForest(formula = medianValue ~ ., data = Boston) ## Type of random forest: regression ## Number of trees: 500 ## No. of variables tried at each split: 4 ## ## Mean of squared residuals: 9.521672 ## % Var explained: 88.72 #List and plot the variable importances importance(rfBoston) ## IncNodePurity ## crimeRate 2602.1550 ## zone 258.8057 ## indus 2599.6635 ## charles 240.2879 ## nox 2748.8485 ## rooms 12011.6178 ## age 1083.3242 ## distances 2432.8962 ## highways 393.5599 ## tax 1348.6987 ## teacherRatio 2841.5151 ## color 731.4387 ## status 12735.4046 varImpPlot(rfBoston) ## 1.4b Random Forest-OOB and Cross Validation Error – R code The figure below shows the OOB error and the Cross Validation error vs the ‘mtry’. Here mtry indicates the number of random features that are chosen at each split. The lowest test error occurs when mtry = 8 library(randomForest) df=read.csv("Boston.csv",stringsAsFactors = FALSE) # Data from MASS - SL # Select specific columns Boston <- df %>% dplyr::select("crimeRate","zone","indus","charles","nox","rooms","age", "distances","highways","tax","teacherRatio","color", "status","medianValue") # Split as training and tst sets train_idx <- trainTestSplit(Boston,trainPercent=75,seed=5) train <- Boston[train_idx, ] test <- Boston[-train_idx, ] #Initialize OOD and testError oobError <- NULL testError <- NULL # In the code below the number of variables to consider at each split is increased # from 1 - 13(max features) and the OOB error and the MSE is computed for(i in 1:13){ fitRF=randomForest(medianValue~.,data=train,mtry=i,ntree=400) oobError[i] <-fitRF$mse[400]
pred <- predict(fitRF,newdata=test)
testError[i] <- mean((pred-test$medianValue)^2) } # We can see the OOB and Test Error. It can be seen that the Random Forest performs # best with the lowers MSE at mtry=6 matplot(1:13,cbind(testError,oobError),pch=19,col=c("red","blue"), type="b",xlab="mtry(no of varaibles at each split)", ylab="Mean Squared Error", main="Random Forest - OOB and Test Error") legend("topright",legend=c("OOB","Test"),pch=19,col=c("red","blue")) ## 1.4c Random Forest – Python code The python code for Random Forest Regression is shown below. The training and test score is computed. The variable importance shows that ‘rooms’ and ‘status’ are the most influential of the variables import numpy as np import pandas as pd import os import matplotlib.pyplot as plt from sklearn.model_selection import train_test_split from sklearn.ensemble import RandomForestRegressor df = pd.read_csv("Boston.csv",encoding = "ISO-8859-1") X=df[['crimeRate','zone', 'indus','charles','nox','rooms', 'age','distances','highways','tax', 'teacherRatio','color','status']] y=df['medianValue'] X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 0) regr = RandomForestRegressor(max_depth=4, random_state=0) regr.fit(X_train, y_train) print('R-squared score (training): {:.3f}' .format(regr.score(X_train, y_train))) print('R-squared score (test): {:.3f}' .format(regr.score(X_test, y_test))) feature_names=['crimeRate','zone', 'indus','charles','nox','rooms', 'age','distances','highways','tax', 'teacherRatio','color','status'] print(regr.feature_importances_) plt.figure(figsize=(10,6),dpi=80) c_features=X_train.shape[1] plt.barh(np.arange(c_features),regr.feature_importances_) plt.xlabel("Feature importance") plt.ylabel("Feature name") plt.yticks(np.arange(c_features), feature_names) plt.tight_layout() plt.savefig('fig4.png', bbox_inches='tight')  ## R-squared score (training): 0.917 ## R-squared score (test): 0.734 ## [ 0.03437382 0. 0.00580335 0. 0.00731004 0.36461548 ## 0.00638577 0.03432173 0.0041244 0.01732328 0.01074148 0.0012638 ## 0.51373683] ## 1.4d Random Forest – Cross Validation and OOB Error – Python code As with R the ‘max_features’ determines the random number of features the random forest will use at each split. The plot shows that when max_features=8 the MSE is lowest import numpy as np import pandas as pd import os import matplotlib.pyplot as plt from sklearn.model_selection import train_test_split from sklearn.ensemble import RandomForestRegressor from sklearn.model_selection import cross_val_score df = pd.read_csv("Boston.csv",encoding = "ISO-8859-1") X=df[['crimeRate','zone', 'indus','charles','nox','rooms', 'age','distances','highways','tax', 'teacherRatio','color','status']] y=df['medianValue'] cvError=[] oobError=[] oobMSE=[] for i in range(1,13): regr = RandomForestRegressor(max_depth=4, n_estimators=400,max_features=i,oob_score=True,random_state=0) mse= np.mean(cross_val_score(regr, X, y, cv=5,scoring = 'neg_mean_squared_error')) # Since this is neg_mean_squared_error I have inverted the sign to get MSE cvError.append(-mse) # Fit on all data to compute OOB error regr.fit(X, y) # Record the OOB error for each max_features=i setting oob = 1 - regr.oob_score_ oobError.append(oob) # Get the Out of Bag prediction oobPred=regr.oob_prediction_ # Compute the Mean Squared Error between OOB Prediction and target mseOOB=np.mean(np.square(oobPred-y)) oobMSE.append(mseOOB) # Plot the CV Error and OOB Error # Set max_features maxFeatures=np.arange(1,13) cvError=pd.DataFrame(cvError,index=maxFeatures) oobMSE=pd.DataFrame(oobMSE,index=maxFeatures) #Plot fig8=df.plot() fig8=plt.title('Random forest - CV Error and OOB Error vs max_features') fig8.figure.savefig('fig8.png', bbox_inches='tight')  #Plot the OOB Error vs max_features plt.plot(range(1,13),oobError) fig2=plt.title("Random Forest - OOB Error vs max_features (variable no of features)") fig2=plt.xlabel("max_features (variable no of features)") fig2=plt.ylabel("OOB Error") fig2.figure.savefig('fig7.png', bbox_inches='tight')  ## 1.5a Boosting – R code Here a Gradient Boosted ML Model is built with a n.trees=5000, with a learning rate of 0.01 and depth of 4. The feature importance plot also shows that rooms and status are the 2 most important features. The MSE vs the number of trees plateaus around 2000 trees library(gbm) # Perform gradient boosting on the Boston data set. The distribution is gaussian since we # doing MSE. The interaction depth specifies the number of splits boostBoston=gbm(medianValue~.,data=train,distribution="gaussian",n.trees=5000, shrinkage=0.01,interaction.depth=4) #The summary gives the variable importance. The 2 most significant variables are # number of rooms and lower status summary(boostBoston) ## var rel.inf ## rooms rooms 42.2267200 ## status status 27.3024671 ## distances distances 7.9447972 ## crimeRate crimeRate 5.0238827 ## nox nox 4.0616548 ## teacherRatio teacherRatio 3.1991999 ## age age 2.7909772 ## color color 2.3436295 ## tax tax 2.1386213 ## charles charles 1.3799109 ## highways highways 0.7644026 ## indus indus 0.7236082 ## zone zone 0.1001287 # The plots below show how each variable relates to the median value of the home. As # the number of roomd increase the median value increases and with increase in lower status # the median value decreases par(mfrow=c(1,2)) #Plot the relation between the top 2 features and the target plot(boostBoston,i="rooms") plot(boostBoston,i="status") # Create a sequence of trees between 100-5000 incremented by 50 nTrees=seq(100,5000,by=50) # Predict the values for the test data pred <- predict(boostBoston,newdata=test,n.trees=nTrees) # Compute the mean for each of the MSE for each of the number of trees boostError <- apply((pred-test$medianValue)^2,2,mean)
#Plot the MSE vs the number of trees
plot(nTrees,boostError,pch=19,col="blue",ylab="Mean Squared Error",
main="Boosting Test Error")

## 1.5b Cross Validation Boosting – R code

Included below is a cross validation error vs the learning rate. The lowest error is when learning rate = 0.09

cvError <- NULL
s <- c(.001,0.01,0.03,0.05,0.07,0.09,0.1)
for(i in seq_along(s)){
cvBoost=gbm(medianValue~.,data=train,distribution="gaussian",n.trees=5000,
shrinkage=s[i],interaction.depth=4,cv.folds=5)
cvError[i] <- mean(cvBoost$cv.error) } # Create a data frame for plotting a <- rbind(s,cvError) b <- as.data.frame(t(a)) # It can be seen that a shrinkage parameter of 0,05 gives the lowes CV Error ggplot(b,aes(s,cvError)) + geom_point() + geom_line(color="blue") + xlab("Shrinkage") + ylab("Cross Validation Error") + ggtitle("Gradient boosted trees - Cross Validation error vs Shrinkage") ## 1.5c Boosting – Python code A gradient boost ML model in Python is created below. The Rsquared score is computed on the training and test data. import numpy as np import pandas as pd import os import matplotlib.pyplot as plt from sklearn.model_selection import train_test_split from sklearn.ensemble import GradientBoostingRegressor df = pd.read_csv("Boston.csv",encoding = "ISO-8859-1") X=df[['crimeRate','zone', 'indus','charles','nox','rooms', 'age','distances','highways','tax', 'teacherRatio','color','status']] y=df['medianValue'] X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 0) regr = GradientBoostingRegressor() regr.fit(X_train, y_train) print('R-squared score (training): {:.3f}' .format(regr.score(X_train, y_train))) print('R-squared score (test): {:.3f}' .format(regr.score(X_test, y_test))) ## R-squared score (training): 0.983 ## R-squared score (test): 0.821 ## 1.5c Cross Validation Boosting – Python code the cross validation error is computed as the learning rate is varied. The minimum CV eror occurs when lr = 0.04 import numpy as np import pandas as pd import os import matplotlib.pyplot as plt from sklearn.model_selection import train_test_split from sklearn.ensemble import RandomForestRegressor from sklearn.ensemble import GradientBoostingRegressor from sklearn.model_selection import cross_val_score df = pd.read_csv("Boston.csv",encoding = "ISO-8859-1") X=df[['crimeRate','zone', 'indus','charles','nox','rooms', 'age','distances','highways','tax', 'teacherRatio','color','status']] y=df['medianValue'] cvError=[] learning_rate =[.001,0.01,0.03,0.05,0.07,0.09,0.1] for lr in learning_rate: regr = GradientBoostingRegressor(max_depth=4, n_estimators=400,learning_rate =lr,random_state=0) mse= np.mean(cross_val_score(regr, X, y, cv=10,scoring = 'neg_mean_squared_error')) # Since this is neg_mean_squared_error I have inverted the sign to get MSE cvError.append(-mse) learning_rate =[.001,0.01,0.03,0.05,0.07,0.09,0.1] plt.plot(learning_rate,cvError) plt.title("Gradient Boosting - 5-fold CV- Mean Squared Error vs max_features (variable no of features)") plt.xlabel("max_features (variable no of features)") plt.ylabel("Mean Squared Error") plt.savefig('fig6.png', bbox_inches='tight') Conclusion This post covered Splines and Tree based ML models like Bagging, Random Forest and Boosting. Stay tuned for further updates. You may also like To see all posts see Index of posts # Practical Machine Learning with R and Python – Part 4 This is the 4th installment of my ‘Practical Machine Learning with R and Python’ series. In this part I discuss classification with Support Vector Machines (SVMs), using both a Linear and a Radial basis kernel, and Decision Trees. Further, a closer look is taken at some of the metrics associated with binary classification, namely accuracy vs precision and recall. I also touch upon Validation curves, Precision-Recall, ROC curves and AUC with equivalent code in R and Python This post is a continuation of my 3 earlier posts on Practical Machine Learning in R and Python 1. Practical Machine Learning with R and Python – Part 1 2. Practical Machine Learning with R and Python – Part 2 3. Practical Machine Learning with R and Python – Part 3 The RMarkdown file with the code and the associated data files can be downloaded from Github at MachineLearning-RandPython-Part4 Note: Please listen to my video presentations Machine Learning in youtube 1. Machine Learning in plain English-Part 1 2. Machine Learning in plain English-Part 2 3. Machine Learning in plain English-Part 3 Check out my compact and minimal book “Practical Machine Learning with R and Python:Third edition- Machine Learning in stereo” available in Amazon in paperback($12.99) and kindle(8.99) versions. My book includes implementations of key ML algorithms and associated measures and metrics. The book is ideal for anybody who is familiar with the concepts and would like a quick reference to the different ML algorithms that can be applied to problems and how to select the best model. Pick your copy today!! Support Vector Machines (SVM) are another useful Machine Learning model that can be used for both regression and classification problems. SVMs used in classification, compute the hyperplane, that separates the 2 classes with the maximum margin. To do this the features may be transformed into a larger multi-dimensional feature space. SVMs can be used with different kernels namely linear, polynomial or radial basis to determine the best fitting model for a given classification problem. In the 2nd part of this series Practical Machine Learning with R and Python – Part 2, I had mentioned the various metrics that are used in classification ML problems namely Accuracy, Precision, Recall and F1 score. Accuracy gives the fraction of data that were correctly classified as belonging to the +ve or -ve class. However ‘accuracy’ in itself is not a good enough measure because it does not take into account the fraction of the data that were incorrectly classified. This issue becomes even more critical in different domains. For e.g a surgeon who would like to detect cancer, would like to err on the side of caution, and classify even a possibly non-cancerous patient as possibly having cancer, rather than mis-classifying a malignancy as benign. Here we would like to increase recall or sensitivity which is given by Recall= TP/(TP+FN) or we try reduce mis-classification by either increasing the (true positives) TP or reducing (false negatives) FN On the other hand, search algorithms would like to increase precision which tries to reduce the number of irrelevant results in the search result. Precision= TP/(TP+FP). In other words we do not want ‘false positives’ or irrelevant results to come in the search results and there is a need to reduce the false positives. When we try to increase ‘precision’, we do so at the cost of ‘recall’, and vice-versa. I found this diagram and explanation in Wikipedia very useful Source: Wikipedia “Consider a brain surgeon tasked with removing a cancerous tumor from a patient’s brain. The surgeon needs to remove all of the tumor cells since any remaining cancer cells will regenerate the tumor. Conversely, the surgeon must not remove healthy brain cells since that would leave the patient with impaired brain function. The surgeon may be more liberal in the area of the brain she removes to ensure she has extracted all the cancer cells. This decision increases recall but reduces precision. On the other hand, the surgeon may be more conservative in the brain she removes to ensure she extracts only cancer cells. This decision increases precision but reduces recall. That is to say, greater recall increases the chances of removing healthy cells (negative outcome) and increases the chances of removing all cancer cells (positive outcome). Greater precision decreases the chances of removing healthy cells (positive outcome) but also decreases the chances of removing all cancer cells (negative outcome).” ## 1.1a. Linear SVM – R code In R code below I use SVM with linear kernel source('RFunctions-1.R') library(dplyr) library(e1071) library(caret) library(reshape2) library(ggplot2) # Read data. Data from SKLearn cancer <- read.csv("cancer.csv") cancertarget <- as.factor(cancer$target) # Split into training and test sets train_idx <- trainTestSplit(cancer,trainPercent=75,seed=5) train <- cancer[train_idx, ] test <- cancer[-train_idx, ] # Fit a linear basis kernel. DO not scale the data svmfit=svm(target~., data=train, kernel="linear",scale=FALSE) ypred=predict(svmfit,test) #Print a confusion matrix confusionMatrix(ypred,test$target)
## Confusion Matrix and Statistics
##
##           Reference
## Prediction  0  1
##          0 54  3
##          1  3 82
##
##                Accuracy : 0.9577
##                  95% CI : (0.9103, 0.9843)
##     No Information Rate : 0.5986
##     P-Value [Acc > NIR] : <2e-16
##
##                   Kappa : 0.9121
##  Mcnemar's Test P-Value : 1
##
##             Sensitivity : 0.9474
##             Specificity : 0.9647
##          Pos Pred Value : 0.9474
##          Neg Pred Value : 0.9647
##              Prevalence : 0.4014
##          Detection Rate : 0.3803
##    Detection Prevalence : 0.4014
##       Balanced Accuracy : 0.9560
##
##        'Positive' Class : 0
## 

## 1.1b Linear SVM – Python code

The code below creates a SVM with linear basis in Python and also dumps the corresponding classification metrics

import numpy as np
import pandas as pd
import os
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.svm import LinearSVC

from sklearn.datasets import make_classification, make_blobs

from sklearn.metrics import confusion_matrix
from matplotlib.colors import ListedColormap
(X_cancer, y_cancer) = load_breast_cancer(return_X_y = True)
X_train, X_test, y_train, y_test = train_test_split(X_cancer, y_cancer,
random_state = 0)
clf = LinearSVC().fit(X_train, y_train)
print('Breast cancer dataset')
print('Accuracy of Linear SVC classifier on training set: {:.2f}'
.format(clf.score(X_train, y_train)))
print('Accuracy of Linear SVC classifier on test set: {:.2f}'
.format(clf.score(X_test, y_test)))
## Breast cancer dataset
## Accuracy of Linear SVC classifier on training set: 0.92
## Accuracy of Linear SVC classifier on test set: 0.94

## 1.2 Dummy classifier

Often when we perform classification tasks using any ML model namely logistic regression, SVM, neural networks etc. it is very useful to determine how well the ML model performs agains at dummy classifier. A dummy classifier uses some simple computation like frequency of majority class, instead of fitting and ML model. It is essential that our ML model does much better that the dummy classifier. This problem is even more important in imbalanced classes where we have only about 10% of +ve samples. If any ML model we create has a accuracy of about 0.90 then it is evident that our classifier is not doing any better than a dummy classsfier which can just take a majority count of this imbalanced class and also come up with 0.90. We need to be able to do better than that.

In the examples below (1.3a & 1.3b) it can be seen that SVMs with ‘radial basis’ kernel with unnormalized data, for both R and Python, do not perform any better than the dummy classifier.

## 1.2a Dummy classifier – R code

R does not seem to have an explicit dummy classifier. I created a simple dummy classifier that predicts the majority class. SKlearn in Python also includes other strategies like uniform, stratified etc. but this should be possible to create in R also.

# Create a simple dummy classifier that computes the ratio of the majority class to the totla
DummyClassifierAccuracy <- function(train,test,type="majority"){
if(type=="majority"){
count <- sum(train$target==1)/dim(train)[1] } count } cancer <- read.csv("cancer.csv") cancer$target <- as.factor(cancer$target) # Create training and test sets train_idx <- trainTestSplit(cancer,trainPercent=75,seed=5) train <- cancer[train_idx, ] test <- cancer[-train_idx, ] #Dummy classifier majority class acc=DummyClassifierAccuracy(train,test) sprintf("Accuracy is %f",acc) ## [1] "Accuracy is 0.638498" ## 1.2b Dummy classifier – Python code This dummy classifier uses the majority class. import numpy as np import pandas as pd import os import matplotlib.pyplot as plt from sklearn.datasets import load_breast_cancer from sklearn.model_selection import train_test_split from sklearn.dummy import DummyClassifier from sklearn.metrics import confusion_matrix (X_cancer, y_cancer) = load_breast_cancer(return_X_y = True) X_train, X_test, y_train, y_test = train_test_split(X_cancer, y_cancer, random_state = 0) # Negative class (0) is most frequent dummy_majority = DummyClassifier(strategy = 'most_frequent').fit(X_train, y_train) y_dummy_predictions = dummy_majority.predict(X_test) print('Dummy classifier accuracy on test set: {:.2f}' .format(dummy_majority.score(X_test, y_test)))  ## Dummy classifier accuracy on test set: 0.63 ## 1.3a – Radial SVM (un-normalized) – R code SVMs perform better when the data is normalized or scaled. The 2 examples below show that SVM with radial basis kernel does not perform any better than the dummy classifier library(dplyr) library(e1071) library(caret) library(reshape2) library(ggplot2) # Radial SVM unnormalized train_idx <- trainTestSplit(cancer,trainPercent=75,seed=5) train <- cancer[train_idx, ] test <- cancer[-train_idx, ] # Unnormalized data svmfit=svm(target~., data=train, kernel="radial",cost=10,scale=FALSE) ypred=predict(svmfit,test) confusionMatrix(ypred,test$target)
## Confusion Matrix and Statistics
##
##           Reference
## Prediction  0  1
##          0  0  0
##          1 57 85
##
##                Accuracy : 0.5986
##                  95% CI : (0.5131, 0.6799)
##     No Information Rate : 0.5986
##     P-Value [Acc > NIR] : 0.5363
##
##                   Kappa : 0
##  Mcnemar's Test P-Value : 1.195e-13
##
##             Sensitivity : 0.0000
##             Specificity : 1.0000
##          Pos Pred Value :    NaN
##          Neg Pred Value : 0.5986
##              Prevalence : 0.4014
##          Detection Rate : 0.0000
##    Detection Prevalence : 0.0000
##       Balanced Accuracy : 0.5000
##
##        'Positive' Class : 0
## 

## 1.4b – Radial SVM (un-normalized) – Python code

import numpy as np
import pandas as pd
import os
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC

(X_cancer, y_cancer) = load_breast_cancer(return_X_y = True)
X_train, X_test, y_train, y_test = train_test_split(X_cancer, y_cancer,
random_state = 0)

clf = SVC(C=10).fit(X_train, y_train)
print('Breast cancer dataset (unnormalized features)')
print('Accuracy of RBF-kernel SVC on training set: {:.2f}'
.format(clf.score(X_train, y_train)))
print('Accuracy of RBF-kernel SVC on test set: {:.2f}'
.format(clf.score(X_test, y_test)))
## Breast cancer dataset (unnormalized features)
## Accuracy of RBF-kernel SVC on training set: 1.00
## Accuracy of RBF-kernel SVC on test set: 0.63

## 1.5a – Radial SVM (Normalized) -R Code

The data is scaled (normalized ) before using the SVM model. The SVM model has 2 paramaters a) C – Large C (less regularization), more regularization b) gamma – Small gamma has larger decision boundary with more misclassfication, and larger gamma has tighter decision boundary

The R code below computes the accuracy as the regularization paramater is changed

trainingAccuracy <- NULL
testAccuracy <- NULL
C1 <- c(.01,.1, 1, 10, 20)
for(i in  C1){

ypredTrain <-predict(svmfit,train)
ypredTest=predict(svmfit,test)
a <-confusionMatrix(ypredTrain,train$target) b <-confusionMatrix(ypredTest,test$target)
trainingAccuracy <-c(trainingAccuracy,a$overall[1]) testAccuracy <-c(testAccuracy,b$overall[1])

}
print(trainingAccuracy)
##  Accuracy  Accuracy  Accuracy  Accuracy  Accuracy
## 0.6384977 0.9671362 0.9906103 0.9976526 1.0000000
print(testAccuracy)
##  Accuracy  Accuracy  Accuracy  Accuracy  Accuracy
## 0.5985915 0.9507042 0.9647887 0.9507042 0.9507042
a <-rbind(C1,as.numeric(trainingAccuracy),as.numeric(testAccuracy))
b <- data.frame(t(a))
names(b) <- c("C1","trainingAccuracy","testAccuracy")
df <- melt(b,id="C1")
ggplot(df) + geom_line(aes(x=C1, y=value, colour=variable),size=2) +
xlab("C (SVC regularization)value") + ylab("Accuracy") +
ggtitle("Training and test accuracy vs C(regularization)")

## 1.5b – Radial SVM (normalized) – Python

The Radial basis kernel is used on normalized data for a range of ‘C’ values and the result is plotted.

import numpy as np
import pandas as pd
import os
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()

(X_cancer, y_cancer) = load_breast_cancer(return_X_y = True)
X_train, X_test, y_train, y_test = train_test_split(X_cancer, y_cancer,
random_state = 0)
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

print('Breast cancer dataset (normalized with MinMax scaling)')
trainingAccuracy=[]
testAccuracy=[]
for C1 in [.01,.1, 1, 10, 20]:
clf = SVC(C=C1).fit(X_train_scaled, y_train)
acctrain=clf.score(X_train_scaled, y_train)
accTest=clf.score(X_test_scaled, y_test)
trainingAccuracy.append(acctrain)
testAccuracy.append(accTest)

# Create a dataframe
C1=[.01,.1, 1, 10, 20]
trainingAccuracy=pd.DataFrame(trainingAccuracy,index=C1)
testAccuracy=pd.DataFrame(testAccuracy,index=C1)

# Plot training and test R squared as a function of alpha
df=pd.concat([trainingAccuracy,testAccuracy],axis=1)
df.columns=['trainingAccuracy','trainingAccuracy']

fig1=df.plot()
fig1=plt.title('Training and test accuracy vs C (SVC)')
fig1.figure.savefig('fig1.png', bbox_inches='tight')
## Breast cancer dataset (normalized with MinMax scaling)

Output image:

## 1.6a Validation curve – R code

Sklearn includes code creating validation curves by varying paramaters and computing and plotting accuracy as gamma or C or changd. I did not find this R but I think this is a useful function and so I have created the R equivalent of this.

# The R equivalent of np.logspace
seqLogSpace <- function(start,stop,len){
a=seq(log10(10^start),log10(10^stop),length=len)
10^a
}

# Read the data. This is taken the SKlearn cancer data
cancer$target <- as.factor(cancer$target)

set.seed(6)

# Create the range of C1 in log space
param_range = seqLogSpace(-3,2,20)
# Initialize the overall training and test accuracy to NULL
overallTrainAccuracy <- NULL
overallTestAccuracy <- NULL

# Loop over the parameter range of Gamma
for(i in param_range){
# Set no of folds
noFolds=5
# Create the rows which fall into different folds from 1..noFolds
folds = sample(1:noFolds, nrow(cancer), replace=TRUE)
# Initialize the training and test accuracy of folds to 0
trainingAccuracy <- 0
testAccuracy <- 0

# Loop through the folds
for(j in 1:noFolds){
# The training is all rows for which the row is != j (k-1 folds -> training)
train <- cancer[folds!=j,]
# The rows which have j as the index become the test set
test <- cancer[folds==j,]
# Create a SVM model for this

# Add up all the fold accuracy for training and test separately
ypredTrain <-predict(svmfit,train)
ypredTest=predict(svmfit,test)

# Create confusion matrix
a <-confusionMatrix(ypredTrain,train$target) b <-confusionMatrix(ypredTest,test$target)
# Get the accuracy
trainingAccuracy <-trainingAccuracy + a$overall[1] testAccuracy <-testAccuracy+b$overall[1]

}
# Compute the average of accuracy for K folds for number of features 'i'
overallTrainAccuracy=c(overallTrainAccuracy,trainingAccuracy/noFolds)
overallTestAccuracy=c(overallTestAccuracy,testAccuracy/noFolds)
}
#Create a dataframe
a <- rbind(param_range,as.numeric(overallTrainAccuracy),
as.numeric(overallTestAccuracy))
b <- data.frame(t(a))
names(b) <- c("C1","trainingAccuracy","testAccuracy")
df <- melt(b,id="C1")
#Plot in log axis
ggplot(df) + geom_line(aes(x=C1, y=value, colour=variable),size=2) +
xlab("C (SVC regularization)value") + ylab("Accuracy") +
ggtitle("Training and test accuracy vs C(regularization)") + scale_x_log10()

## 1.6b Validation curve – Python

Compute and plot the validation curve as gamma is varied.

import numpy as np
import pandas as pd
import os
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler
from sklearn.svm import SVC
from sklearn.model_selection import validation_curve

(X_cancer, y_cancer) = load_breast_cancer(return_X_y = True)
scaler = MinMaxScaler()
X_scaled = scaler.fit_transform(X_cancer)

# Create a gamma values from 10^-3 to 10^2 with 20 equally spaced intervals
param_range = np.logspace(-3, 2, 20)
# Compute the validation curve
train_scores, test_scores = validation_curve(SVC(), X_scaled, y_cancer,
param_name='gamma',
param_range=param_range, cv=10)

#Plot the figure
fig2=plt.figure()

#Compute the mean
train_scores_mean = np.mean(train_scores, axis=1)
train_scores_std = np.std(train_scores, axis=1)
test_scores_mean = np.mean(test_scores, axis=1)
test_scores_std = np.std(test_scores, axis=1)

fig2=plt.title('Validation Curve with SVM')
fig2=plt.xlabel('$\gamma$ (gamma)')
fig2=plt.ylabel('Score')
fig2=plt.ylim(0.0, 1.1)
lw = 2

fig2=plt.semilogx(param_range, train_scores_mean, label='Training score',
color='darkorange', lw=lw)

fig2=plt.fill_between(param_range, train_scores_mean - train_scores_std,
train_scores_mean + train_scores_std, alpha=0.2,
color='darkorange', lw=lw)

fig2=plt.semilogx(param_range, test_scores_mean, label='Cross-validation score',
color='navy', lw=lw)

fig2=plt.fill_between(param_range, test_scores_mean - test_scores_std,
test_scores_mean + test_scores_std, alpha=0.2,
color='navy', lw=lw)
fig2.figure.savefig('fig2.png', bbox_inches='tight')


Output image:

## 1.7a Validation Curve (Preventing data leakage) – Python code

In this course Applied Machine Learning in Python, the Professor states that when we apply the same data transformation to a entire dataset, it will cause a data leakage. “The proper way to do cross-validation when you need to scale the data is not to scale the entire dataset with a single transform, since this will indirectly leak information into the training data about the whole dataset, including the test data (see the lecture on data leakage later in the course). Instead, scaling/normalizing must be computed and applied for each cross-validation fold separately”

So I apply separate scaling to the training and testing folds and plot. In the lecture the Prof states that this can be done using pipelines.

import numpy as np
import pandas as pd
import os
import matplotlib.pyplot as plt
from sklearn.cross_validation import  KFold
from sklearn.preprocessing import MinMaxScaler
from sklearn.svm import SVC

(X_cancer, y_cancer) = load_breast_cancer(return_X_y = True)
# Set the parameter range
param_range = np.logspace(-3, 2, 20)

# Set number of folds
folds=5
#Initialize
overallTrainAccuracy=[]
overallTestAccuracy=[]

# Loop over the paramater range
for c in  param_range:
trainingAccuracy=0
testAccuracy=0
kf = KFold(len(X_cancer),n_folds=folds)
# Partition into training and test folds
for train_index, test_index in kf:
# Partition the data acccording the fold indices generated
X_train, X_test = X_cancer[train_index], X_cancer[test_index]
y_train, y_test = y_cancer[train_index], y_cancer[test_index]

# Scale the X_train and X_test
scaler = MinMaxScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
# Fit a SVC model for each C
clf = SVC(C=c).fit(X_train_scaled, y_train)
#Compute the training and test score
acctrain=clf.score(X_train_scaled, y_train)
accTest=clf.score(X_test_scaled, y_test)
trainingAccuracy += np.sum(acctrain)
testAccuracy += np.sum(accTest)
# Compute the mean training and testing accuracy
overallTrainAccuracy.append(trainingAccuracy/folds)
overallTestAccuracy.append(testAccuracy/folds)

overallTrainAccuracy=pd.DataFrame(overallTrainAccuracy,index=param_range)
overallTestAccuracy=pd.DataFrame(overallTestAccuracy,index=param_range)

# Plot training and test R squared as a function of alpha
df=pd.concat([overallTrainAccuracy,overallTestAccuracy],axis=1)
df.columns=['trainingAccuracy','testAccuracy']

fig3=plt.title('Validation Curve with SVM')
fig3=plt.xlabel('$\gamma$ (gamma)')
fig3=plt.ylabel('Score')
fig3=plt.ylim(0.5, 1.1)
lw = 2

fig3=plt.semilogx(param_range, overallTrainAccuracy, label='Training score',
color='darkorange', lw=lw)

fig3=plt.semilogx(param_range, overallTestAccuracy, label='Cross-validation score',
color='navy', lw=lw)

fig3=plt.legend(loc='best')
fig3.figure.savefig('fig3.png', bbox_inches='tight')


Output image:

## 1.8 a Decision trees – R code

Decision trees in R can be plotted using RPart package

library(rpart)
library(rpart.plot)
rpart = NULL
# Create a decision tree
m <-rpart(Species~.,data=iris)
#Plot
rpart.plot(m,extra=2,main="Decision Tree - IRIS")

## 1.8 b Decision trees – Python code

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier
from sklearn import tree
from sklearn.model_selection import train_test_split
import graphviz

X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, random_state = 3)
clf = DecisionTreeClassifier().fit(X_train, y_train)

print('Accuracy of Decision Tree classifier on training set: {:.2f}'
.format(clf.score(X_train, y_train)))
print('Accuracy of Decision Tree classifier on test set: {:.2f}'
.format(clf.score(X_test, y_test)))

dot_data = tree.export_graphviz(clf, out_file=None,
feature_names=iris.feature_names,
class_names=iris.target_names,
filled=True, rounded=True,
special_characters=True)
graph = graphviz.Source(dot_data)
graph
## Accuracy of Decision Tree classifier on training set: 1.00
## Accuracy of Decision Tree classifier on test set: 0.97

## 1.9a Feature importance – R code

I found the following code which had a snippet for feature importance. Sklean has a nice method for this. For some reason the results in R and Python are different. Any thoughts?

set.seed(3)
library(mlbench)
library(caret)
cancer$target <- as.factor(cancer$target)
# Split as data
data <- cancer[,1:31]
target <- cancer[,32]

# Train the model
model <- train(data, target, method="rf", preProcess="scale", trControl=trainControl(method = "cv"))
# Compute variable importance
importance <- varImp(model)
# summarize importance
print(importance)
# plot importance
plot(importance)

## 1.9b Feature importance – Python code

import numpy as np
import pandas as pd
import os
import matplotlib.pyplot as plt
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
import numpy as np
(X_cancer, y_cancer) = load_breast_cancer(return_X_y = True)
X_train, X_test, y_train, y_test = train_test_split(X_cancer, y_cancer, random_state = 0)
# Use the DecisionTreClassifier
clf = DecisionTreeClassifier(max_depth = 4, min_samples_leaf = 8,
random_state = 0).fit(X_train, y_train)

c_features=len(cancer.feature_names)
print('Breast cancer dataset: decision tree')
print('Accuracy of DT classifier on training set: {:.2f}'
.format(clf.score(X_train, y_train)))
print('Accuracy of DT classifier on test set: {:.2f}'
.format(clf.score(X_test, y_test)))

# Plot the feature importances
fig4=plt.figure(figsize=(10,6),dpi=80)

fig4=plt.barh(range(c_features), clf.feature_importances_)
fig4=plt.xlabel("Feature importance")
fig4=plt.ylabel("Feature name")
fig4=plt.yticks(np.arange(c_features), cancer.feature_names)
fig4=plt.tight_layout()
plt.savefig('fig4.png', bbox_inches='tight')

## Breast cancer dataset: decision tree
## Accuracy of DT classifier on training set: 0.96
## Accuracy of DT classifier on test set: 0.94

Output image:

## 1.10a Precision-Recall, ROC curves & AUC- R code

I tried several R packages for plotting the Precision and Recall and AUC curve. PRROC seems to work well. The Precision-Recall curves show the tradeoff between precision and recall. The higher the precision, the lower the recall and vice versa.AUC curves that hug the top left corner indicate a high sensitivity,specificity and an excellent accuracy.

source("RFunctions-1.R")
library(dplyr)
library(caret)
library(e1071)
library(PRROC)
# Read the data (this data is from sklearn!)
digits <- d[2:66]
digits$X64 <- as.factor(digits$X64)

# Split as training and test sets
train_idx <- trainTestSplit(digits,trainPercent=75,seed=5)
train <- digits[train_idx, ]
test <- digits[-train_idx, ]

# Fit a SVM model with linear basis kernel with probabilities
svmfit=svm(X64~., data=train, kernel="linear",scale=FALSE,probability=TRUE)
ypred=predict(svmfit,test,probability=TRUE)
head(attr(ypred,"probabilities"))
##               0            1
## 6  7.395947e-01 2.604053e-01
## 8  9.999998e-01 1.842555e-07
## 12 1.655178e-05 9.999834e-01
## 13 9.649997e-01 3.500032e-02
## 15 9.994849e-01 5.150612e-04
## 16 9.999987e-01 1.280700e-06
# Store the probability of 0s and 1s
m0<-attr(ypred,"probabilities")[,1]
m1<-attr(ypred,"probabilities")[,2]

# Create a dataframe of scores
scores <- data.frame(m1,test$X64) # Class 0 is data points of +ve class (in this case, digit 1) and -ve class (digit 0) #Compute Precision Recall pr <- pr.curve(scores.class0=scores[scores$test.X64=="1",]$m1, scores.class1=scores[scores$test.X64=="0",]$m1, curve=T) # Plot precision-recall curve plot(pr) #Plot the ROC curve roc<-roc.curve(m0, m1,curve=TRUE) plot(roc) ## 1.10b Precision-Recall, ROC curves & AUC- Python code For Python Logistic Regression is used to plot Precision Recall, ROC curve and compute AUC import numpy as np import pandas as pd import matplotlib.pyplot as plt from sklearn.model_selection import train_test_split from sklearn.linear_model import LogisticRegression from sklearn.datasets import load_digits from sklearn.metrics import precision_recall_curve from sklearn.metrics import roc_curve, auc #Load the digits dataset = load_digits() X, y = dataset.data, dataset.target #Create 2 classes -i) Digit 1 (from digit 1) ii) Digit 0 (from all other digits) # Make a copy of the target z= y.copy() # Replace all non 1's as 0 z[z != 1] = 0 X_train, X_test, y_train, y_test = train_test_split(X, z, random_state=0) # Fit a LR model lr = LogisticRegression().fit(X_train, y_train) #Compute the decision scores y_scores_lr = lr.fit(X_train, y_train).decision_function(X_test) y_score_list = list(zip(y_test[0:20], y_scores_lr[0:20])) #Show the decision_function scores for first 20 instances y_score_list precision, recall, thresholds = precision_recall_curve(y_test, y_scores_lr) closest_zero = np.argmin(np.abs(thresholds)) closest_zero_p = precision[closest_zero] closest_zero_r = recall[closest_zero] #Plot plt.figure() plt.xlim([0.0, 1.01]) plt.ylim([0.0, 1.01]) plt.plot(precision, recall, label='Precision-Recall Curve') plt.plot(closest_zero_p, closest_zero_r, 'o', markersize = 12, fillstyle = 'none', c='r', mew=3) plt.xlabel('Precision', fontsize=16) plt.ylabel('Recall', fontsize=16) plt.axes().set_aspect('equal') plt.savefig('fig5.png', bbox_inches='tight') #Compute and plot the ROC y_score_lr = lr.fit(X_train, y_train).decision_function(X_test) fpr_lr, tpr_lr, _ = roc_curve(y_test, y_score_lr) roc_auc_lr = auc(fpr_lr, tpr_lr) plt.figure() plt.xlim([-0.01, 1.00]) plt.ylim([-0.01, 1.01]) plt.plot(fpr_lr, tpr_lr, lw=3, label='LogRegr ROC curve (area = {:0.2f})'.format(roc_auc_lr)) plt.xlabel('False Positive Rate', fontsize=16) plt.ylabel('True Positive Rate', fontsize=16) plt.title('ROC curve (1-of-10 digits classifier)', fontsize=16) plt.legend(loc='lower right', fontsize=13) plt.plot([0, 1], [0, 1], color='navy', lw=3, linestyle='--') plt.axes() plt.savefig('fig6.png', bbox_inches='tight')  output ## 1.10c Precision-Recall, ROC curves & AUC- Python code In the code below classification probabilities are used to compute and plot precision-recall, roc and AUC import numpy as np import pandas as pd import matplotlib.pyplot as plt from sklearn.model_selection import train_test_split from sklearn.datasets import load_digits from sklearn.svm import LinearSVC from sklearn.calibration import CalibratedClassifierCV dataset = load_digits() X, y = dataset.data, dataset.target # Make a copy of the target z= y.copy() # Replace all non 1's as 0 z[z != 1] = 0 X_train, X_test, y_train, y_test = train_test_split(X, z, random_state=0) svm = LinearSVC() # Need to use CalibratedClassifierSVC to redict probabilities for lInearSVC clf = CalibratedClassifierCV(svm) clf.fit(X_train, y_train) y_proba_lr = clf.predict_proba(X_test) from sklearn.metrics import precision_recall_curve precision, recall, thresholds = precision_recall_curve(y_test, y_proba_lr[:,1]) closest_zero = np.argmin(np.abs(thresholds)) closest_zero_p = precision[closest_zero] closest_zero_r = recall[closest_zero] #plt.figure(figsize=(15,15),dpi=80) plt.figure() plt.xlim([0.0, 1.01]) plt.ylim([0.0, 1.01]) plt.plot(precision, recall, label='Precision-Recall Curve') plt.plot(closest_zero_p, closest_zero_r, 'o', markersize = 12, fillstyle = 'none', c='r', mew=3) plt.xlabel('Precision', fontsize=16) plt.ylabel('Recall', fontsize=16) plt.axes().set_aspect('equal') plt.savefig('fig7.png', bbox_inches='tight') # Analysis of IPL T20 matches with yorkr templates ## Introduction In this post I create RMarkdown templates for end-to-end analysis of IPL T20 matches, that are available on Cricsheet based on my R package yorkr. With these templates you can convert all IPL data which is in yaml format to R dataframes. Further I create data and the necessary templates for analyzing IPL matches, teams and players. All of these can be accessed at yorkrIPLTemplate. If you are passionate about cricket, and love analyzing cricket performances, then check out my 2 racy books on cricket! In my books, I perform detailed yet compact analysis of performances of both batsmen, bowlers besides evaluating team & match performances in Tests , ODIs, T20s & IPL. You can buy my books on cricket from Amazon at$12.99 for the paperback and $4.99/$6.99 respectively for the kindle versions. The books can be accessed at Cricket analytics with cricketr  and Beaten by sheer pace-Cricket analytics with yorkr  A must read for any cricket lover! Check it out!!

9/Rs 320 and $6.99/Rs448 respectively The templates are 1. Template for conversion and setup – IPLT20Template.Rmd 2. Any IPL match – IPLMatchtemplate.Rmd 3. IPL matches between 2 nations – IPLMatches2TeamTemplate.Rmd 4. A IPL nations performance against all other IPL nations – IPLAllMatchesAllOppnTemplate.Rmd 5. Analysis of IPL batsmen and bowlers of all IPL nations – IPLBatsmanBowlerTemplate.Rmd Besides the templates the repository also includes the converted data for all IPL matches I downloaded from Cricsheet in Dec 2016. So this data is complete till the 2016 IPL season. You can recreate the files as more matches are added to Cricsheet site in IPL 2017 and future seasons. This post contains all the steps needed for detailed analysis of IPL matches, teams and IPL player. This will also be my reference in future if I decide to analyze IPL in future! See my earlier posts where I analyze IPL T20 1. yorkr crashes the IPL party ! – Part 1 2. yorkr crashes the IPL party! – Part 2 3. yorkr crashes the IPL party! – Part 3! 4. yorkr crashes the IPL party! – Part 4 There will be 5 folders at the root 1. IPLdata – Match files as yaml from Cricsheet 2. IPLMatches – Yaml match files converted to dataframes 3. IPLMatchesBetween2Teams – All Matches between any 2 IPL teams 4. allMatchesAllOpposition – An IPL teams’s performance against all other teams 5. BattingBowlingDetails – Batting and bowling details of all IPL teams library(yorkr) library(dplyr) The first few steps take care of the data setup. This needs to be done before any of the analysis of IPL batsmen, bowlers, any IPL match, matches between any 2 IPL countries or analysis of a teams performance against all other countries There will be 5 folders at the root 1. data 2. IPLMatches 3. IPLMatchesBetween2Teams 4. allMatchesAllOpposition 5. BattingBowlingDetails # The source YAML files will be in IPLData folder # 1.Create directory of IPLMatches Some files may give conversions errors. You could try to debug the problem or just remove it from the IPLdata folder. At most 2-4 file will have conversion problems and I usally remove then from the files to be converted. Also take a look at my GooglyPlus shiny app which was created after performing the same conversion on the Dec 16 data . convertAllYaml2RDataframesT20("data","IPLMatches") ### 2.Save all matches between all combinations of IPL nations This function will create the set of all matches between each IPL team against every other IPL team. This uses the data that was created in IPLMatches, with the convertAllYaml2RDataframesIPL() function. setwd("./IPLMatchesBetween2Teams") saveAllMatchesBetween2IPLTeams("../IPLMatches") ### 3.Save all matches against all opposition This will create a consolidated dataframe of all matches played by every IPL playing nation against all other nattions. This also uses the data that was created in IPLMatches, with the convertAllYaml2RDataframesIPL() function. setwd("../allMatchesAllOpposition") saveAllMatchesAllOppositionIPLT20("../IPLMatches") ### 4. Create batting and bowling details for each IPL team These are the current IPL playing teams. You can add to this vector as newer IPL teams start playing IPL. You will get to know all IPL teams by also look at the directory created above namely allMatchesAllOpposition. This also uses the data that was created in IPLMatches, with the convertAllYaml2RDataframesIPL() function. setwd("../BattingBowlingDetails") ipl_teams <- list("Chennai Super Kings","Deccan Chargers", "Delhi Daredevils","Kings XI Punjab", "Kochi Tuskers Kerala","Kolkata Knight Riders","Mumbai Indians","Pune Warriors", "Rajasthan Royals","Royal Challengers Bangalore","Sunrisers Hyderabad","Gujarat Lions", "Rising Pune Supergiants") for(i in seq_along(ipl_teams)){ print(ipl_teams[i]) val <- paste(ipl_teams[i],"-details",sep="") val <- getTeamBattingDetails(ipl_teams[i],dir="../IPLMatches", save=TRUE) } for(i in seq_along(ipl_teams)){ print(ipl_teams[i]) val <- paste(ipl_teams[i],"-details",sep="") val <- getTeamBowlingDetails(ipl_teams[i],dir="../IPLMatches", save=TRUE) } ### 5. Get the list of batsmen for a particular IPL team The following code is needed for analyzing individual IPL batsmen. In IPL a player could have played in multiple IPL teams. getBatsmen <- function(df){ bmen <- df %>% distinct(batsman) bmen <- as.character(bmen$batsman)
batsmen <- sort(bmen)
}
csk_details <- battingDetails
dc_details <- battingDetails
dd_details <- battingDetails
kxip_details <- battingDetails
ktk_details <- battingDetails
kkr_details <- battingDetails
mi_details <- battingDetails
pw_details <- battingDetails
rr_details <- battingDetails
rcb_details <- battingDetails
sh_details <- battingDetails
gl_details <- battingDetails
rps_details <- battingDetails

#Get the batsmen for each IPL team
csk_batsmen <- getBatsmen(csk_details)
dc_batsmen <- getBatsmen(dc_details)
dd_batsmen <- getBatsmen(dd_details)
kxip_batsmen <- getBatsmen(kxip_details)
ktk_batsmen <- getBatsmen(ktk_details)
kkr_batsmen <- getBatsmen(kkr_details)
mi_batsmen <- getBatsmen(mi_details)
pw_batsmen <- getBatsmen(pw_details)
rr_batsmen <- getBatsmen(rr_details)
rcb_batsmen <- getBatsmen(rcb_details)
sh_batsmen <- getBatsmen(sh_details)
gl_batsmen <- getBatsmen(gl_details)
rps_batsmen <- getBatsmen(rps_details)

# Save the dataframes
save(csk_batsmen,file="csk.RData")
save(dc_batsmen, file="dc.RData")
save(dd_batsmen, file="dd.RData")
save(kxip_batsmen, file="kxip.RData")
save(ktk_batsmen, file="ktk.RData")
save(kkr_batsmen, file="kkr.RData")
save(mi_batsmen , file="mi.RData")
save(pw_batsmen, file="pw.RData")
save(rr_batsmen, file="rr.RData")
save(rcb_batsmen, file="rcb.RData")
save(sh_batsmen, file="sh.RData")
save(gl_batsmen, file="gl.RData")
save(rps_batsmen, file="rps.RData")

### 6. Get the list of bowlers for a particular IPL team

The method below can get the list of bowler names for any IPL team.The following code is needed for analyzing individual IPL bowlers. In IPL a player could have played in multiple IPL teams.

getBowlers <- function(df){
bwlr <- df %>% distinct(bowler)
bwlr <- as.character(bwlr\$bowler)
bowler <- sort(bwlr)
}

csk_details <- bowlingDetails
dc_details <- bowlingDetails
dd_details <- bowlingDetails
kxip_details <- bowlingDetails
ktk_details <- bowlingDetails
kkr_details <- bowlingDetails
mi_details <- bowlingDetails
pw_details <- bowlingDetails
rr_details <- bowlingDetails
rcb_details <- bowlingDetails
sh_details <- bowlingDetails
gl_details <- bowlingDetails
rps_details <- bowlingDetails

# Get the bowlers for each team
csk_bowlers <- getBowlers(csk_details)
dc_bowlers <- getBowlers(dc_details)
dd_bowlers <- getBowlers(dd_details)
kxip_bowlers <- getBowlers(kxip_details)
ktk_bowlers <- getBowlers(ktk_details)
kkr_bowlers <- getBowlers(kkr_details)
mi_bowlers <- getBowlers(mi_details)
pw_bowlers <- getBowlers(pw_details)
rr_bowlers <- getBowlers(rr_details)
rcb_bowlers <- getBowlers(rcb_details)
sh_bowlers <- getBowlers(sh_details)
gl_bowlers <- getBowlers(gl_details)
rps_bowlers <- getBowlers(rps_details)

#Save the dataframes
save(csk_bowlers,file="csk1.RData")
save(dc_bowlers, file="dc1.RData")
save(dd_bowlers, file="dd1.RData")
save(kxip_bowlers, file="kxip1.RData")
save(ktk_bowlers, file="ktk1.RData")
save(kkr_bowlers, file="kkr1.RData")
save(mi_bowlers , file="mi1.RData")
save(pw_bowlers, file="pw1.RData")
save(rr_bowlers, file="rr1.RData")
save(rcb_bowlers, file="rcb1.RData")
save(sh_bowlers, file="sh1.RData")
save(gl_bowlers, file="gl1.RData")
save(rps_bowlers, file="rps1.RData")

### 1 IPL Match Analysis

Load any match data from the ./IPLMatches folder for e.g. Chennai Super Kings-Deccan Chargers-2008-05-06.RData

setwd("./IPLMatches")
csk_dc<- overs
#The steps are
IPLTeam1_IPLTeam2 <- overs

All analysis for this match can be done now

### 2. Scorecard

teamBattingScorecardMatch(IPLTeam1_IPLTeam2,"IPLTeam1")
teamBattingScorecardMatch(IPLTeam1_IPLTeam2,"IPLTeam2")

### 3.Batting Partnerships

teamBatsmenPartnershipMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2")
teamBatsmenPartnershipMatch(IPLTeam1_IPLTeam2,"IPLTeam2","IPLTeam1")

### 4. Batsmen vs Bowler Plot

teamBatsmenVsBowlersMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2",plot=TRUE)
teamBatsmenVsBowlersMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2",plot=FALSE)

### 5. Team bowling scorecard

teamBowlingScorecardMatch(IPLTeam1_IPLTeam2,"IPLTeam1")
teamBowlingScorecardMatch(IPLTeam1_IPLTeam2,"IPLTeam2")

### 6. Team bowling Wicket kind match

teamBowlingWicketKindMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2")
m <-teamBowlingWicketKindMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2",plot=FALSE)
m

### 7. Team Bowling Wicket Runs Match

teamBowlingWicketRunsMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2")
m <-teamBowlingWicketRunsMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2",plot=FALSE)
m

### 8. Team Bowling Wicket Match

m <-teamBowlingWicketMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2",plot=FALSE)
m
teamBowlingWicketMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2")

### 9. Team Bowler vs Batsmen

teamBowlersVsBatsmenMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2")
m <- teamBowlersVsBatsmenMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2",plot=FALSE)
m

### 10. Match Worm chart

matchWormGraph(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2")

### 1 IPL Match Analysis

Load any match data from the ./IPLMatches folder for e.g. Chennai Super Kings-Deccan Chargers-2008-05-06.RData

setwd("./IPLMatches")
csk_dc<- overs
#The steps are
IPLTeam1_IPLTeam2 <- overs

All analysis for this match can be done now

### 2. Scorecard

teamBattingScorecardMatch(IPLTeam1_IPLTeam2,"IPLTeam1")
teamBattingScorecardMatch(IPLTeam1_IPLTeam2,"IPLTeam2")

### 3.Batting Partnerships

teamBatsmenPartnershipMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2")
teamBatsmenPartnershipMatch(IPLTeam1_IPLTeam2,"IPLTeam2","IPLTeam1")

### 4. Batsmen vs Bowler Plot

teamBatsmenVsBowlersMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2",plot=TRUE)
teamBatsmenVsBowlersMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2",plot=FALSE)

### 5. Team bowling scorecard

teamBowlingScorecardMatch(IPLTeam1_IPLTeam2,"IPLTeam1")
teamBowlingScorecardMatch(IPLTeam1_IPLTeam2,"IPLTeam2")

### 6. Team bowling Wicket kind match

teamBowlingWicketKindMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2")
m <-teamBowlingWicketKindMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2",plot=FALSE)
m

### 7. Team Bowling Wicket Runs Match

teamBowlingWicketRunsMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2")
m <-teamBowlingWicketRunsMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2",plot=FALSE)
m

### 8. Team Bowling Wicket Match

m <-teamBowlingWicketMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2",plot=FALSE)
m
teamBowlingWicketMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2")

### 9. Team Bowler vs Batsmen

teamBowlersVsBatsmenMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2")
m <- teamBowlersVsBatsmenMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2",plot=FALSE)
m

### 10. Match Worm chart

matchWormGraph(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2")

### 1. IPL Matches for a team against all other teams

Load the data between for a IPL team against all other countries ./allMatchesAllOpposition for e.g all matches of Kolkata Knight Riders

load("allMatchesAllOpposition-Kolkata Knight Riders.RData")
kkr_matches <- matches
IPLTeam="IPLTeam1"
allMatches <- paste("allMatchesAllOposition-",IPLTeam,".RData",sep="")
IPLTeam1AllMatches <- matches


### 2. Team’s batting scorecard all Matches

m <-teamBattingScorecardAllOppnAllMatches(IPLTeam1AllMatches,theTeam="IPLTeam1")
m

### 3. Batting scorecard of opposing team

m <-teamBattingScorecardAllOppnAllMatches(matches=IPLTeam1AllMatches,theTeam="IPLTeam2")

### 4. Team batting partnerships

m <- teamBatsmenPartnershipAllOppnAllMatches(IPLTeam1AllMatches,theTeam="IPLTeam1")
m
m <- teamBatsmenPartnershipAllOppnAllMatches(IPLTeam1AllMatches,theTeam='IPLTeam1',report="detailed")
m <- teamBatsmenPartnershipAllOppnAllMatches(IPLTeam1AllMatches,theTeam='IPLTeam1',report="summary")
m

### 5. Team batting partnerships plot

teamBatsmenPartnershipAllOppnAllMatchesPlot(IPLTeam1AllMatches,"IPLTeam1",main="IPLTeam1")
teamBatsmenPartnershipAllOppnAllMatchesPlot(IPLTeam1AllMatches,"IPLTeam1",main="IPLTeam2")

### 6, Team batsmen vs bowlers report

m <-teamBatsmenVsBowlersAllOppnAllMatchesRept(IPLTeam1AllMatches,"IPLTeam1",rank=0)
m
m <-teamBatsmenVsBowlersAllOppnAllMatchesRept(IPLTeam1AllMatches,"IPLTeam1",rank=1,dispRows=30)
m
m <-teamBatsmenVsBowlersAllOppnAllMatchesRept(matches=IPLTeam1AllMatches,theTeam="IPLTeam2",rank=1,dispRows=25)
m

### 7. Team batsmen vs bowler plot

d <- teamBatsmenVsBowlersAllOppnAllMatchesRept(IPLTeam1AllMatches,"IPLTeam1",rank=1,dispRows=50)
d
teamBatsmenVsBowlersAllOppnAllMatchesPlot(d)
d <- teamBatsmenVsBowlersAllOppnAllMatchesRept(IPLTeam1AllMatches,"IPLTeam1",rank=2,dispRows=50)
teamBatsmenVsBowlersAllOppnAllMatchesPlot(d)

### 8. Team bowling scorecard

teamBowlingScorecardAllOppnAllMatchesMain(matches=IPLTeam1AllMatches,theTeam="IPLTeam1")
teamBowlingScorecardAllOppnAllMatches(IPLTeam1AllMatches,'IPLTeam2')

### 9. Team bowler vs batsmen

teamBowlersVsBatsmenAllOppnAllMatchesMain(IPLTeam1AllMatches,theTeam="IPLTeam1",rank=0)
teamBowlersVsBatsmenAllOppnAllMatchesMain(IPLTeam1AllMatches,theTeam="IPLTeam1",rank=2)
teamBowlersVsBatsmenAllOppnAllMatchesRept(matches=IPLTeam1AllMatches,theTeam="IPLTeam1",rank=0)

### 10. Team Bowler vs bastmen

df <- teamBowlersVsBatsmenAllOppnAllMatchesRept(IPLTeam1AllMatches,theTeam="IPLTeam1",rank=1)
teamBowlersVsBatsmenAllOppnAllMatchesPlot(df,"IPLTeam1","IPLTeam1")

### 11. Team bowler wicket kind

teamBowlingWicketKindAllOppnAllMatches(IPLTeam1AllMatches,t1="IPLTeam1",t2="All")
teamBowlingWicketKindAllOppnAllMatches(IPLTeam1AllMatches,t1="IPLTeam1",t2="IPLTeam2")


### 12.

teamBowlingWicketRunsAllOppnAllMatches(IPLTeam1AllMatches,t1="IPLTeam1",t2="All",plot=TRUE)
teamBowlingWicketRunsAllOppnAllMatches(IPLTeam1AllMatches,t1="IPLTeam1",t2="IPLTeam2",plot=TRUE)

### 1 IPL Batsman setup functions

Get the batsman’s details for a batsman

setwd("../BattingBowlingDetails")
# IPL Team names
IPLTeamNames <- list("Chennai Super Kings","Deccan Chargers", "Delhi Daredevils","Kings Xi Punjab",
"Kochi Tuskers Kerala","Kolkata Knight Riders","Mumbai Indians","Pune Warriors",
"Rajasthan Royals","Royal Challengers Bangalore","Sunrisers Hyderabad","Gujarat Lions",
"Rising Pune Supergiants")

# Check and get the team indices of IPL teams in which the batsman has played
getTeamIndex <- function(batsman){
setwd("./BattingBowlingDetails")
setwd("..")
getwd()
print(ls())
teams_batsmen = list(csk_batsmen,dc_batsmen,dd_batsmen,kxip_batsmen,ktk_batsmen,kkr_batsmen,mi_batsmen,
pw_batsmen,rr_batsmen,rcb_batsmen,sh_batsmen,gl_batsmen,rps_batsmen)
b <- NULL
for (i in 1:length(teams_batsmen)){
a <- which(teams_batsmen[[i]] == batsman)

if(length(a) != 0)
b <- c(b,i)
}
b
}

# Get the list of the IPL team names from the indices passed
getTeams <- function(x){

l <- NULL
# Get the teams passed in as indexes
for (i in seq_along(x)){

l <- c(l, IPLTeamNames[[x[i]]])

}
l
}

# Create a consolidated data frame with all teams the IPL batsman has played for
getIPLBatsmanDF <- function(teamNames){
batsmanDF <- NULL
# Create a consolidated Data frame of batsman for all IPL teams played
for (i in seq_along(teamNames)){
df <- getBatsmanDetails(team=teamNames[i],name=IPLBatsman,dir="./BattingBowlingDetails")
batsmanDF <- rbind(batsmanDF,df)

}
batsmanDF
}


### 2. Create a consolidated IPL batsman data frame

# Since an IPL batsman coculd have played in multiple teams we need to determine these teams and
# create a consolidated data frame for the analysis
# For example to check MS Dhoni we need to do the following

IPLBatsman = "MS Dhoni"
#Check and get the team indices of IPL teams in which the batsman has played
i <- getTeamIndex(IPLBatsman)

# Get the team names in which the IPL batsman has played
teamNames <- getTeams(i)
# Check if file exists in the directory. This check is necessary when moving between matchType

############## Create a consolidated IPL batsman dataframe for analysis
batsmanDF <- getIPLBatsmanDF(teamNames)


### 3. Runs vs deliveries

# For e.g. batsmanName="MS Dhoni""
#batsmanRunsVsDeliveries(batsmanDF, "MS Dhoni")
batsmanRunsVsDeliveries(batsmanDF,"batsmanName")

### 4. Batsman 4s & 6s

batsman46 <- select(batsmanDF,batsman,ballsPlayed,fours,sixes,runs)
p1 <- batsmanFoursSixes(batsman46,"batsmanName")

### 5. Batsman dismissals

batsmanDismissals(batsmanDF,"batsmanName")

### 6. Runs vs Strike rate

batsmanRunsVsStrikeRate(batsmanDF,"batsmanName")

### 7. Batsman Moving Average

batsmanMovingAverage(batsmanDF,"batsmanName")

### 8. Batsman cumulative average

batsmanCumulativeAverageRuns(batsmanDF,"batsmanName")

### 9. Batsman cumulative strike rate

batsmanCumulativeStrikeRate(batsmanDF,"batsmanName")

### 10. Batsman runs against oppositions

batsmanRunsAgainstOpposition(batsmanDF,"batsmanName")

### 11. Batsman runs vs venue

batsmanRunsVenue(batsmanDF,"batsmanName")

### 12. Batsman runs predict

batsmanRunsPredict(batsmanDF,"batsmanName")

### 13.Bowler set up functions

setwd("../BattingBowlingDetails")
# IPL Team names
IPLTeamNames <- list("Chennai Super Kings","Deccan Chargers", "Delhi Daredevils","Kings Xi Punjab",
"Kochi Tuskers Kerala","Kolkata Knight Riders","Mumbai Indians","Pune Warriors",
"Rajasthan Royals","Royal Challengers Bangalore","Sunrisers Hyderabad","Gujarat Lions",
"Rising Pune Supergiants")

# Get the team indices of IPL teams for which the bowler as played
getTeamIndex_bowler <- function(bowler){
setwd("./data")
setwd("..")
teams_bowlers = list(csk_bowlers,dc_bowlers,dd_bowlers,kxip_bowlers,ktk_bowlers,kkr_bowlers,mi_bowlers,
pw_bowlers,rr_bowlers,rcb_bowlers,sh_bowlers,gl_bowlers,rps_bowlers)
b <- NULL
for (i in 1:length(teams_bowlers)){
a <- which(teams_bowlers[[i]] == bowler)
if(length(a) != 0){
b <- c(b,i)
}
}
b
}

# Get the list of the IPL team names from the indices passed
getTeams <- function(x){

l <- NULL
# Get the teams passed in as indexes
for (i in seq_along(x)){

l <- c(l, IPLTeamNames[[x[i]]])

}
l
}

# Get the team names
teamNames <- getTeams(i)

getIPLBowlerDF <- function(teamNames){
bowlerDF <- NULL

# Create a consolidated Data frame of batsman for all IPL teams played
for (i in seq_along(teamNames)){
df <- getBowlerWicketDetails(team=teamNames[i],name=IPLBowler,dir="./BattingBowlingDetails")
bowlerDF <- rbind(bowlerDF,df)

}
bowlerDF
}

### 14. Get the consolidated data frame for an IPL bowler

# Since an IPL bowler could have played in multiple teams we need to determine these teams and
# create a consolidated data frame for the analysis
# For example to check R Ashwin we need to do the following

IPLBowler = "R Ashwin"
#Check and get the team indices of IPL teams in which the batsman has played
i <- getTeamIndex(IPLBowler)

# Get the team names in which the IPL batsman has played
teamNames <- getTeams(i)
# Check if file exists in the directory. This check is necessary when moving between matchType

############## Create a consolidated IPL batsman dataframe for analysis
bowlerDF <- getIPLBowlerDF(teamNames)


### 15. Bowler Mean Economy rate

# For e.g. to get the details of R Ashwin do
#bowlerMeanEconomyRate(bowlerDF,"R Ashwin")
bowlerMeanEconomyRate(bowlerDF,"bowlerName")

### 16. Bowler mean runs conceded

bowlerMeanRunsConceded(bowlerDF,"bowlerName")

### 17. Bowler Moving Average

bowlerMovingAverage(bowlerDF,"bowlerName")

### 18. Bowler cumulative average wickets

bowlerCumulativeAvgWickets(bowlerDF,"bowlerName")

### 19. Bowler cumulative Economy Rate (ER)

bowlerCumulativeAvgEconRate(bowlerDF,"bowlerName")

### 20. Bowler wicket plot

bowlerWicketPlot(bowlerDF,"bowlerName")

### 21. Bowler wicket against opposition

bowlerWicketsAgainstOpposition(bowlerDF,"bowlerName")

### 22. Bowler wicket at cricket grounds

bowlerWicketsVenue(bowlerDF,"bowlerName")

### 23. Predict number of deliveries to wickets

setwd("./IPLMatches")
bowlerDF1 <- getDeliveryWickets(team="IPLTeam1",dir=".",name="bowlerName",save=FALSE)
bowlerWktsPredict(bowlerDF1,"bowlerName")

# Video presentation on Machine Learning, Data Science, NLP and Big Data – Part 4

In this presentation I demonstrate some Shiny R apps that I have developed. The demo also includes googleMotion Charts using googleVis

# Video presentation on Machine Learning, Data Science, NLP and Big Data – Part 3

This presentation includes the 3 part and touches upon Apache Edgent, NLP and Big Data with Spark.