# Benford’s law meets IPL, Intl. T20 and ODI cricket

“To grasp how different a million is from a billion, think about it like this: A million seconds is a little under two weeks; a billion seconds is about thirty-two years.”

“One of the pleasures of looking at the world through mathematical eyes is that you can see certain patterns that would otherwise be hidden.”

``               Steven Strogatz, Prof at Cornell University``

## Introduction

Within the last two weeks, I was introduced to Benford’s Law by 2 of my friends. Initially, I looked it up and Google and was quite intrigued by the law. Subsequently another friends asked me to check the ‘Digits’ episode, from the “Connected” series on Netflix by Latif Nasser, which I strongly recommend you watch.

Benford’s Law also called the Newcomb–Benford law, the law of anomalous numbers, or the First Digit Law states that, when dealing with quantities obtained from Nature, the frequency of appearance of each digit in the first significant place is logarithmic. For example, in sets that obey the law, the number 1 appears as the leading significant digit about 30.1% of the time, the number 2 about 17.6%, number 3 about 12.5% all the way to the number 9 at 4.6%. This interesting logarithmic pattern is observed in most natural datasets from population densities, river lengths, heights of skyscrapers, tax returns etc. What is really curious about this law, is that when we measure the lengths of rivers, the law holds perfectly regardless of the units used to measure. So the length of the rivers would obey the law whether we measure in meters, feet, miles etc. There is something almost mystical about this law.

The law has also been used widely to detect financial fraud, manipulations in tax statements, bots in twitter, fake accounts in social networks, image manipulation etc. In this age of deep fakes, the ability to detect fake images will assume paramount importance. While deviations from Benford Law do not always signify fraud, to large extent they point to an aberration. Prof Nigrini, of Cape Town used this law to identify financial discrepancies in Enron’s financial statement resulting in the infamous scandal. Also the 2009 Iranian election was found to be fradulent as the first digit percentages did not conform to those specified by Benford’s Law.

While it cannot be said with absolute certainty, marked deviations from Benford’s law could possibly indicate that there has been manipulation of natural processes. Possibly Benford’s law could be used to detect large scale match-fixing in cricket tournaments. However, we cannot look at this in isolation and the other statistical and forensic methods may be required to determine if there is fraud. Here is an interesting paper Promises and perils of Benford’s law

A set of numbers is said to satisfy Benford’s law if the leading digit d (d ∈ {1, …, 9}) occurs with probability

$P(d)=log_{10}(1+1/d)$

This law also works for number in other bases, in base b >=2

$P(d)=log_{b}(1+1/d)$

Interestingly, this law also applies to sports on the number of point scored in basketball etc. I was curious to see if this applied to cricket. Previously, using my R package yorkr, I had already converted all T20 data and ODI data from Cricsheet which is available at yorkrData2020, I wanted to check if Benford’s Law worked on the runs scored, or deliveries faced by batsmen at team level or at a tournament level (IPL, Intl. T20 or ODI).

Thankfully, R has a package benford.analysis to check for data behaviour in accordance to Benford’s Law, and I have used this package in my post

This post is also available in RPubs as Benford’s Law meets IPL, Intl. T20 and ODI

``````library(data.table)
library(reshape2)``````
``library(dplyr)``
``````library(benford.analysis)
library(yorkr)``````

In this post, I have randomly check data with Benford’s law. The fully converted dataset is available in yorkrData2020 which I have included above. You can try on any dataset including ODI (men,women),Intl T20(men,women),IPL,BBL,PSL,NTB and WBB.

## 1. Check the runs distribution by Royal Challengers Bangalore

We can see the behaviour is as expected with Benford’s law, with minor deviations

``````load("/Users/tvganesh/backup/software/cricket-package/yorkr-cricsheet/yorkrData2020/ipl/iplBattingBowlingDetails/Royal Challengers Bangalore-BattingDetails.RData")
rcbRunsTrends = benford(battingDetails\$runs, number.of.digits = 1, discrete = T, sign = "positive")
rcbRunsTrends``````
``````##
## Benford object:
##
## Data: battingDetails\$runs
## Number of observations used = 1205
## Number of obs. for second order = 99
## First digits analysed = 1
##
## Mantissa:
##
##    Statistic  Value
##         Mean  0.458
##          Var  0.091
##  Ex.Kurtosis -1.213
##     Skewness -0.025
##
##
## The 5 largest deviations:
##
##   digits absolute.diff
## 1      1         14.26
## 2      7         13.88
## 3      9          8.14
## 4      6          5.33
## 5      4          4.78
##
## Stats:
##
##  Pearson's Chi-squared test
##
## data:  battingDetails\$runs
## X-squared = 5.2091, df = 8, p-value = 0.735
##
##
##  Mantissa Arc Test
##
## data:  battingDetails\$runs
## L2 = 0.0022852, df = 2, p-value = 0.06369
##
## Mean Absolute Deviation (MAD): 0.004941381
## MAD Conformity - Nigrini (2012): Close conformity
## Distortion Factor: -18.8725
##
## Remember: Real data will never conform perfectly to Benford's Law. You should not focus on p-values!``````

## 2. Check the ‘balls played’ distribution by Royal Challengers Bangalore

``````load("/Users/tvganesh/backup/software/cricket-package/yorkr-cricsheet/yorkrData2020/ipl/iplBattingBowlingDetails/Royal Challengers Bangalore-BattingDetails.RData")
rcbBallsPlayedTrends = benford(battingDetails\$ballsPlayed, number.of.digits = 1, discrete = T, sign = "positive")
plot(rcbBallsPlayedTrends)``````

## 3. Check the runs distribution by Chennai Super Kings

The trend seems to deviate from the expected behavior to some extent in the number of digits for 5 & 7.

``````load("/Users/tvganesh/backup/software/cricket-package/yorkr-cricsheet/yorkrData2020/ipl/iplBattingBowlingDetails/Chennai Super Kings-BattingDetails.RData")
cskRunsTrends = benford(battingDetails\$runs, number.of.digits = 1, discrete = T, sign = "positive")
cskRunsTrends``````
``````##
## Benford object:
##
## Data: battingDetails\$runs
## Number of observations used = 1054
## Number of obs. for second order = 94
## First digits analysed = 1
##
## Mantissa:
##
##    Statistic  Value
##         Mean  0.466
##          Var  0.081
##  Ex.Kurtosis -1.100
##     Skewness -0.054
##
##
## The 5 largest deviations:
##
##   digits absolute.diff
## 1      5         27.54
## 2      2         18.40
## 3      1         17.29
## 4      9         14.23
## 5      7         14.12
##
## Stats:
##
##  Pearson's Chi-squared test
##
## data:  battingDetails\$runs
## X-squared = 22.862, df = 8, p-value = 0.003545
##
##
##  Mantissa Arc Test
##
## data:  battingDetails\$runs
## L2 = 0.002376, df = 2, p-value = 0.08173
##
## Mean Absolute Deviation (MAD): 0.01309597
## MAD Conformity - Nigrini (2012): Marginally acceptable conformity
## Distortion Factor: -17.90664
##
## Remember: Real data will never conform perfectly to Benford's Law. You should not focus on p-values!``````

## 4. Check runs distribution in all of Indian Premier League (IPL)

``````battingDF <- NULL
teams <-c("Chennai Super Kings","Deccan Chargers","Delhi Daredevils",
"Kings XI Punjab", 'Kochi Tuskers Kerala',"Kolkata Knight Riders",
"Mumbai Indians", "Pune Warriors","Rajasthan Royals",
"Rising Pune Supergiants")

setwd("/Users/tvganesh/backup/software/cricket-package/yorkr-cricsheet/yorkrData2020/ipl/iplBattingBowlingDetails")
for(team in teams){
battingDetails <- NULL
val <- paste(team,"-BattingDetails.RData",sep="")
print(val)
error = function(e) {
print("No data1")
setNext=TRUE
}

)
details <- battingDetails
battingDF <- rbind(battingDF,details)
}``````
``````## [1] "Chennai Super Kings-BattingDetails.RData"
## [1] "Deccan Chargers-BattingDetails.RData"
## [1] "Delhi Daredevils-BattingDetails.RData"
## [1] "Kings XI Punjab-BattingDetails.RData"
## [1] "Kochi Tuskers Kerala-BattingDetails.RData"
## [1] "Kolkata Knight Riders-BattingDetails.RData"
## [1] "Mumbai Indians-BattingDetails.RData"
## [1] "Pune Warriors-BattingDetails.RData"
## [1] "Rajasthan Royals-BattingDetails.RData"
## [1] "Royal Challengers Bangalore-BattingDetails.RData"
## [1] "Gujarat Lions-BattingDetails.RData"
## [1] "Rising Pune Supergiants-BattingDetails.RData"``````
``````trends = benford(battingDF\$runs, number.of.digits = 1, discrete = T, sign = "positive")
trends``````
``````##
## Benford object:
##
## Data: battingDF\$runs
## Number of observations used = 10129
## Number of obs. for second order = 123
## First digits analysed = 1
##
## Mantissa:
##
##    Statistic   Value
##         Mean  0.4521
##          Var  0.0856
##  Ex.Kurtosis -1.1570
##     Skewness -0.0033
##
##
## The 5 largest deviations:
##
##   digits absolute.diff
## 1      2        159.37
## 2      9        121.48
## 3      7         93.40
## 4      8         83.12
## 5      1         61.87
##
## Stats:
##
##  Pearson's Chi-squared test
##
## data:  battingDF\$runs
## X-squared = 78.166, df = 8, p-value = 1.143e-13
##
##
##  Mantissa Arc Test
##
## data:  battingDF\$runs
## L2 = 5.8237e-05, df = 2, p-value = 0.5544
##
## Mean Absolute Deviation (MAD): 0.006627966
## MAD Conformity - Nigrini (2012): Acceptable conformity
## Distortion Factor: -20.90333
##
## Remember: Real data will never conform perfectly to Benford's Law. You should not focus on p-values!``````

## 5. Check Benford’s law in India matches

``````setwd("/Users/tvganesh/backup/software/cricket-package/yorkr-cricsheet/yorkrData2020/t20/t20BattingBowlingDetails")

indiaTrends = benford(battingDetails\$runs, number.of.digits = 1, discrete = T, sign = "positive")
plot(indiaTrends)``````

## 6. Check Benford’s law in all of Intl. T20

``````setwd("/Users/tvganesh/backup/software/cricket-package/yorkr-cricsheet/yorkrData2020/t20/t20BattingBowlingDetails")
teams <-c("Australia","India","Pakistan","West Indies", 'Sri Lanka',
"Bermuda","Kenya","Hong Kong","Nepal","Oman","Papua New Guinea",
"United Arab Emirates","Namibia","Cayman Islands","Singapore",
"United States of America","Bhutan","Maldives","Botswana","Nigeria",
"Denmark","Germany","Jersey","Norway","Qatar","Malaysia","Vanuatu",
"Thailand")

for(team in teams){
battingDetails <- NULL
val <- paste(team,"-BattingDetails.RData",sep="")
print(val)
error = function(e) {
print("No data1")
setNext=TRUE
}

)
details <- battingDetails
battingDF <- rbind(battingDF,details)

}
``````intlT20Trends = benford(battingDF\$runs, number.of.digits = 1, discrete = T, sign = "positive")
intlT20Trends``````
``````##
## Benford object:
##
## Data: battingDF\$runs
## Number of observations used = 21833
## Number of obs. for second order = 131
## First digits analysed = 1
##
## Mantissa:
##
##    Statistic  Value
##         Mean  0.447
##          Var  0.085
##  Ex.Kurtosis -1.158
##     Skewness  0.018
##
##
## The 5 largest deviations:
##
##   digits absolute.diff
## 1      2        361.40
## 2      9        276.02
## 3      1        264.61
## 4      7        210.14
## 5      8        198.81
##
## Stats:
##
##  Pearson's Chi-squared test
##
## data:  battingDF\$runs
## X-squared = 202.29, df = 8, p-value < 2.2e-16
##
##
##  Mantissa Arc Test
##
## data:  battingDF\$runs
## L2 = 5.3983e-06, df = 2, p-value = 0.8888
##
## Mean Absolute Deviation (MAD): 0.007821098
## MAD Conformity - Nigrini (2012): Acceptable conformity
## Distortion Factor: -24.11086
##
## Remember: Real data will never conform perfectly to Benford's Law. You should not focus on p-values!``````

## Conclusion

Maths rules our lives, more than we are aware, more that we like to admit. It is there in all of nature. Whether it is the recursive patterns of Mandelbrot sets, the intrinsic notion of beauty through the golden ratio, the murmuration of swallows, the synchronous blinking of fireflies or in the almost univerality of Benford’s law on natural datasets, mathematics govern us.

Isn’t it strange that while we humans pride ourselves of freewill, the runs scored by batsmen in particular formats conform to Benford’s rule for the first digits. It almost looks like, the runs that will be scored is almost to extent predetermined to fall within specified ranges obeying Benford’s law. So much for choice.

Something to be pondered over!

Also see

# Introducing GooglyPlusPlus!!!

“We can lift ourselves out of ignorance, we can find ourselves as creatures of excellence and intelligence and skill.”
“Heaven is not a place, and it is not a time. Heaven is being perfect.”
“Your whole body, from wingtip to wingtip, is nothing more than your thought itself, in a form you can see. Break the chains of your thought, and you break the chains of your body, too.”

From Jonathan Livingstone Seagull, by Richard Bach

## Introduction

The metamorphosis is complete, from eggs to the butterfly! My R package yorkr, went on to become Googly,  and then to GooglyPlus and  now finally GooglyPlusPlus. My latest R Shiny app now provides interactive visualisation of almost all data in Cricsheet. GooglyPlusPlus visualizes the following matches

1. ODI (men)
2. ODI (women)
3. Intl. T20 (men)
4. Intl T20 (women)
5. IPL (Indian Premier League)
6. BBL (Big Bash League)
7. NTB (Natwest T20)
8. PSL (Pakistan Super League)
9. WBBL – Women’s BBL

GooglyPlusPlus is entirely based on my R package yorkr. To know more about yorkr see ‘Revitalizing R package yorkr‘ and the roughly 25+ posts on yorkr in Index of posts

This Shiny app was quite involved, and it took a lot of work to keep things organised and separate for the different forms of cricket. Anyway it is done and I am happy with the outcome.

Before you use the app, I would suggest that you take a look at the video “How to use GooglyPlusPlus?“. In this video, I show the different features of GooglyPlusPlus and how to navigate through them.

Check out GooglyPlusPlus Shiny at GooglyPlusPlus

You can clone/fork and play around with the code of GooglyPlusPlus here at Github

## A. Highlights of GooglyPlusPlus.

The R Shiny app GooglyPlusPlus has the following main pages for the 9 different cricket formats. See below

Important note: Below I will be including some random output from the GooglyPlusPlus app for different match formats, however there is a lot more features in GooglyPlusPlus

## Conclusion

There you have it. I have randomly shown  2 functions for each cricket format. There are many functions in each tab for the for the different match formats – namely IPL, BBL, Intl T20 (men,women), PSL etc.  Go ahead and give GooglyPlusPlus a spin!

To try out GooglyPlusPlus click GooglyPlusPlus. Don’t forget to check out the video How to use GooglyPlusPlus?

You can clone/fork the code from Github at GooglyPlusPlus

Hope you have fun with GooglyPlusPlus!!

You may also like

To see all posts click Index of posts

# Big Data 7: yorkr waltzes with Apache NiFi

In this post, I construct an end-to-end Apache NiFi pipeline with my R package yorkr. This post is a mirror of my earlier post Big Data-5: kNiFing through cricket data with yorkpy based on my Python package yorkpy. The  Apache NiFi Data Pipeilne  flows all the way from the source, where the data is obtained, all the way  to target analytics output. Apache NiFi was created to automate the flow of data between systems.  NiFi dataflows enable the automated and managed flow of information between systems. This post automates the flow of data from Cricsheet, from where the zip file it is downloaded, unpacked, processed, transformed and finally T20 players are ranked.

This post uses the functions of my R package yorkr to rank IPL players. This is a example flow, of a typical Big Data pipeline where the data is ingested from many diverse source systems, transformed and then finally insights are generated. While I execute this NiFi example with my R package yorkr, in a typical Big Data pipeline where the data is huge, of the order of 100s of GB, we would be using the Hadoop ecosystem with Hive, HDFS Spark and so on. Since the data is taken from Cricsheet, which are few Megabytes, this approach would suffice. However if we hypothetically assume that there are several batches of cricket data that are being uploaded to the source, of different cricket matches happening all over the world, and the historical data exceeds several GBs, then we could use a similar Apache NiFi pattern to process the data and generate insights. If the data is was large and distributed across the Hadoop cluster , then we would need to use SparkR or SparklyR to process the data.

This is shown below pictorially

While this post displays the ranks of IPL batsmen, it is possible to create a cool dashboard using UI/UX technologies like AngularJS/ReactJS.  Take a look at my post Big Data 6: The T20 Dance of Apache NiFi and yorkpy where I create a simple dashboard of multiple analytics

My R package yorkr can handle both men’s and women’s ODI, and all formats of T20 in Cricsheet namely Intl. T20 (men’s, women’s), IPL, BBL, Natwest T20, PSL, Women’s BBL etc. To know more details about yorkr see Revitalizing R package yorkr

The code can be forked from Github at yorkrWithApacheNiFi

You can take a look at the live demo of the NiFi pipeline at yorkr waltzes with Apache NiFi

Basic Flow

## 1. Overall flow

The overall NiFi flow contains 2 Process Groups a) DownloadAnd Unpack. b) Convert and Rank IPL batsmen. While it appears that the Process Groups are disconnected, they are not. The first process group downloads the T20 zip file, unpacks the. zip file and saves the YAML files in a specific folder. The second process group monitors this folder and starts processing as soon the YAML files are available. It processes the YAML converting it into dataframes before storing it as CSV file. The next  processor then does the actual ranking of the batsmen before writing the output into IPLrank.txt

This process group is shown below

#### 1.1.1 GetT20Data

The \${T20data} variable points to the specific T20 format that needs to be downloaded. I have set this to https://cricsheet.org/downloads/ipl.zip. This could be set any other data set. In fact we could have parallel data flows for different T20/ Sports data sets and generate

#### 1.1.2 SaveUnpackedData

This processor stores the YAML files in a predetermined folder, so that the data can be picked up  by the 2nd Process Group for processing

### 1.2 ProcessAndRankT20Players Process Group

This is the second process group which converts the YAML files to pandas dataframes before storing them as. CSV files. The RankIPLPlayers will then read all the CSV files, stack them and then proceed to rank the IPL players. The Process Group is shown below

#### 1.2.1 ListFile and FetchFile Processors

The left 2 Processors ListFile and FetchFile get all the YAML files from the folder and pass it to the next processor

#### 1.2.2 convertYaml2DataFrame Processor

The convertYaml2DataFrame Processor uses the ExecuteStreamCommand which call Rscript. The Rscript invoked the yorkr function convertYaml2DataframeT20() as shown below

I also use a 16 concurrent tasks to convert 16 different flowfiles at once

```library(yorkr)
args<-commandArgs(TRUE)
convertYaml2RDataframeT20(args[1], args[2], args[3])
```

#### 1.2.3 MergeContent Processor

This processor’s only job is to trigger the rankIPLPlayers when all the FlowFiles have merged into 1 file.

#### 1.2.4 RankT20Players

This processor is an ExecuteStreamCommand Processor that executes a Rscript which invokes a yorrkr function rankIPLT20Batsmen()

```library(yorkr)
args<-commandArgs(TRUE)

rankIPLBatsmen(args[1],args[2],args[3])
```

#### 1.2.5 OutputRankofT20Player Processor

This processor writes the generated rank to an output file.

### 1.3 Final Ranking of IPL T20 players

The Nodejs based web server picks up this file and displays on the web page the final ranks (the code is based on a good youtube for reading from file)

```[1] "Chennai Super Kings"
[1] "Deccan Chargers"
[1] "Delhi Daredevils"
[1] "Kings XI Punjab"
[1] "Kochi Tuskers Kerala"
[1] "Kolkata Knight Riders"
[1] "Mumbai Indians"
[1] "Pune Warriors"
[1] "Rajasthan Royals"
[1] "Royal Challengers Bangalore"
[1] "Gujarat Lions"
[1] "Rising Pune Supergiants"
[1] "Chennai Super Kings-BattingDetails.RData"
[1] "Deccan Chargers-BattingDetails.RData"
[1] "Delhi Daredevils-BattingDetails.RData"
[1] "Kings XI Punjab-BattingDetails.RData"
[1] "Kochi Tuskers Kerala-BattingDetails.RData"
[1] "Kolkata Knight Riders-BattingDetails.RData"
[1] "Mumbai Indians-BattingDetails.RData"
[1] "Pune Warriors-BattingDetails.RData"
[1] "Rajasthan Royals-BattingDetails.RData"
[1] "Royal Challengers Bangalore-BattingDetails.RData"
[1] "Gujarat Lions-BattingDetails.RData"
[1] "Rising Pune Supergiants-BattingDetails.RData"
# A tibble: 429 x 4
batsman     matches meanRuns meanSR
<chr>         <int>    <dbl>  <dbl>
1 DA Warner       130     37.9   128.
2 LMP Simmons      29     37.2   106.
3 CH Gayle        125     36.2   134.
4 HM Amla          16     36.1   108.
5 ML Hayden        30     35.9   129.
6 SE Marsh         67     35.9   120.
7 RR Pant          39     35.3   135.
8 MEK Hussey       59     33.8   105.
9 KL Rahul         59     33.5   128.
10 MN van Wyk        5     33.4   112.
# â€¦ with 419 more rows```

## Conclusion

This post demonstrated an end-to-end pipeline with Apache NiFi and R package yorkr. You can this pipeline and generated different analytics using the various functions of yorkr and display them on a dashboard.

Hope you enjoyed with post!

To see posts click Index of posts

# Getting started with Tensorflow, Keras in Python and R

The Pale Blue Dot

“From this distant vantage point, the Earth might not seem of any particular interest. But for us, it’s different. Consider again that dot. That’s here, that’s home, that’s us. On it everyone you love, everyone you know, everyone you ever heard of, every human being who ever was, lived out their lives. The aggregate of our joy and suffering, thousands of confident religions, ideologies, and economic doctrines, every hunter and forager, every hero and coward, every creator and destroyer of civilization, every king and peasant, every young couple in love, every mother and father, hopeful child, inventor and explorer, every teacher of morals, every corrupt politician, every “superstar,” every “supreme leader,” every saint and sinner in the history of our species lived there—on the mote of dust suspended in a sunbeam.”

Carl Sagan

Tensorflow and Keras are Deep Learning frameworks that really simplify a lot of things to the user. If you are familiar with Machine Learning and Deep Learning concepts then Tensorflow and Keras are really a playground to realize your ideas.  In this post I show how you can get started with Tensorflow in both Python and R

### Tensorflow in Python

For tensorflow in Python, I found Google’s Colab an ideal environment for running your Deep Learning code. This is an Google’s research project  where you can execute your code  on GPUs, TPUs etc

### Tensorflow in R (RStudio)

To execute tensorflow in R (RStudio) you need to install tensorflow and keras as shown below
In this post I show how to get started with Tensorflow and Keras in R.

``````# Install Tensorflow in RStudio
#install_tensorflow()
# Install Keras
#install_packages("keras")
library(tensorflow)
libary(keras)``````

This post takes 3 different Machine Learning problems and uses the
Tensorflow/Keras framework to solve it

Note:
You can view the Google Colab notebook at Tensorflow in Python
The RMarkdown file has been published at RPubs and can be accessed
at Getting started with Tensorflow in R

Checkout my book ‘Deep Learning from first principles: Second Edition – In vectorized Python, R and Octave’. My book starts with the implementation of a simple 2-layer Neural Network and works its way to a generic L-Layer Deep Learning Network, with all the bells and whistles. The derivations have been discussed in detail. The code has been extensively commented and included in its entirety in the Appendix sections. My book is available on Amazon as paperback (\$14.99) and in kindle version(\$9.99/Rs449).

## 1. Multivariate regression with Tensorflow – Python

This code performs multivariate regression using Tensorflow and keras on the advent of Parkinson disease through sound recordings see Parkinson Speech Dataset with Multiple Types of Sound Recordings Data Set . The clinician’s motorUPDRS score has to be predicted from the set of features

In [0]:
```# Import tensorflow
import tensorflow as tf
from tensorflow import keras
```
In [2]:
```#Get the data rom the UCI Machine Learning repository
dataset = keras.utils.get_file("parkinsons_updrs.data", "https://archive.ics.uci.edu/ml/machine-learning-databases/parkinsons/telemonitoring/parkinsons_updrs.data")
```
```Downloading data from https://archive.ics.uci.edu/ml/machine-learning-databases/parkinsons/telemonitoring/parkinsons_updrs.data
917504/911261 [==============================] - 0s 0us/step
```
In [3]:
```# Read the CSV file
import pandas as pd
parkinsons = pd.read_csv(dataset, na_values = "?", comment='\t',
sep=",", skipinitialspace=True)
print(parkinsons.shape)
print(parkinsons.columns)
#Check if there are any NAs in the rows
parkinsons.isna().sum()
```
```(5875, 22)
Index(['subject#', 'age', 'sex', 'test_time', 'motor_UPDRS', 'total_UPDRS',
'Jitter(%)', 'Jitter(Abs)', 'Jitter:RAP', 'Jitter:PPQ5', 'Jitter:DDP',
'Shimmer', 'Shimmer(dB)', 'Shimmer:APQ3', 'Shimmer:APQ5',
'Shimmer:APQ11', 'Shimmer:DDA', 'NHR', 'HNR', 'RPDE', 'DFA', 'PPE'],
dtype='object')
```
Out[3]:
```subject#         0
age              0
sex              0
test_time        0
motor_UPDRS      0
total_UPDRS      0
Jitter(%)        0
Jitter(Abs)      0
Jitter:RAP       0
Jitter:PPQ5      0
Jitter:DDP       0
Shimmer          0
Shimmer(dB)      0
Shimmer:APQ3     0
Shimmer:APQ5     0
Shimmer:APQ11    0
Shimmer:DDA      0
NHR              0
HNR              0
RPDE             0
DFA              0
PPE              0
dtype: int64```
Note: To see how to create dummy variables see my post Practical Machine Learning with R and Python – Part 2
In [4]:
```# Drop the columns subject number as it is not relevant
parkinsons1=parkinsons.drop(['subject#'],axis=1)

# Create dummy variables for sex (M/F)
parkinsons2=pd.get_dummies(parkinsons1,columns=['sex'])

Out[4]
age test_time motor_UPDRS total_UPDRS Jitter(%) Jitter(Abs) Jitter:RAP Jitter:PPQ5 Jitter:DDP Shimmer Shimmer(dB) Shimmer:APQ3 Shimmer:APQ5 Shimmer:APQ11 Shimmer:DDA NHR HNR RPDE DFA PPE sex_0 sex_1
0 72 5.6431 28.199 34.398 0.00662 0.000034 0.00401 0.00317 0.01204 0.02565 0.230 0.01438 0.01309 0.01662 0.04314 0.014290 21.640 0.41888 0.54842 0.16006 1 0
1 72 12.6660 28.447 34.894 0.00300 0.000017 0.00132 0.00150 0.00395 0.02024 0.179 0.00994 0.01072 0.01689 0.02982 0.011112 27.183 0.43493 0.56477 0.10810 1 0
2 72 19.6810 28.695 35.389 0.00481 0.000025 0.00205 0.00208 0.00616 0.01675 0.181 0.00734 0.00844 0.01458 0.02202 0.020220 23.047 0.46222 0.54405 0.21014 1 0
3 72 25.6470 28.905 35.810 0.00528 0.000027 0.00191 0.00264 0.00573 0.02309 0.327 0.01106 0.01265 0.01963 0.03317 0.027837 24.445 0.48730 0.57794 0.33277 1 0
4 72 33.6420 29.187 36.375 0.00335 0.000020 0.00093 0.00130 0.00278 0.01703 0.176 0.00679 0.00929 0.01819 0.02036 0.011625 26.126 0.47188 0.56122 0.19361 1 0
```

```# Create a training and test data set with 80%/20%
train_dataset = parkinsons2.sample(frac=0.8,random_state=0)
test_dataset = parkinsons2.drop(train_dataset.index)

# Select columns
train_dataset1= train_dataset[['age', 'test_time', 'Jitter(%)', 'Jitter(Abs)',
'Jitter:RAP', 'Jitter:PPQ5', 'Jitter:DDP', 'Shimmer', 'Shimmer(dB)',
'Shimmer:APQ3', 'Shimmer:APQ5', 'Shimmer:APQ11', 'Shimmer:DDA', 'NHR',
'HNR', 'RPDE', 'DFA', 'PPE', 'sex_0', 'sex_1']]
test_dataset1= test_dataset[['age','test_time', 'Jitter(%)', 'Jitter(Abs)',
'Jitter:RAP', 'Jitter:PPQ5', 'Jitter:DDP', 'Shimmer', 'Shimmer(dB)',
'Shimmer:APQ3', 'Shimmer:APQ5', 'Shimmer:APQ11', 'Shimmer:DDA', 'NHR',
'HNR', 'RPDE', 'DFA', 'PPE', 'sex_0', 'sex_1']]
```
In [7]:
```# Generate the statistics of the columns for use in normalization of the data
train_stats = train_dataset1.describe()
train_stats = train_stats.transpose()
train_stats
```
Out[7]:
count mean std min 25% 50% 75% max
age 4700.0 64.792766 8.870401 36.000000 58.000000 65.000000 72.000000 85.000000
test_time 4700.0 93.399490 53.630411 -4.262500 46.852250 93.405000 139.367500 215.490000
Jitter(%) 4700.0 0.006136 0.005612 0.000830 0.003560 0.004900 0.006770 0.099990
Jitter(Abs) 4700.0 0.000044 0.000036 0.000002 0.000022 0.000034 0.000053 0.000396
Jitter:RAP 4700.0 0.002969 0.003089 0.000330 0.001570 0.002235 0.003260 0.057540
Jitter:PPQ5 4700.0 0.003271 0.003760 0.000430 0.001810 0.002480 0.003460 0.069560
Jitter:DDP 4700.0 0.008908 0.009267 0.000980 0.004710 0.006705 0.009790 0.172630
Shimmer 4700.0 0.033992 0.025922 0.003060 0.019020 0.027385 0.039810 0.268630
Shimmer(dB) 4700.0 0.310487 0.231016 0.026000 0.175000 0.251000 0.363250 2.107000
Shimmer:APQ3 4700.0 0.017125 0.013275 0.001610 0.009190 0.013615 0.020562 0.162670
Shimmer:APQ5 4700.0 0.020151 0.016848 0.001940 0.010750 0.015785 0.023733 0.167020
Shimmer:APQ11 4700.0 0.027508 0.020270 0.002490 0.015630 0.022685 0.032713 0.275460
Shimmer:DDA 4700.0 0.051375 0.039826 0.004840 0.027567 0.040845 0.061683 0.488020
NHR 4700.0 0.032116 0.060206 0.000304 0.010827 0.018403 0.031452 0.748260
HNR 4700.0 21.704631 4.288853 1.659000 19.447750 21.973000 24.445250 37.187000
RPDE 4700.0 0.542549 0.100212 0.151020 0.471235 0.543490 0.614335 0.966080
DFA 4700.0 0.653015 0.070446 0.514040 0.596470 0.643285 0.710618 0.865600
PPE 4700.0 0.219559 0.091506 0.021983 0.156470 0.205340 0.264017 0.731730
sex_0 4700.0 0.681489 0.465948 0.000000 0.000000 1.000000 1.000000 1.000000
sex_1 4700.0 0.318511 0.465948 0.000000 0.000000 0.000000 1.000000 1.000000
In [0]:
```# Create the target variable
train_labels = train_dataset.pop('motor_UPDRS')
test_labels = test_dataset.pop('motor_UPDRS')
```
In [0]:
```# Normalize the data by subtracting the mean and dividing by the standard deviation
def normalize(x):
return (x - train_stats['mean']) / train_stats['std']

# Create normalized training and test data
normalized_train_data = normalize(train_dataset1)
normalized_test_data = normalize(test_dataset1)
```
In [0]:
```# Create a Deep Learning model with keras
model = tf.keras.Sequential([
keras.layers.Dense(6, activation=tf.nn.relu, input_shape=[len(train_dataset1.keys())]),
keras.layers.Dense(9, activation=tf.nn.relu),
keras.layers.Dense(6,activation=tf.nn.relu),
keras.layers.Dense(1)
])

# Use the Adam optimizer with a learning rate of 0.01

# Set the metrics required to be Mean Absolute Error and Mean Squared Error.For regression, the loss is mean_squared_error
model.compile(loss='mean_squared_error',
optimizer=optimizer,
metrics=['mean_absolute_error', 'mean_squared_error'])
```
In [0]:
```# Create a model
history=model.fit(
normalized_train_data, train_labels,
epochs=1000, validation_data = (normalized_test_data,test_labels), verbose=0)
```
In [26]:
```hist = pd.DataFrame(history.history)
hist['epoch'] = history.epoch
hist.tail()
```
Out[26]:
loss mean_absolute_error mean_squared_error val_loss val_mean_absolute_error val_mean_squared_error epoch
995 15.773989 2.936990 15.773988 16.980803 3.028168 16.980803 995
996 15.238623 2.873420 15.238622 17.458752 3.101033 17.458752 996
997 15.437594 2.895500 15.437593 16.926016 2.971508 16.926018 997
998 15.867891 2.943521 15.867892 16.950249 2.985036 16.950249 998
999 15.846878 2.938914 15.846880 17.095623 3.014504 17.095625 999
In [30]:
```def plot_history(history):
hist = pd.DataFrame(history.history)
hist['epoch'] = history.epoch

plt.figure()
plt.xlabel('Epoch')
plt.ylabel('Mean Abs Error')
plt.plot(hist['epoch'], hist['mean_absolute_error'],
label='Train Error')
plt.plot(hist['epoch'], hist['val_mean_absolute_error'],
label = 'Val Error')
plt.ylim([2,5])
plt.legend()

plt.figure()
plt.xlabel('Epoch')
plt.ylabel('Mean Square Error ')
plt.plot(hist['epoch'], hist['mean_squared_error'],
label='Train Error')
plt.plot(hist['epoch'], hist['val_mean_squared_error'],
label = 'Val Error')
plt.ylim([10,40])
plt.legend()
plt.show()

plot_history(history)
```

### Observation

It can be seen that the mean absolute error is on an average about +/- 4.0. The validation error also is about the same. This can be reduced by playing around with the hyperparamaters and increasing the number of iterations

### 1a. Multivariate Regression in Tensorflow – R

``````# Install Tensorflow in RStudio
#install_tensorflow()
# Install Keras
#install_packages("keras")
library(tensorflow)``````
``````library(keras)
``````
``library(dplyr)``
``library(dummies)``
``## dummies-1.5.6 provided by Decision Patterns``
``````library(tensorflow)
library(keras)``````

## Multivariate regression

This code performs multivariate regression using Tensorflow and keras on the advent of Parkinson disease through sound recordings see Parkinson Speech Dataset with Multiple Types of Sound Recordings Data Set. The clinician’s motorUPDRS score has to be predicted from the set of features.

``````# Download the Parkinson's data from UCI Machine Learning repository

# Set the column names
names(dataset) <- c("subject","age", "sex", "test_time","motor_UPDRS","total_UPDRS","Jitter","Jitter.Abs",
"Jitter.RAP","Jitter.PPQ5","Jitter.DDP","Shimmer", "Shimmer.dB", "Shimmer.APQ3",
"Shimmer.APQ5","Shimmer.APQ11","Shimmer.DDA", "NHR","HNR", "RPDE", "DFA","PPE")

# Remove the column 'subject' as it is not relevant to analysis
dataset1 <- subset(dataset, select = -c(subject))

# Make the column 'sex' as a factor for using dummies
dataset1\$sex=as.factor(dataset1\$sex)
# Add dummy variables for categorical cariable 'sex'
dataset2 <- dummy.data.frame(dataset1, sep = ".")``````
``````## Warning in model.matrix.default(~x - 1, model.frame(~x - 1), contrasts =
## FALSE): non-list contrasts argument ignored``````
``dataset3 <- na.omit(dataset2)``

### Split the data as training and test in 80/20

``````## Split data 80% training and 20% test
sample_size <- floor(0.8 * nrow(dataset3))

## set the seed to make your partition reproducible
set.seed(12)
train_index <- sample(seq_len(nrow(dataset3)), size = sample_size)

train_dataset <- dataset3[train_index, ]
test_dataset <- dataset3[-train_index, ]

train_data <- train_dataset %>% select(sex.0,sex.1,age, test_time,Jitter,Jitter.Abs,Jitter.PPQ5,Jitter.DDP,
Shimmer, Shimmer.dB,Shimmer.APQ3,Shimmer.APQ11,
Shimmer.DDA,NHR,HNR,RPDE,DFA,PPE)

train_labels <- select(train_dataset,motor_UPDRS)
test_data <- test_dataset %>% select(sex.0,sex.1,age, test_time,Jitter,Jitter.Abs,Jitter.PPQ5,Jitter.DDP,
Shimmer, Shimmer.dB,Shimmer.APQ3,Shimmer.APQ11,
Shimmer.DDA,NHR,HNR,RPDE,DFA,PPE)
test_labels <- select(test_dataset,motor_UPDRS)``````

## Normalize the data

`````` # Normalize the data by subtracting the mean and dividing by the standard deviation
normalize<-function(x) {
y<-(x - mean(x)) / sd(x)
return(y)
}

normalized_train_data <-apply(train_data,2,normalize)
# Convert to matrix
train_labels <- as.matrix(train_labels)
normalized_test_data <- apply(test_data,2,normalize)
test_labels <- as.matrix(test_labels)``````

### Create the Deep Learning Model

``````model <- keras_model_sequential()
model %>%
layer_dense(units = 6, activation = 'relu', input_shape = dim(normalized_train_data)[2]) %>%
layer_dense(units = 9, activation = 'relu') %>%
layer_dense(units = 6, activation = 'relu') %>%
layer_dense(units = 1)

# Set the metrics required to be Mean Absolute Error and Mean Squared Error.For regression, the loss is
# mean_squared_error
model %>% compile(
loss = 'mean_squared_error',
optimizer = optimizer_rmsprop(),
metrics = c('mean_absolute_error','mean_squared_error')
)

# Fit the model
# Use the test data for validation
history <- model %>% fit(
normalized_train_data, train_labels,
epochs = 30, batch_size = 128,
validation_data = list(normalized_test_data,test_labels)
)``````

### Plot mean squared error, mean absolute error and loss for training data and test data

``````plot(history)
``````

Fig1

## 2. Binary classification in Tensorflow – Python

This is a simple binary classification problem from UCI Machine Learning repository and deals with data on Breast cancer from the Univ. of Wisconsin Breast Cancer Wisconsin (Diagnostic) Data Set bold text

In [31]:
```import tensorflow as tf
from tensorflow import keras
import pandas as pd
# Read the data set from UCI ML site
dataset_path = keras.utils.get_file("breast-cancer-wisconsin.data", "https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/breast-cancer-wisconsin.data")
raw_dataset = pd.read_csv(dataset_path, sep=",", na_values = "?", skipinitialspace=True,)
dataset = raw_dataset.copy()

#Check for Null and drop
dataset.isna().sum()
dataset = dataset.dropna()
dataset.isna().sum()

# Set the column names
"barenuclei","chromatin","normalnucleoli","mitoses","class"]
```
```Downloading data from https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/breast-cancer-wisconsin.data
24576/19889 [=====================================] - 0s 1us/step
id	thickness	cellsize	cellshape	adhesion	epicellsize	barenuclei	chromatin	normalnucleoli	mitoses	class
0	1002945	5	4	4	5	7	10.0	3	2	1	2
1	1015425	3	1	1	1	2	2.0	3	1	1	2
2	1016277	6	8	8	1	3	4.0	3	7	1	2
3	1017023	4	1	1	3	2	1.0	3	1	1	2
4	1017122	8	10	10	8	7	10.0	9	7	1	4```
```# Create a training/test set in the ratio 80/20
train_dataset = dataset.sample(frac=0.8,random_state=0)
test_dataset = dataset.drop(train_dataset.index)

# Set the training and test set
'epicellsize', 'barenuclei', 'chromatin', 'normalnucleoli','mitoses']]
'epicellsize', 'barenuclei', 'chromatin', 'normalnucleoli','mitoses']]
```
In [34]:
```# Generate the stats for each column to be used for normalization
train_stats = train_dataset1.describe()
train_stats = train_stats.transpose()
train_stats
```
Out[34]:
count mean std min 25% 50% 75% max
thickness 546.0 4.430403 2.812768 1.0 2.0 4.0 6.0 10.0
cellsize 546.0 3.179487 3.083668 1.0 1.0 1.0 5.0 10.0
cellshape 546.0 3.225275 3.005588 1.0 1.0 1.0 5.0 10.0
adhesion 546.0 2.921245 2.937144 1.0 1.0 1.0 4.0 10.0
epicellsize 546.0 3.261905 2.252643 1.0 2.0 2.0 4.0 10.0
barenuclei 546.0 3.560440 3.651946 1.0 1.0 1.0 7.0 10.0
chromatin 546.0 3.483516 2.492687 1.0 2.0 3.0 5.0 10.0
normalnucleoli 546.0 2.875458 3.064305 1.0 1.0 1.0 4.0 10.0
mitoses 546.0 1.609890 1.736762 1.0 1.0 1.0 1.0 10.0
In [0]:
```# Create target variables
train_labels = train_dataset.pop('class')
test_labels = test_dataset.pop('class')
```
In [0]:
```# Set the target variables as 0 or 1
train_labels[train_labels==2] =0 # benign
train_labels[train_labels==4] =1 # malignant

test_labels[test_labels==2] =0 # benign
test_labels[test_labels==4] =1 # malignant
```
In [0]:
```# Normalize by subtracting mean and dividing by standard deviation
def normalize(x):
return (x - train_stats['mean']) / train_stats['std']

# Convert columns to numeric
train_dataset1 = train_dataset1.apply(pd.to_numeric)
test_dataset1 = test_dataset1.apply(pd.to_numeric)

# Normalize
normalized_train_data = normalize(train_dataset1)
normalized_test_data = normalize(test_dataset1)
```
In [0]:
```# Create a model
model = tf.keras.Sequential([
keras.layers.Dense(6, activation=tf.nn.relu, input_shape=[len(train_dataset1.keys())]),
keras.layers.Dense(9, activation=tf.nn.relu),
keras.layers.Dense(6,activation=tf.nn.relu),
keras.layers.Dense(1)
])

# Use the RMSProp optimizer
optimizer = tf.keras.optimizers.RMSprop(0.01)

# Since this is binary classification use binary_crossentropy
model.compile(loss='binary_crossentropy',
optimizer=optimizer,
metrics=['acc'])

# Fit a model
history=model.fit(
normalized_train_data, train_labels,
epochs=1000, validation_data=(normalized_test_data,test_labels), verbose=0)
```
In [55]:
```hist = pd.DataFrame(history.history)
hist['epoch'] = history.epoch
hist.tail()
```
loss acc val_loss val_acc epoch
995 0.112499 0.992674 0.454739 0.970588 995
996 0.112499 0.992674 0.454739 0.970588 996
997 0.112499 0.992674 0.454739 0.970588 997
998 0.112499 0.992674 0.454739 0.970588 998
999 0.112499 0.992674 0.454739 0.970588 999
In [58]:
```# Plot training and test accuracy
plt.plot(history.history['acc'])
plt.plot(history.history['val_acc'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.ylim([0.9,1])
plt.show()

# Plot training and test loss
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.ylim([0,0.5])
plt.show()

```

### 2a. Binary classification in Tensorflow -R

This is a simple binary classification problem from UCI Machine Learning repository and deals with data on Breast cancer from the Univ. of Wisconsin Breast Cancer Wisconsin (Diagnostic) Data Set

``````# Read the data for Breast cancer (Wisconsin)

# Rename the columns
"barenuclei","chromatin","normalnucleoli","mitoses","class")

# Remove the columns id and class
dataset1 <- subset(dataset, select = -c(id, class))
dataset2 <- na.omit(dataset1)

# Convert the column to numeric
dataset2\$barenuclei <- as.numeric(dataset2\$barenuclei)``````

## Normalize the data

``````train_data <-apply(dataset2,2,normalize)
train_labels <- as.matrix(select(dataset,class))

# Set the target variables as 0 or 1 as it binary classification
train_labels[train_labels==2,]=0
train_labels[train_labels==4,]=1``````

### Create the Deep Learning model

``````model <- keras_model_sequential()
model %>%
layer_dense(units = 6, activation = 'relu', input_shape = dim(train_data)[2]) %>%
layer_dense(units = 9, activation = 'relu') %>%
layer_dense(units = 6, activation = 'relu') %>%
layer_dense(units = 1)

# Since this is a binary classification we use binary cross entropy
model %>% compile(
loss = 'binary_crossentropy',
optimizer = optimizer_rmsprop(),
metrics = c('accuracy')  # Metrics is accuracy
)``````

### Fit the model. Use 20% of data for validation

``````history <- model %>% fit(
train_data, train_labels,
epochs = 30, batch_size = 128,
validation_split = 0.2
)``````

### Plot the accuracy and loss for training and validation data

``````plot(history)
``````

### 3. MNIST in Tensorflow – Python

This takes the famous MNIST handwritten digits . It ca be seen that Tensorflow and Keras make short work of this famous problem of the late 1980s

```# Download MNIST data
mnist=tf.keras.datasets.mnist
# Set training and test data and labels

print(training_images.shape)
print(test_images.shape)
```
```(60000, 28, 28)
(10000, 28, 28)
```
In [61]:
```# Plot a sample image from MNIST and show contents
import matplotlib.pyplot as plt
plt.imshow(training_images[1])
print(training_images[1])
[[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 51 159 253
159 50 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 48 238 252 252
252 237 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 54 227 253 252 239
233 252 57 6 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 10 60 224 252 253 252 202
84 252 253 122 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 163 252 252 252 253 252 252
96 189 253 167 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 51 238 253 253 190 114 253 228
47 79 255 168 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 48 238 252 252 179 12 75 121 21
0 0 253 243 50 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 38 165 253 233 208 84 0 0 0 0
0 0 253 252 165 0 0 0 0 0]
[ 0 0 0 0 0 0 0 7 178 252 240 71 19 28 0 0 0 0
0 0 253 252 195 0 0 0 0 0]
[ 0 0 0 0 0 0 0 57 252 252 63 0 0 0 0 0 0 0
0 0 253 252 195 0 0 0 0 0]
[ 0 0 0 0 0 0 0 198 253 190 0 0 0 0 0 0 0 0
0 0 255 253 196 0 0 0 0 0]
[ 0 0 0 0 0 0 76 246 252 112 0 0 0 0 0 0 0 0
0 0 253 252 148 0 0 0 0 0]
[ 0 0 0 0 0 0 85 252 230 25 0 0 0 0 0 0 0 0
7 135 253 186 12 0 0 0 0 0]
[ 0 0 0 0 0 0 85 252 223 0 0 0 0 0 0 0 0 7
131 252 225 71 0 0 0 0 0 0]
[ 0 0 0 0 0 0 85 252 145 0 0 0 0 0 0 0 48 165
252 173 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 86 253 225 0 0 0 0 0 0 114 238 253
162 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 85 252 249 146 48 29 85 178 225 253 223 167
56 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 85 252 252 252 229 215 252 252 252 196 130 0
0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 28 199 252 252 253 252 252 233 145 0 0 0
0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 25 128 252 253 252 141 37 0 0 0 0
0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0]]

```
```# Normalize the images by dividing by 255.0
training_images = training_images/255.0
test_images = test_images/255.0

# Create a Sequential Keras model
model = tf.keras.models.Sequential([tf.keras.layers.Flatten(),
tf.keras.layers.Dense(1024,activation=tf.nn.relu),
tf.keras.layers.Dense(10,activation=tf.nn.softmax)])
```
In [68]:
```history=model.fit(training_images,training_labels,validation_data=(test_images, test_labels), epochs=5, verbose=1)
```
```Train on 60000 samples, validate on 10000 samples
Epoch 1/5
60000/60000 [==============================] - 17s 291us/sample - loss: 0.0020 - acc: 0.9999 - val_loss: 0.0719 - val_acc: 0.9810
Epoch 2/5
60000/60000 [==============================] - 17s 284us/sample - loss: 0.0021 - acc: 0.9998 - val_loss: 0.0705 - val_acc: 0.9821
Epoch 3/5
60000/60000 [==============================] - 17s 286us/sample - loss: 0.0017 - acc: 0.9999 - val_loss: 0.0729 - val_acc: 0.9805
Epoch 4/5
60000/60000 [==============================] - 17s 284us/sample - loss: 0.0014 - acc: 0.9999 - val_loss: 0.0762 - val_acc: 0.9804
Epoch 5/5
60000/60000 [==============================] - 17s 280us/sample - loss: 0.0015 - acc: 0.9999 - val_loss: 0.0735 - val_acc: 0.9812```

Fig 1

Fig 2

## MNIST in Tensorflow – R

The following code uses Tensorflow to learn MNIST’s handwritten digits ### Load MNIST data

``````mnist <- dataset_mnist()
x_train <- mnist\$train\$x
y_train <- mnist\$train\$y
x_test <- mnist\$test\$x
y_test <- mnist\$test\$y``````

### Reshape and rescale

``````# Reshape the array
x_train <- array_reshape(x_train, c(nrow(x_train), 784))
x_test <- array_reshape(x_test, c(nrow(x_test), 784))
# Rescale
x_train <- x_train / 255
x_test <- x_test / 255``````

### Convert out put to One Hot encoded format

``````y_train <- to_categorical(y_train, 10)
y_test <- to_categorical(y_test, 10)``````

### Fit the model

Use the softmax activation for recognizing 10 digits and categorical cross entropy for loss

``````model <- keras_model_sequential()
model %>%
layer_dense(units = 256, activation = 'relu', input_shape = c(784)) %>%
layer_dense(units = 128, activation = 'relu') %>%
layer_dense(units = 10, activation = 'softmax') # Use softmax

model %>% compile(
loss = 'categorical_crossentropy',
optimizer = optimizer_rmsprop(),
metrics = c('accuracy')
)``````

### Fit the model

Note: A smaller number of epochs has been used. For better performance increase number of epochs

``````history <- model %>% fit(
x_train, y_train,
epochs = 5, batch_size = 128,
validation_data = list(x_test,y_test)
)``````

### Plot the accuracy and loss for training and test data

``````plot(history)
``````

Conclusion
This post shows how to use Tensorflow and Keras in both Python & R
Hope you have fun with Tensorflow!!

To see all posts click Index of posts

# Cricketr learns new tricks : Performs fine-grained analysis of players

“He felt that his whole life was some kind of dream and he sometimes wondered whose it was and whether they were enjoying it.”

“The ships hung in the sky in much the same way that bricks don’t.”

“We demand rigidly defined areas of doubt and uncertainty!”

“For a moment, nothing happened. Then, after a second or so, nothing continued to happen.”

“The Answer to the Great Question… Of Life, the Universe and Everything… Is… Forty-two,’ said Deep Thought, with infinite majesty and calm.”

``                 The Hitchhiker's Guide to the Galaxy - Douglas Adams``

## Introduction

In this post, I introduce 2 new functions in my R package ‘cricketr’ (cricketr v0.22) see Re-introducing cricketr! : An R package to analyze performances of cricketers which enable granular analysis of batsmen and bowlers. They are

1. Step 1: getPlayerDataHA – This function is a wrapper around getPlayerData(), getPlayerDataOD() and getPlayerDataTT(), and adds an extra column ‘homeOrAway’ which says whether the match was played at home/away/neutral venues. A CSV file is created with this new column.
2. Setp 2: getPlayerDataOppnHA – This function allows you to slice & dice the data for batsmen and bowlers against specific oppositions, at home/away/neutral venues and between certain periods. This reduced subset of data can be used to perform analyses. A CSV file is created as an output based on the parameters of opposition, home or away and the interval of time

Note All the existing cricketr functions can be used on this smaller fine-grained data set for a closer analysis of players

Note 1: You have to call the above functions only once. You can reuse the CSV files in other functions

Important note: Don’t go too fine-grained by choosing just one opposition, in one of home/away/neutral and for too short a period. Too small a dataset may defeat the purpose of the analysis!

This post has been published in Rpubs and can be accessed at Cricketr learns new tricks

If you are passionate about cricket, and love analyzing cricket performances, then check out my racy book on cricket ‘Cricket analytics with cricketr and cricpy – Analytics harmony with R & Python’! This book discusses and shows how to use my R package ‘cricketr’ and my Python package ‘cricpy’ to analyze batsmen and bowlers in all formats of the game (Test, ODI and T20). The paperback is available on Amazon at \$21.99 and  the kindle version at \$9.99/Rs 449/-. A must read for any cricket lover! Check it out!!

## 1. Analyzing Tendulkar at 3 different stages of his career

The following functions analyze Sachin Tendulkar during 3 different periods of his illustrious career. a) 1st Jan 2001-1st Jan 2002 b) 1st Jan 2005-1st Jan 2006 c) 1st Jan 2012-1st Jan 2013

``````# Get the homeOrAway dataset for Tendulkar in matches
#Note: I have commented the lines to getPlayerDataHA() as I already have
# CSV file
#df=getPlayerDataHA(35320,tfile="tendulkarTestHA.csv",matchType="Test")

# Get Tendulkar's data for 2001-02
df1=getPlayerDataOppnHA(infile="tendulkarHA.csv",outfile="tendulkarTest2001.csv",
startDate="2001-01-01",endDate="2002-01-01")

# Get Tendulkar's data for 2005-06
df2=getPlayerDataOppnHA(infile="tendulkarHA.csv",outfile="tendulkarTest2005.csv",

startDate="2005-01-01",endDate="2006-01-01")

# Get Tendulkar's data for 20012-13
#df3=getPlayerDataOppnHA(infile="tendulkarHA.csv",outfile="tendulkarTest2012.csv",
#                        startDate="2012-01-01",endDate="2013-01-01")``````

`

## 1a Mean strike rate of Tendulkar in 2001,2005,2012

Note: Any of the cricketr R functions can be used on the fine-grained subset of data as below. The mean strike rate of Tendulkar is of the order of 60+ in 2001 which decreases to 50 and later to around 45

``````# Compute and plot mean strike rate of Tendulkar in the 3 periods
batsmanMeanStrikeRate ("./tendulkarTest2001.csv","Tendulkar-2001")``````

``batsmanMeanStrikeRate ("./tendulkarTest2005.csv","Tendulkar-2005")``

``batsmanMeanStrikeRate ("./tendulkarTest2012.csv","Tendulkar-2012")``

## 1b. Plot the performance of Tendulkar at venues during 2001,2005,2012

On an average Tendulkar score 60+ in 2001 and is really blazing. This performance decreases in 2005 and later in 2012

``````par(mfrow=c(1,3))
par(mar=c(4,4,2,2))
batsmanAvgRunsGround("tendulkarTest2001.csv","Tendulkar-2001")
batsmanAvgRunsGround("tendulkarTest2005.csv","Tendulkar-2005")
batsmanAvgRunsGround("tendulkarTest2012.csv","Tendulkar-2012")``````

``dev.off()``

## 1c. Plot the performance of Tendulkar against different oppositions during 2001,2005,2012

Sachin uniformly scores 50+ against formidable oppositions in 2001. In 2005 this decreases to 40 in 2005 and in 2012 it is around 25

``batsmanAvgRunsOpposition("tendulkarTest2001.csv","Tendulkar-2001")``
``batsmanAvgRunsOpposition("tendulkarTest2005.csv","Tendulkar-2005")``

``batsmanAvgRunsOpposition("tendulkarTest2012.csv","Tendulkar-2012")``

## 1d. Plot the relative cumulative average and relative strike rate of Tendulkar in 2001,2005,2012

The plot below compares Tendulkar’s cumulative strike rate and cumulative average during 3 different stages of his career

1. The cumulative average runs of Tendulkar is in the high 60+ in 2001, which drops to ~50 in 2005 and later plummets to the low 25s in 2012
2. The strike rate in 2001 for Tendulkar is amazing 60+
``````frames=list("tendulkarTest2001.csv","tendulkarTest2005.csv","tendulkarTest2012.csv")
names=list("Tendulkar-2001","Tendulkar-2005","Tendulkar-2012")
relativeBatsmanCumulativeAvgRuns(frames,names)``````

``relativeBatsmanCumulativeStrikeRate(frames,names)``

## 2. Analyzing Virat Kohli’s performance against England in England in 2014 and 2018

The analysis below looks at Kohli’s performance against England in ‘away’ venues (England) in 2014 and 2018

``````# Get the homeOrAway data for Kohli in Test matches
#df=getPlayerDataHA(253802,tfile="kohliTestHA.csv",type="batting",matchType="Test")

# Get the subset if data of Kohli's performance against England in England in 2014
df=getPlayerDataOppnHA(infile="kohliTestHA.csv",outfile="kohliTestEng2014.csv",
opposition=c("England"),homeOrAway=c("away"),startDate="2014-01-01",endDate="2015-01-01")

# Get the subset if data of Kohli's performance against England in England in 2018
df1=getPlayerDataOppnHA(infile="kohliHA.csv",outfile="kohliTestEng2018.csv",
opposition=c("England"),homeOrAway=c("away"),startDate="2018-01-01",endDate="2019-01-01")``````

## 2a. Kohli’s performance at England grounds in 2014 & 2018

Kohli had a miserable outing to England in 2014 with a string of low scores. In 2018 Kohli pulls himself out of the morass

``batsmanAvgRunsGround("kohliTestEng2014.csv","Kohli-Eng-2014")``

``batsmanAvgRunsGround("kohliTestEng2018.csv","Kohli-Eng-2018")``

## 2a. Kohli’s cumulative average runs in 2014 & 2018

Kohli’s cumulative average runs in 2014 is in the low 15s, while in 2018 it is 70+. Kohli stamps his class back again and undoes the bad memories of 2014

``batsmanCumulativeAverageRuns("kohliTestEng2014.csv", "Kohli-Eng-2014")``

``batsmanCumulativeAverageRuns("kohliTestEng2018.csv", "Kohli-Eng-2018")``

## 3a. Compare the performances of Ganguly, Dravid and VVS Laxman against opposition in ‘away’ matches in Tests

The analyses below compares the performances of Sourav Ganguly, Rahul Dravid and VVS Laxman against Australia, South Africa, and England in ‘away’ venues between 01 Jan 2002 to 01 Jan 2008

``````#Get the HA data for Ganguly, Dravid and Laxman
#df=getPlayerDataHA(28779,tfile="gangulyTestHA.csv",type="batting",matchType="Test")
#df=getPlayerDataHA(28114,tfile="dravidTestHA.csv",type="batting",matchType="Test")
#df=getPlayerDataHA(30750,tfile="laxmanTestHA.csv",type="batting",matchType="Test")

# Slice the data
df=getPlayerDataOppnHA(infile="gangulyTestHA.csv",outfile="gangulyTestAES2002-08.csv"
,opposition=c("Australia", "England", "South Africa"),
homeOrAway=c("away"),startDate="2002-01-01",endDate="2008-01-01")

df=getPlayerDataOppnHA(infile="dravidTestHA.csv",outfile="dravidTestAES2002-08.csv"
,opposition=c("Australia", "England", "South Africa"),
homeOrAway=c("away"),startDate="2002-01-01",endDate="2008-01-01")

df=getPlayerDataOppnHA(infile="laxmanTestHA.csv",outfile="laxmanTestAES2002-08.csv"
,opposition=c("Australia", "England", "South Africa"),
homeOrAway=c("away"),startDate="2002-01-01",endDate="2008-01-01")``````

## 3b Plot the relative cumulative average runs and relative cumative strike rate

Plot the relative cumulative average runs and relative cumative strike rate of Ganguly, Dravid and Laxman

-Dravid towers over Laxman and Ganguly with respect to cumulative average runs. – Ganguly has a superior strike rate followed by Laxman and then Dravid

``````frames=list("gangulyTestAES2002-08.csv","dravidTestAES2002-08.csv","laxmanTestAES2002-08.csv")
names=list("GangulyAusEngSA2002-08","DravidAusEngSA2002-08","LaxmanAusEngSA2002-08")
relativeBatsmanCumulativeAvgRuns(frames,names)``````

``relativeBatsmanCumulativeStrikeRate(frames,names)``

## 4. Compare the ODI performances of Rohit Sharma, Joe Root and Kane Williamson against opposition

Compare the performances of Rohit Sharma, Joe Root and Kane williamson in away & neutral venues against Australia, West Indies and Soouth Africa

• Joe Root piles us the runs in about 15 matches. Rohit has played far more ODIs than the other two and averages a steady 35+
``````# Get the ODI HA data for Rohit, Root and Williamson
#df=getPlayerDataHA(34102,tfile="rohitODIHA.csv",type="batting",matchType="ODI")
#df=getPlayerDataHA(303669,tfile="joerootODIHA.csv",type="batting",matchType="ODI")
#df=getPlayerDataHA(277906,tfile="williamsonODIHA.csv",type="batting",matchType="ODI")

# Subset the data for specific opposition in away and neutral venues
df=getPlayerDataOppnHA(infile="rohitODIHA.csv",outfile="rohitODIAusWISA.csv"
,opposition=c("Australia", "West Indies", "South Africa"),
homeOrAway=c("away","neutral"))

df=getPlayerDataOppnHA(infile="joerootODIHA.csv",outfile="joerootODIAusWISA.csv"
,opposition=c("Australia", "West Indies", "South Africa"),
homeOrAway=c("away","neutral"))

df=getPlayerDataOppnHA(infile="williamsonODIHA.csv",outfile="williamsonODIAusWiSA.csv"
,opposition=c("Australia", "West Indies", "South Africa"),
homeOrAway=c("away","neutral"))``````

## 4a. Compare cumulative strike rates and cumulative average runs of Rohit, Root and Williamson

The relative cumulative strike rate of all 3 are comparable

``````frames=list("rohitODIAusWISA.csv","joerootODIAusWISA.csv","williamsonODIAusWiSA.csv")
names=list("Rohit-ODI-AusWISA","Joe Root-ODI-AusWISA","Williamson-ODI-AusWISA")
relativeBatsmanCumulativeAvgRuns(frames,names)``````

``relativeBatsmanCumulativeStrikeRate(frames,names)``

## 5. Plot the performance of Dhoni in T20s against specific opposition at all venues

Plot the performances of Dhoni against Australia, West Indies, South Africa and England

``````# Get the HA T20 data for Dhoni
#df=getPlayerDataHA(28081,tfile="dhoniT20HA.csv",type="batting",matchType="T20")

#Subset the data
df=getPlayerDataOppnHA(infile="dhoniT20HA.csv",outfile="dhoniT20AusWISAEng.csv"
,opposition=c("Australia", "West Indies", "South Africa","England"),
homeOrAway=c("all"))``````

## 5a. Plot Dhoni’s performances in T20

Note You can use any of cricketr’s functions against the fine grained data

``batsmanAvgRunsOpposition("dhoniT20AusWISAEng.csv","Dhoni")``

``batsmanAvgRunsGround("dhoniT20AusWISAEng.csv","Dhoni")``

``batsmanCumulativeStrikeRate("dhoniT20AusWISAEng.csv","Dhoni")``

``batsmanCumulativeAverageRuns("dhoniT20AusWISAEng.csv","Dhoni")``

## 6. Compute and performances of Anil Kumble, Muralitharan and Warne in ‘away’ test matches

Compute the performances of Kumble, Warne and Maralitharan against New Zealand, West Indies, South Africa and England in pitches that are not ‘home’ pithes

``````# Get the bowling data for Kumble, Warne and Muralitharan in Test matches
#df=getPlayerDataHA(30176,tfile="kumbleTestHA.csv",type="bowling",matchType="Test")
#df=getPlayerDataHA(8166,tfile="warneTestHA.csv",type="bowling",matchType="Test")
#df=getPlayerDataHA(49636,tfile="muraliTestHA.csv",type="bowling",matchType="Test")

# Subset the data
df=getPlayerDataOppnHA(infile="kumbleTestHA.csv",outfile="kumbleTest-NZWISAEng.csv"
,opposition=c("New Zealand", "West Indies", "South Africa","England"),
homeOrAway=c("away"))

df=getPlayerDataOppnHA(infile="warneTestHA.csv",outfile="warneTest-NZWISAEng.csv"
,opposition=c("New Zealand", "West Indies", "South Africa","England"),
homeOrAway=c("away"))

df=getPlayerDataOppnHA(infile="muraliTestHA.csv",outfile="muraliTest-NZWISAEng.csv"
,opposition=c("New Zealand", "West Indies", "South Africa","England"),
homeOrAway=c("away"))``````

## 6a. Plot the average wickets of Kumble, Warne and Murali

``bowlerAvgWktsOpposition("kumbleTest-NZWISAEng.csv","Kumble-NZWISAEng-AN")``

``bowlerAvgWktsOpposition("warneTest-NZWISAEng.csv","Warne-NZWISAEng-AN")``

``bowlerAvgWktsOpposition("muraliTest-NZWISAEng.csv","Murali-NZWISAEng-AN")``

## 6b. Plot the average wickets in different grounds of Kumble, Warne and Murali

``bowlerAvgWktsGround("kumbleTest-NZWISAEng.csv","Kumblew")``

``bowlerAvgWktsGround("warneTest-NZWISAEng.csv","Warne")``

``bowlerAvgWktsGround("muraliTest-NZWISAEng.csv","murali")``

## 6c. Plot the cumulative average wickets and cumulative economy rate of Kumble, Warne and Murali

• Murali has the best economy rate followed by Kumble and then Warne
• Again Murali has the best cumulative average wickets followed by Warne and then Kumble
``````frames=list("kumbleTest-NZWISAEng.csv","warneTest-NZWISAEng.csv","muraliTest-NZWISAEng.csv")
names=list("Kumble","Warne","Murali")
relativeBowlerCumulativeAvgEconRate(frames,names)``````

``relativeBowlerCumulativeAvgWickets(frames,names)``

## 7. Compute and plot the performances of Bumrah in 2016, 2017 and 2018 in ODIs

``````# Get the HA data for Bumrah in ODI in bowling
df=getPlayerDataHA(625383,tfile="bumrahODIHA.csv",type="bowling",matchType="ODI")``````
``## [1] "Working..."``
``````# Slice the data for periods 2016, 2017 and 2018
df=getPlayerDataOppnHA(infile="bumrahODIHA.csv",outfile="bumrahODI2016.csv",
startDate="2016-01-01",endDate="2017-01-01")

df=getPlayerDataOppnHA(infile="bumrahODIHA.csv",outfile="bumrahODI2017.csv",
startDate="2017-01-01",endDate="2018-01-01")

df=getPlayerDataOppnHA(infile="bumrahODIHA.csv",outfile="bumrahODI2018.csv",
startDate="2018-01-01",endDate="2019-01-01")``````

## 7a. Compute the performances of Bumrah in 2016, 2017 and 2018

• Very clearly Bumrah is getting better at his art. His economy rate in 2018 is the best!!!
• Bumrah has had a very prolific year in 2017. However all the years he seems to be quite effective
``````frames=list("bumrahODI2016.csv","bumrahODI2017.csv","bumrahODI2018.csv")
names=list("Bumrah-2016","Bumrah-2017","Bumrah-2018")
relativeBowlerCumulativeAvgEconRate(frames,names)``````

``relativeBowlerCumulativeAvgWickets(frames,names)``

## 8. Compute and plot the performances of Shakib, Bumrah and Jadeja in T20 matches for bowling

``````# Get the HA bowling data for Shakib, Bumrah and Jadeja
df=getPlayerDataHA(56143,tfile="shakibT20HA.csv",type="bowling",matchType="T20")``````
``## [1] "Working..."``
``df=getPlayerDataHA(625383,tfile="bumrahT20HA.csv",type="bowling",matchType="T20")``
``## [1] "Working..."``
``df=getPlayerDataHA(234675,tfile="jadejaT20HA.csv",type="bowling",matchType="T20")``
``## [1] "Working..."``
``````# Slice the data for performances against Sri Lanka, Australia, South Africa and England
df=getPlayerDataOppnHA(infile="shakibT20HA.csv",outfile="shakibT20-SLAusSAEng.csv"
,opposition=c("Sri Lanka","Australia", "South Africa","England"),
homeOrAway=c("all"))
df=getPlayerDataOppnHA(infile="bumrahT20HA.csv",outfile="bumrahT20-SLAusSAEng.csv"
,opposition=c("Sri Lanka","Australia", "South Africa","England"),
homeOrAway=c("all"))

,opposition=c("Sri Lanka","Australia", "South Africa","England"),
homeOrAway=c("all"))``````

## 8a. Compare the relative performances of Shakib, Bumrah and Jadeja

• Jadeja and Bumrah have comparable economy rates. Shakib is more expensive
• Shakib pips Bumrah in number of cumulative wickets, though Bumrah is close behind
``````frames=list("shakibT20-SLAusSAEng.csv","bumrahT20-SLAusSAEng.csv","jadejaT20-SLAusSAEng.csv")
relativeBowlerCumulativeAvgEconRate(frames,names)``````

``relativeBowlerCumulativeAvgWickets(frames,names)``

## Conclusion

By getting the homeOrAway data for players using the profileNo, you can slice and dice the data based on your choice of opposition, whether you want matches that were played at home/away/neutral venues. Finally by specifying the period for which the data has to be subsetted you can create fine grained analysis.

Hope you have a great time with cricketr!!!

Also see

To see all posts click Index of posts

# Analyzing cricketers’ and cricket team’s performances with cricketr template

This post includes a template which you can use for analyzing the performances of cricketers, both batsmen and bowlers in Test, ODI and Twenty 20 cricket. Additionally this template can also be used for analyzing performances of teams in Test, ODI and T20 matches using my R package cricketr. To see actual usage of functions related to players in the R package cricketr see Introducing cricketr! : An R package to analyze performances of cricketers and associated posts on cricket in Index of posts. For the analyses on team performances see https://gigadom.in/2019/06/21/cricpy-adds-team-analytics-to-its-repertoire/

The ‘cricketr’ package uses the statistics info available in ESPN Cricinfo Statsguru. The current version of this package supports all formats of the game including Test, ODI and Twenty20 versions.

You should be able to install the package from GitHub and use the many functions available in the package. Please mindful of the ESPN Cricinfo Terms of Use

Take a look at my short video tutorial on my R package cricketr on Youtube – R package cricketr – A short tutorial

Do check out my interactive Shiny app implementation using the cricketr package – Sixer – R package cricketr’s new Shiny avatar

# The cricketr package

The cricketr package has several functions that perform several different analyses on both batsman and bowlers. The package can also analyze performances of teams The package has function that plot percentage frequency runs or wickets, runs likelihood for a batsman, relative run/strike rates of batsman and relative performance/economy rate for bowlers are available. Other interesting functions include batting performance moving average, forecast and a function to check whether the batsmans in in-form or out-of-form.

In addition performances of teams against different oppositions at different venues can be computed and plotted. The timeline of wins & losses can be plotted.

## A. Performances of batsmen and bowlers

The data for a particular player can be obtained with the getPlayerData() function. To do you will need to go to ESPN CricInfo Player and type in the name of the player for e.g Ricky Ponting, Sachin Tendulkar etc. This will bring up a page which have the profile number for the player e.g. for Sachin Tendulkar this would be http://www.espncricinfo.com/india/content/player/35320.html. Hence, Sachin’s profile is 35320. This can be used to get the data for Tendulkar as shown below

The cricketr package is now available from CRAN!!! You should be able to install as below

### 1. Install the cricketr package

``````if (!require("cricketr")){
install.packages("cricketr",lib = "c:/test")
}
library(cricketr)

```The cricketr package includes some pre-packaged sample (.csv) files. You can use these sample to test functions as shown below```
``````# Retrieve the file path of a data file installed with cricketr
#pathToFile <- system.file("data", "tendulkar.csv", package = "cricketr")
#batsman4s(pathToFile, "Sachin Tendulkar")

# The general format is pkg-function(pathToFile,par1,...)
#batsman4s(<path-To-File>,"Sachin Tendulkar")``````

The pre-packaged files can be accessed as shown above. To get the data of any player use the function in Test, ODI and Twenty20 use the following

### 2. For Test cricket

``#tendulkar <- getPlayerData(35320,dir="..",file="tendulkar.csv",type="batting",homeOrAway=c(1,2), result=c(1,2,4))``

### 2a. For ODI cricket

``#tendulkarOD <- getPlayerDataOD(35320,dir="..",file="tendulkarOD.csv",type="batting")``

### 2b For Twenty 20 cricket

``#tendulkarT20 <- getPlayerDataTT(35320,dir="..",file="tendulkarT20.csv",type="batting")``

Important Note 1: This needs to be done only once for a player. This function stores the player’s data in a CSV file (for e.g. tendulkar.csv as above) which can then be reused for all other functions. Once we have the data for the players many analyses can be done. This post will use the stored CSV file obtained with a prior getPlayerData for all subsequent analyses

Important Note 2: The same set of functions can be used for Tests, ODI and T20s. I have mentioned wherever you may need special functions for ODI and T20 below

## Sachin Tendulkar’s performance – Basic Analyses

The 3 plots below provide the following for Tendulkar

1. Frequency percentage of runs in each run range over the whole career
2. Mean Strike Rate for runs scored in the given range
3. A histogram of runs frequency percentages in runs ranges For example

### 3. Basic analyses

``````par(mfrow=c(1,3))
par(mar=c(4,4,2,2))
#batsmanRunsFreqPerf("./tendulkar.csv","Tendulkar")
#batsmanMeanStrikeRate("./tendulkar.csv","Tendulkar")
#batsmanRunsRanges("./tendulkar.csv","Tendulkar")
dev.off()``````
``````## null device
##           1``````
1. Player 1
2. Player 2
3. Player 3
4. Player 4

### 4. More analyses

``````par(mfrow=c(1,3))
par(mar=c(4,4,2,2))
#batsman4s("./player1.csv","Player1")
#batsman6s("./player1.csv","Player1")
#batsmanMeanStrikeRate("./player1.csv","Player1")

# For ODI and T20
#batsmanScoringRateODTT("./player1.csv","Player1")
dev.off()``````
``````## null device
##           1``````
``````par(mfrow=c(1,3))
par(mar=c(4,4,2,2))
#batsman4s("./player2.csv","Player2")
#batsman6s("./player2.csv","Player2")
#batsmanMeanStrikeRate("./player2.csv","Player2")
# For ODI and T20
#batsmanScoringRateODTT("./player1.csv","Player1")
dev.off()``````
``````## null device
##           1``````
``````par(mfrow=c(1,3))
par(mar=c(4,4,2,2))
#batsman4s("./player3.csv","Player3")
#batsman6s("./player3.csv","Player3")
#batsmanMeanStrikeRate("./player3.csv","Player3")
# For ODI and T20
#batsmanScoringRateODTT("./player1.csv","Player1")

dev.off()``````
``````## null device
##           1``````
``````par(mfrow=c(1,3))
par(mar=c(4,4,2,2))
#batsman4s("./player4.csv","Player4")
#batsman6s("./player4.csv","Player4")
#batsmanMeanStrikeRate("./player4.csv","Player4")
# For ODI and T20
#batsmanScoringRateODTT("./player1.csv","Player1")
dev.off()``````
``````## null device
##           1``````

Note: For mean strike rate in ODI and Twenty20 use the function batsmanScoringRateODTT()

### 5.Boxplot histogram plot

This plot shows a combined boxplot of the Runs ranges and a histogram of the Runs Frequency

``````#batsmanPerfBoxHist("./player1.csv","Player1")
#batsmanPerfBoxHist("./player2.csv","Player2")
#batsmanPerfBoxHist("./player3.csv","Player3")
#batsmanPerfBoxHist("./player4.csv","Player4")``````

### 6. Contribution to won and lost matches

For the 2 functions below you will have to use the getPlayerDataSp() function. I have commented this as I already have these files. This function can only be used for Test matches

``````#player1sp <- getPlayerDataSp(xxxx,tdir=".",tfile="player1sp.csv",ttype="batting")
#player2sp <- getPlayerDataSp(xxxx,tdir=".",tfile="player2sp.csv",ttype="batting")
#player3sp <- getPlayerDataSp(xxxx,tdir=".",tfile="player3sp.csv",ttype="batting")
#player4sp <- getPlayerDataSp(xxxx,tdir=".",tfile="player4sp.csv",ttype="batting")``````
``````par(mfrow=c(2,2))
par(mar=c(4,4,2,2))
#batsmanContributionWonLost("player1sp.csv","Player1")
#batsmanContributionWonLost("player2sp.csv","Player2")
#batsmanContributionWonLost("player3sp.csv","Player3")
#batsmanContributionWonLost("player4sp.csv","Player4")
dev.off()``````
``````## null device
##           1``````

### 7, Performance at home and overseas

This function also requires the use of getPlayerDataSp() as shown above. This can only be used for Test matches

``````par(mfrow=c(2,2))
par(mar=c(4,4,2,2))
#batsmanPerfHomeAway("player1sp.csv","Player1")
#batsmanPerfHomeAway("player2sp.csv","Player2")
#batsmanPerfHomeAway("player3sp.csv","Player3")
#batsmanPerfHomeAway("player4sp.csv","Player4")
dev.off()``````
``````## null device
##           1``````

### 8. Batsman average at different venues

``````par(mfrow=c(2,2))
par(mar=c(4,4,2,2))
#batsmanAvgRunsGround("./player1.csv","Player1")
#batsmanAvgRunsGround("./player2.csv","Player2")
#batsmanAvgRunsGround("./player3.csv","Ponting")
#batsmanAvgRunsGround("./player4.csv","Player4")
dev.off()``````
``````## null device
##           1``````

### 9. Batsman average against different opposition

``````par(mfrow=c(2,2))
par(mar=c(4,4,2,2))
#batsmanAvgRunsOpposition("./player1.csv","Player1")
#batsmanAvgRunsOpposition("./player2.csv","Player2")
#batsmanAvgRunsOpposition("./player3.csv","Ponting")
#batsmanAvgRunsOpposition("./player4.csv","Player4")
dev.off()``````
``````## null device
##           1``````

### 10. Runs Likelihood of batsman

``````par(mfrow=c(2,2))
par(mar=c(4,4,2,2))
#batsmanRunsLikelihood("./player1.csv","Player1")
#batsmanRunsLikelihood("./player2.csv","Player2")
#batsmanRunsLikelihood("./player3.csv","Ponting")
#batsmanRunsLikelihood("./player4.csv","Player4")
dev.off()``````
``````## null device
##           1``````

### 11. Moving Average of runs in career

``````par(mfrow=c(2,2))
par(mar=c(4,4,2,2))
#batsmanMovingAverage("./player1.csv","Player1")
#batsmanMovingAverage("./player2.csv","Player2")
#batsmanMovingAverage("./player3.csv","Ponting")
#batsmanMovingAverage("./player4.csv","Player4")
dev.off()``````
``````## null device
##           1``````

### 12. Cumulative Average runs of batsman in career

``````par(mfrow=c(2,2))
par(mar=c(4,4,2,2))
#batsmanCumulativeAverageRuns("./player1.csv","Player1")
#batsmanCumulativeAverageRuns("./player2.csv","Player2")
#batsmanCumulativeAverageRuns("./player3.csv","Ponting")
#batsmanCumulativeAverageRuns("./player4.csv","Player4")
dev.off()``````
``````## null device
##           1``````

### 13. Cumulative Average strike rate of batsman in career

``````par(mfrow=c(2,2))
par(mar=c(4,4,2,2))
#batsmanCumulativeStrikeRate("./player1.csv","Player1")
#batsmanCumulativeStrikeRate("./player2.csv","Player2")
#batsmanCumulativeStrikeRate("./player3.csv","Ponting")
#batsmanCumulativeStrikeRate("./player4.csv","Player4")
dev.off()``````
``````## null device
##           1``````

### 14. Future Runs forecast

Here are plots that forecast how the batsman will perform in future. In this case 90% of the career runs trend is uses as the training set. the remaining 10% is the test set.

A Holt-Winters forecating model is used to forecast future performance based on the 90% training set. The forecated runs trend is plotted. The test set is also plotted to see how close the forecast and the actual matches

Take a look at the runs forecasted for the batsman below.

``````par(mfrow=c(2,2))
par(mar=c(4,4,2,2))
#batsmanPerfForecast("./player1.csv","Player1")
#batsmanPerfForecast("./player2.csv","Player2")
#batsmanPerfForecast("./player3.csv","Player3")
#batsmanPerfForecast("./player4.csv","Player4")
dev.off()``````
``````## null device
##           1``````

### 15. Relative Mean Strike Rate plot

The plot below compares the Mean Strike Rate of the batsman for each of the runs ranges of 10 and plots them. The plot indicate the following

``````frames <- list("./player1.csv","./player2.csv","player3.csv","player4.csv")
names <- list("Player1","Player2","Player3","Player4")
#relativeBatsmanSR(frames,names)``````

### 16. Relative Runs Frequency plot

The plot below gives the relative Runs Frequency Percetages for each 10 run bucket. The plot below show

``````frames <- list("./player1.csv","./player2.csv","player3.csv","player4.csv")
names <- list("Player1","Player2","Player3","Player4")
#relativeRunsFreqPerf(frames,names)``````

### 17. Relative cumulative average runs in career

``````frames <- list("./player1.csv","./player2.csv","player3.csv","player4.csv")
names <- list("Player1","Player2","Player3","Player4")
#relativeBatsmanCumulativeAvgRuns(frames,names)``````

### 18. Relative cumulative average strike rate in career

``````frames <- list("./player1.csv","./player2.csv","player3.csv","player4.csv")
names <- list("Player1","Player2","Player3","player4")
#relativeBatsmanCumulativeStrikeRate(frames,names)``````

### 19. Check Batsman In-Form or Out-of-Form

The below computation uses Null Hypothesis testing and p-value to determine if the batsman is in-form or out-of-form. For this 90% of the career runs is chosen as the population and the mean computed. The last 10% is chosen to be the sample set and the sample Mean and the sample Standard Deviation are caculated.

The Null Hypothesis (H0) assumes that the batsman continues to stay in-form where the sample mean is within 95% confidence interval of population mean The Alternative (Ha) assumes that the batsman is out of form the sample mean is beyond the 95% confidence interval of the population mean.

A significance value of 0.05 is chosen and p-value us computed If p-value >= .05 – Batsman In-Form If p-value < 0.05 – Batsman Out-of-Form

Note Ideally the p-value should be done for a population that follows the Normal Distribution. But the runs population is usually left skewed. So some correction may be needed. I will revisit this later

This is done for the Top 4 batsman

``````#checkBatsmanInForm("./player1.csv","Player1")
#checkBatsmanInForm("./player2.csv","Player2")
#checkBatsmanInForm("./player3.csv","Player3")
#checkBatsmanInForm("./player4.csv","Player4")``````

### 20. 3D plot of Runs vs Balls Faced and Minutes at Crease

The plot is a scatter plot of Runs vs Balls faced and Minutes at Crease. A prediction plane is fitted

``````par(mfrow=c(1,2))
par(mar=c(4,4,2,2))
#battingPerf3d("./player1.csv","Player1")
#battingPerf3d("./player2.csv","Player2")``````
``````par(mfrow=c(1,2))
par(mar=c(4,4,2,2))
#battingPerf3d("./player3.csv","Player3")
#battingPerf3d("./player4.csv","player4")
dev.off()``````
``````## null device
##           1``````

### 21. Predicting Runs given Balls Faced and Minutes at Crease

A multi-variate regression plane is fitted between Runs and Balls faced +Minutes at crease.

``````BF <- seq( 10, 400,length=15)
Mins <- seq(30,600,length=15)
newDF <- data.frame(BF,Mins)
#Player1 <- batsmanRunsPredict("./player1.csv","Player1",newdataframe=newDF)
#Player2 <- batsmanRunsPredict("./player2.csv","Player2",newdataframe=newDF)
#ponting <- batsmanRunsPredict("./player3.csv","Player3",newdataframe=newDF)
#sangakkara <- batsmanRunsPredict("./player4.csv","Player4",newdataframe=newDF)``````
``````#batsmen <-cbind(round(Player1\$Runs),round(Player2\$Runs),round(Player3\$Runs),round(Player4\$Runs))
#colnames(batsmen) <- c("Player1","Player2","Player3","Player4")
#newDF <- data.frame(round(newDF\$BF),round(newDF\$Mins))
#colnames(newDF) <- c("BallsFaced","MinsAtCrease")
#predictedRuns <- cbind(newDF,batsmen)
#predictedRuns``````

## Analysis of bowlers

1. Bowler1
2. Bowler2
3. Bowler3
4. Bowler4

player1 <- getPlayerData(xxxx,dir=“..”,file=“player1.csv”,type=“bowling”) Note For One day you will have to use getPlayerDataOD() and for Twenty20 it is getPlayerDataTT()

### 21. Wicket Frequency Plot

This plot below computes the percentage frequency of number of wickets taken for e.g 1 wicket x%, 2 wickets y% etc and plots them as a continuous line

``````par(mfrow=c(1,3))
par(mar=c(4,4,2,2))
#bowlerWktsFreqPercent("./bowler1.csv","Bowler1")
#bowlerWktsFreqPercent("./bowler2.csv","Bowler2")
#bowlerWktsFreqPercent("./bowler3.csv","Bowler3")
dev.off()``````
``````## null device
##           1``````

### 22. Wickets Runs plot

``````par(mfrow=c(1,3))
par(mar=c(4,4,2,2))
#bowlerWktsRunsPlot("./bowler1.csv","Bowler1")
#bowlerWktsRunsPlot("./bowler2.csv","Bowler2")
#bowlerWktsRunsPlot("./bowler3.csv","Bowler3")
dev.off()``````
``````## null device
##           1``````

### 23. Average wickets at different venues

``#bowlerAvgWktsGround("./bowler3.csv","Bowler3")``

### 24. Average wickets against different opposition

``#bowlerAvgWktsOpposition("./bowler3.csv","Bowler3")``

### 25. Wickets taken moving average

``````par(mfrow=c(1,3))
par(mar=c(4,4,2,2))
#bowlerMovingAverage("./bowler1.csv","Bowler1")
#bowlerMovingAverage("./bowler2.csv","Bowler2")
#bowlerMovingAverage("./bowler3.csv","Bowler3")

dev.off()``````
``````## null device
##           1``````

### 26. Cumulative Wickets taken

``````par(mfrow=c(1,3))
par(mar=c(4,4,2,2))
#bowlerCumulativeAvgWickets("./bowler1.csv","Bowler1")
#bowlerCumulativeAvgWickets("./bowler2.csv","Bowler2")
#bowlerCumulativeAvgWickets("./bowler3.csv","Bowler3")
dev.off()``````
``````## null device
##           1``````

### 27. Cumulative Economy rate

``````par(mfrow=c(1,3))
par(mar=c(4,4,2,2))
#bowlerCumulativeAvgEconRate("./bowler1.csv","Bowler1")
#bowlerCumulativeAvgEconRate("./bowler2.csv","Bowler2")
#bowlerCumulativeAvgEconRate("./bowler3.csv","Bowler3")
dev.off()``````
``````## null device
##           1``````

### 28. Future Wickets forecast

Here are plots that forecast how the bowler will perform in future. In this case 90% of the career wickets trend is used as the training set. the remaining 10% is the test set.

A Holt-Winters forecating model is used to forecast future performance based on the 90% training set. The forecated wickets trend is plotted. The test set is also plotted to see how close the forecast and the actual matches

``````par(mfrow=c(1,3))
par(mar=c(4,4,2,2))
#bowlerPerfForecast("./bowler1.csv","Bowler1")
#bowlerPerfForecast("./bowler2.csv","Bowler2")
#bowlerPerfForecast("./bowler3.csv","Bowler3")
dev.off()``````
``````## null device
##           1``````

### 29. Contribution to matches won and lost

As discussed above the next 2 charts require the use of getPlayerDataSp(). This can only be done for Test matches

``````#bowler1sp <- getPlayerDataSp(xxxx,tdir=".",tfile="bowler1sp.csv",ttype="bowling")
#bowler2sp <- getPlayerDataSp(xxxx,tdir=".",tfile="bowler2sp.csv",ttype="bowling")
#bowler3sp <- getPlayerDataSp(xxxx,tdir=".",tfile="bowler3sp.csv",ttype="bowling")``````
``````par(mfrow=c(1,3))
par(mar=c(4,4,2,2))
#bowlerContributionWonLost("bowler1sp","Bowler1")
#bowlerContributionWonLost("bowler2sp","Bowler2")
#bowlerContributionWonLost("bowler3sp","Bowler3")
dev.off()``````
``````## null device
##           1``````

### 30. Performance home and overseas.

This can only be done for Test matches

``````par(mfrow=c(1,3))
par(mar=c(4,4,2,2))
#bowlerPerfHomeAway("bowler1sp","Bowler1")
#bowlerPerfHomeAway("bowler2sp","Bowler2")
#bowlerPerfHomeAway("bowler3sp","Bowler3")
dev.off()``````
``````## null device
##           1``````

### 31 Relative Wickets Frequency Percentage

``````frames <- list("./bowler1.csv","./bowler3.csv","bowler2.csv")
names <- list("Bowler1","Bowler3","Bowler2")
#relativeBowlingPerf(frames,names)``````

### 32 Relative Economy Rate against wickets taken

``````frames <- list("./bowler1.csv","./bowler3.csv","bowler2.csv")
names <- list("Bowler1","Bowler3","Bowler2")
#relativeBowlingER(frames,names)``````

### 33 Relative cumulative average wickets of bowlers in career

``````frames <- list("./bowler1.csv","./bowler3.csv","bowler2.csv")
names <- list("Bowler1","Bowler3","Bowler2")
#relativeBowlerCumulativeAvgWickets(frames,names)``````

### 34 Relative cumulative average economy rate of bowlers

``````frames <- list("./bowler1.csv","./bowler3.csv","bowler2.csv")
names <- list("Bowler1","Bowler3","Bowler2")
#relativeBowlerCumulativeAvgEconRate(frames,names)``````

### 35 Check for bowler in-form/out-of-form

The below computation uses Null Hypothesis testing and p-value to determine if the bowler is in-form or out-of-form. For this 90% of the career wickets is chosen as the population and the mean computed. The last 10% is chosen to be the sample set and the sample Mean and the sample Standard Deviation are caculated.

The Null Hypothesis (H0) assumes that the bowler continues to stay in-form where the sample mean is within 95% confidence interval of population mean The Alternative (Ha) assumes that the bowler is out of form the sample mean is beyond the 95% confidence interval of the population mean.

A significance value of 0.05 is chosen and p-value us computed If p-value >= .05 – Batsman In-Form If p-value < 0.05 – Batsman Out-of-Form

Note Ideally the p-value should be done for a population that follows the Normal Distribution. But the runs population is usually left skewed. So some correction may be needed. I will revisit this later

Note: The check for the form status of the bowlers indicate

``````#checkBowlerInForm("./bowler1.csv","Bowler1")
#checkBowlerInForm("./bowler2.csv","Bowler2")
#checkBowlerInForm("./bowler3.csv","Bowler3")
dev.off()``````
``````## null device
##           1``````

### 36. Performing granular analysis of batsmen and bowlers

To perform granular analysis of batsmen and bowlers do the following 2 steps

1. Step 1: getPlayerDataHA – This function is a wrapper around getPlayerData(), getPlayerDataOD() and getPlayerDataTT(), and adds an extra column ‘homeOrAway’ which says whether the match was played at home/away/neutral venues. A CSV file is created with this new column.
2. Step2:getPlayerDataOppnHA – This function allows you to slice & dice the data for batsmen and bowlers against specific oppositions, at home/away/neutral venues and between certain periods. This reduced subset of data can be used to perform analyses. A CSV file is created as an output based on the parameters of opposition, home or away and the interval of time

### 37. GetPlayerDataHA (Batsmen, Tests)

``````#This saves a file playerTestHA.csv
#df=getPlayerDataHA(<profileNo>,tfile="playerTestHA.csv",type="batting", matchType="Test")

#Use the generate file to create a subset of data
#df1=getPlayerDataOppnHA(infile="playerTestHA.csv",outfile="playerTestFile1.csv",
#                         startDate=<start Date>,endDate=<end Date>)``````

### 38. GetPlayerDataHA (Bowlers, Tests)

``````#This saves a file playerTestHA.csv
#df=getPlayerDataHA(<profileNo>,tfile="playerTestHA.csv",type="bowling", matchType="Test")

#Use the generate file to create a subset of data
#df1=getPlayerDataOppnHA(infile="playerTestHA.csv",outfile="playerTestFile1.csv",
#                         startDate=<start Date>,endDate=<end Date>)``````

### 39. GetPlayerDataHA (Batsmen, ODI)

``````#This saves a file playerTestHA.csv
#df=getPlayerDataHA(<profileNo>,tfile="playerODIHA.csv",type="batting", matchType="ODI")

#Use the generate file to create a subset of data
#df1=getPlayerDataOppnHA(infile="playerODIHA.csv",outfile="playerODIFile1.csv",
#                         startDate=<start Date>,endDate=<end Date>)``````

### 40. GetPlayerDataHA (Bowlers, ODI)

``````#This saves a file playerTestHA.csv
#df=getPlayerDataHA(<profileNo>,tfile="playerODIHA.csv",type="bowling", matchType="ODI")

#Use the generate file to create a subset of data
#df1=getPlayerDataOppnHA(infile="playerODIHA.csv",outfile="playerODIFile1.csv",
#                         startDate=<start Date>,endDate=<end Date>)``````

### 41. GetPlayerDataHA (Batsmen, T20)

``````#This saves a file playerTestHA.csv
#df=getPlayerDataHA(<profileNo>,tfile="playerT20HA.csv",type="batting", matchType="T20")

#Use the generate file to create a subset of data
#df1=getPlayerDataOppnHA(infile="playerT20HA.csv",outfile="playerT20File1.csv",
#                         startDate=<start Date>,endDate=<end Date>)``````

### 42. GetPlayerDataHA (Bowlers, T20)

``````#This saves a file playerTestHA.csv
#df=getPlayerDataHA(<profileNo>,tfile="playerT20HA.csv",type="bowling", matchType="T20")

#Use the generate file to create a subset of data
#df1=getPlayerDataOppnHA(infile="playerT20HA.csv",outfile="playerT20File1.csv",
#                         startDate=<start Date>,endDate=<end Date>)``````

Important Note Once you get the subset of data for batsmen and bowlers playerTestFile1.csv, playerODIFile1.csv or playerT20File1.csv , you can use any of the cricketr functions on the subset of data for a fine-grained analysis

## B. Performances of teams

The following functions will get the team data for Tests, ODI and T20s

### 1a. Get Test team data

``````#country1Test= getTeamDataHomeAway(dir=".",teamView="bat",matchType="Test",file="country1Test.csv",save=True,teamName="Country1")
#country2Test= getTeamDataHomeAway(dir=".",teamView="bat",matchType="Test",file="country2Test.csv",save=True,teamName="Country2")
#country3Test= getTeamDataHomeAway(dir=".",teamView="bat",matchType="Test",file="country3Test.csv",save=True,teamName="Country3")``````

### 1b. Get ODI team data

``````#team1ODI=  getTeamDataHomeAway(dir=".",matchType="ODI",file="team1ODI.csv",save=True,teamName="team1")
#team2ODI=  getTeamDataHomeAway(dir=".",matchType="ODI",file="team2ODI.csv",save=True,teamName="team2")
#team3ODI=  getTeamDataHomeAway(dir=".",matchType="ODI",file="team3ODI.csv",save=True,teamName="team3")``````

### 1c. Get T20 team data

``````#team1T20 = getTeamDataHomeAway(matchType="T20",file="team1T20.csv",save=True,teamName="team1")
#team2T20 = getTeamDataHomeAway(matchType="T20",file="team2T20.csv",save=True,teamName="team2")
#team3T20 = getTeamDataHomeAway(matchType="T20",file="team3T20.csv",save=True,teamName="team3")``````

### 2a. Test – Analyzing test performances against opposition

``````# Get the performance of Indian test team against all teams at all venues as a dataframe
#df <- teamWinLossStatusVsOpposition("country1Test.csv",teamName="Country1",opposition=c("all"),homeOrAway=c("all"),matchType="Test",plot=FALSE)

# Plot the performance of Country1 Test team  against all teams at all venues
#teamWinLossStatusVsOpposition("country1Test.csv",teamName="Country1",opposition=c("all"),homeOrAway=c("all"),matchType="Test",plot=TRUE)

# Plot the performance of Country1 Test team  against specific teams at home/away venues
#teamWinLossStatusVsOpposition("country1Test.csv",teamName="Country1",opposition=c("Country2","Country3","Country4"),homeOrAway=c("home","away","neutral"),matchType="Test",plot=TRUE)``````

### 2b. Test – Analyzing test performances against opposition at different grounds

``````# Get the performance of Indian test team against all teams at all venues as a dataframe
#df <- teamWinLossStatusAtGrounds("country1Test.csv",teamName="Country1",opposition=c("all"),homeOrAway=c("all"),matchType="Test",plot=FALSE)

# Plot the performance of Country1 Test team  against all teams at all venues
#teamWinLossStatusAtGrounds("country1Test.csv",teamName="Country1",opposition=c("all"),homeOrAway=c("all"),matchType="Test",plot=TRUE)

# Plot the performance of Country1 Test team  against specific teams at home/away venues
#teamWinLossStatusAtGrounds("country1Test.csv",teamName="Country1",opposition=c("Country2","Country3","Country4"),homeOrAway=c("home","away","neutral"),matchType="Test",plot=TRUE)``````

### 2c. Test – Plot time lines of wins and losses

``````#plotTimelineofWinsLosses("country1Test.csv",team="Country1",opposition=c("all"), #startDate="1970-01-01",endDate="2017-01-01")
#plotTimelineofWinsLosses("country1Test.csv",team="Country1",opposition=c("Country2","Count#ry3","Country4"), homeOrAway=c("home",away","neutral"), startDate=<start Date> #,endDate=<endDate>)``````

### 3a. ODI – Analyzing test performances against opposition

``````#df <- teamWinLossStatusVsOpposition("team1ODI.csv",teamName="Team1",opposition=c("all"),homeOrAway=c("all"),matchType="ODI",plot=FALSE)

# Plot the performance of team1  in ODIs against Sri Lanka, India at all venues
#teamWinLossStatusVsOpposition("team1ODI.csv",teamName="Team1",opposition=c("all"),homeOrAway=c(all"),matchType="ODI",plot=TRUE)

# Plot the performance of Team1 ODI team  against specific teams at home/away venues
#teamWinLossStatusVsOpposition("team1ODI.csv",teamName="Team1",opposition=c("Team2","Team3","Team4"),homeOrAway=c("home","away","neutral"),matchType="ODI",plot=TRUE)``````

### 3b. ODI – Analyzing test performances against opposition at different venues

``````#df <- teamWinLossStatusAtGrounds("team1ODI.csv",teamName="Team1",opposition=c("all"),homeOrAway=c("all"),matchType="ODI",plot=FALSE)

# Plot the performance of Team1s in ODIs specific ODI teams at all venues
#teamWinLossStatusAtGrounds("team1ODI.csv",teamName="Team1",opposition=c("all"),homeOrAway=c(all"),matchType="ODI",plot=TRUE)

# Plot the performance of Team1 against specific ODI teams at home/away venues
#teamWinLossStatusAtGrounds("team1ODI.csv",teamName="Team1",opposition=c("Team2","Team3","Team4"),homeOrAway=c("home","away","neutral"),matchType="ODI",plot=TRUE)``````

### 3c. ODI – Plot time lines of wins and losses

``````#Plot the time line of wins/losses of Bangladesh ODI team between 2 dates all venues
#plotTimelineofWinsLosses("team1ODI.csv",team="Team1",startDate=<start date> ,endDa#te=<end date>,matchType="ODI")

#Plot the time line of wins/losses against specific opposition between 2 dates
#plotTimelineofWinsLosses("team1ODI.csv",team="Team1",opposition=c("Team2","Team2"), homeOrAway=c("home",away","neutral"), startDate=<start date>,endDate=<end date> ,matchType="ODI")``````

### 4a. T20 – Analyzing test performances against opposition

``````#df <- teamWinLossStatusVsOpposition("teamT20.csv",teamName="Team1",opposition=c("all"),homeOrAway=c("all"),matchType="T20",plot=FALSE)

# Plot the performance of Team1 in T20s  against  all opposition at all venues
#teamWinLossStatusVsOpposition("teamT20.csv",teamName="Team1",opposition=c("all"),homeOrAway=c(all"),matchType="T20",plot=TRUE)

# Plot the performance of T20 Test team  against specific teams at home/away venues
#teamWinLossStatusVsOpposition("teamT20.csv",teamName="Team1",opposition=c("Team2","Team3","Team4"),homeOrAway=c("home","away","neutral"),matchType="T20",plot=TRUE)``````

### 4b. T20 – Analyzing test performances against opposition at different venues

``````#df <- teamWinLossStatusAtGrounds("teamT20.csv",teamName="Team1",opposition=c("all"),homeOrAway=c("all"),matchType="T20",plot=FALSE)

# Plot the performance of Team1s in ODIs specific ODI teams at all venues
#teamWinLossStatusAtGrounds("teamT20.csv",teamName="Team1",opposition=c("all"),homeOrAway=c(all"),matchType="T20",plot=TRUE)

# Plot the performance of Team1 against specific ODI teams at home/away venues
#teamWinLossStatusAtGrounds("teamT20.csv",teamName="Team1",opposition=c("Team2","Team3","Team4"),homeOrAway=c("home","away","neutral"),matchType="T20",plot=TRUE)``````

### 4c. T20 – Plot time lines of wins and losses

``````#Plot the time line of wins/losses of Bangladesh ODI team between 2 dates all venues
#plotTimelineofWinsLosses("teamT20.csv",team="Team1",startDate=<start date> ,endDa#te=<end date>,matchType="T20")

#Plot the time line of wins/losses against specific opposition between 2 dates
#plotTimelineofWinsLosses("teamT20.csv",team="Team1",opposition=c("Team2","Team2"), homeOrAway=c("home",away","neutral"), startDate=<start date>,endDate=<end date> ,matchType="T20")``````

## Conclusion

Using the above template, analysis can be done for both batsmen and bowlers in Test, ODI and T20. Also analysis of any any team in Test, ODI and T20 against other specific opposition, at home/away and neutral venues can be performed.

Have fun with cricketr!!

# Big Data-4: Webserver log analysis with RDDs, Pyspark, SparkR and SparklyR

“There’s something so paradoxical about pi. On the one hand, it represents order, as embodied by the shape of a circle, long held to be a symbol of perfection and eternity. On the other hand, pi is unruly, disheveled in appearance, its digits obeying no obvious rule, or at least none that we can perceive. Pi is elusive and mysterious, forever beyond reach. Its mix of order and disorder is what makes it so bewitching. ”

From  Infinite Powers by Steven Strogatz

Anybody who wants to be “anybody” in Big Data must necessarily be able to work on both large structured and unstructured data.  Log analysis is critical in any enterprise which is usually unstructured. As I mentioned in my previous post Big Data: On RDDs, Dataframes,Hive QL with Pyspark and SparkR-Part 3 RDDs are typically used to handle unstructured data. Spark has the Dataframe abstraction over RDDs which performs better as it is optimized with the Catalyst optimization engine. Nevertheless, it is important to be able to process with RDDs.  This post is a continuation of my 3 earlier posts on Big Data namely

This post uses publicly available Webserver logs from NASA. The logs are for the months Jul 95 and Aug 95 and are a good place to start unstructured text analysis/log analysis. I highly recommend parsing these publicly available logs with regular expressions. It is only when you do that the truth of Jamie Zawinski’s pearl of wisdom

“Some people, when confronted with a problem, think “I know, I’ll use regular expressions.” Now they have two problems.” – Jamie Zawinksi

hits home. I spent many hours struggling with regex!!

For this post for the RDD part,  I had to refer to Dr. Fisseha Berhane’s blog post Webserver Log Analysis and for the Pyspark part, to the Univ. of California Specialization which I had done 3 years back Big Data Analysis with Apache Spark. Once I had played around with the regex for RDDs and PySpark I managed to get SparkR and SparklyR versions to work.

The notebooks used in this post have been published and are available at

An essential and unavoidable aspect of Big Data processing is the need to process unstructured text.Web server logs are one such area which requires Big Data techniques to process massive amounts of logs. The Common Log Format also known as the NCSA Common log format, is a standardized text file format used by web servers when generating server log files. Because the format is standardized, the files can be readily analyzed.

A publicly available webserver logs is the NASA-HTTP Web server logs. This is good dataset with which we can play around to get familiar to handling web server logs. The logs can be accessed at NASA-HTTP

Description These two traces contain two month’s worth of all HTTP requests to the NASA Kennedy Space Center WWW server in Florida.

Format The logs are an ASCII file with one line per request, with the following columns:

-host making the request. A hostname when possible, otherwise the Internet address if the name could not be looked up.

-timestamp in the format “DAY MON DD HH:MM:SS YYYY”, where DAY is the day of the week, MON is the name of the month, DD is the day of the month, HH:MM:SS is the time of day using a 24-hour clock, and YYYY is the year. The timezone is -0400.

-request given in quotes.

## 1 Parse Web server logs with RDDs

### 1.1 Read NASA Web server logs

Read the logs files from NASA for the months Jul 95 and Aug 95

from pyspark import SparkContext, SparkConf
```from pyspark.sql import SQLContext

conf = SparkConf().setAppName("Spark-Logs-Handling").setMaster("local[*]")
sc = SparkContext.getOrCreate(conf)

sqlcontext = SQLContext(sc)
rdd = sc.textFile("/FileStore/tables/NASA_access_log_*.gz")
rdd.count()
Out[1]: 3461613
```

### 1.2Check content

Check the logs to identify the parsing rules required for the logs

i=0
```for line in rdd.sample(withReplacement = False, fraction = 0.00001, seed = 100).collect():
i=i+1
print(line)
if i >5:
break
```
ix-stp-fl2-19.ix.netcom.com – – [03/Aug/1995:23:03:09 -0400] “GET /images/faq.gif HTTP/1.0” 200 263
slip183-1.kw.jp.ibm.net – – [04/Aug/1995:18:42:17 -0400] “GET /shuttle/missions/sts-70/images/DSC-95EC-0001.gif HTTP/1.0” 200 107133
piweba4y.prodigy.com – – [05/Aug/1995:19:17:41 -0400] “GET /icons/menu.xbm HTTP/1.0” 200 527
ruperts.bt-sys.bt.co.uk – – [07/Aug/1995:04:44:10 -0400] “GET /shuttle/countdown/video/livevideo2.gif HTTP/1.0” 200 69067
dal06-04.ppp.iadfw.net – – [07/Aug/1995:21:10:19 -0400] “GET /images/NASA-logosmall.gif HTTP/1.0” 200 786
p15.ppp-1.directnet.com – – [10/Aug/1995:01:22:54 -0400] “GET /images/KSC-logosmall.gif HTTP/1.0” 200 1204

### 1.3 Write the parsing rule for each of the fields

• host
• timestamp
• path
• status
• content_bytes

### 1.21 Get IP address/host name

This regex is at the start of the log and includes any non-white characted

import re
```rslt=(rdd.map(lambda line: re.search('\S+',line)
.group(0))
.take(3)) # Get the IP address \host name
rslt
```
Out[3]: [‘in24.inetnebr.com’, ‘uplherc.upl.com’, ‘uplherc.upl.com’]

## 1.22 Get timestamp

Get the time stamp

rslt=(rdd.map(lambda line: re.search(‘(\S+ -\d{4})’,line)
```    .groups())
.take(3))  #Get the  date
rslt
```
[(‘[01/Aug/1995:00:00:01 -0400’,),
(‘[01/Aug/1995:00:00:07 -0400’,),
(‘[01/Aug/1995:00:00:08 -0400’,)]

### 1.23 HTTP request

Get the HTTP request sent to Web server \w+ {GET}

# Get the REST call with ” “
```rslt=(rdd.map(lambda line: re.search('"\w+\s+([^\s]+)\s+HTTP.*"',line)
.groups())
.take(3)) # Get the REST call
rslt```
[(‘/shuttle/missions/sts-68/news/sts-68-mcc-05.txt’,),
(‘/’,),
(‘/images/ksclogo-medium.gif’,)]

### 1.23Get HTTP response status

Get the HTTP response to the request

```rslt=(rdd.map(lambda line: re.search('"\s(\d{3})',line)
.groups())
.take(3)) #Get the status
rslt```
Out[6]: [(‘200’,), (‘304’,), (‘304’,)]

## 1.24 Get content size

Get the HTTP response in bytes

rslt=(rdd.map(lambda line: re.search(‘^.*\s(\d*)\$’,line)
```    .groups())
.take(3)) # Get the content size
rslt```
Out[7]: [(‘1839’,), (‘0’,), (‘0’,)]

## 1.24 Putting it all together

Now put all the individual pieces together into 1 big regular expression and assign to the groups

1. Host 2. Timestamp 3. Path 4. Status 5. Content_size
```rslt=(rdd.map(lambda line: re.search('^(\S+)((\s)(-))+\s(\[\S+ -\d{4}\])\s("\w+\s+([^\s]+)\s+HTTP.*")\s(\d{3}\s(\d*)\$)',line)
.groups())
.take(3))
rslt
```
[(‘in24.inetnebr.com’,
‘ -‘,
‘ ‘,
‘-‘,
‘[01/Aug/1995:00:00:01 -0400]’,
‘”GET /shuttle/missions/sts-68/news/sts-68-mcc-05.txt HTTP/1.0″‘,
‘/shuttle/missions/sts-68/news/sts-68-mcc-05.txt’,
‘200 1839’,
‘1839’),
(‘uplherc.upl.com’,
‘ -‘,
‘ ‘,
‘-‘,
‘[01/Aug/1995:00:00:07 -0400]’,
‘”GET / HTTP/1.0″‘,
‘/’,
‘304 0’,
‘0’),
(‘uplherc.upl.com’,
‘ -‘,
‘ ‘,
‘-‘,
‘[01/Aug/1995:00:00:08 -0400]’,
‘”GET /images/ksclogo-medium.gif HTTP/1.0″‘,
‘/images/ksclogo-medium.gif’,
‘304 0’,
‘0’)]

### 1.25 Add a log parsing function

import re
```def parse_log1(line):
match = re.search('^(\S+)((\s)(-))+\s(\[\S+ -\d{4}\])\s("\w+\s+([^\s]+)\s+HTTP.*")\s(\d{3}\s(\d*)\$)',line)
if match is None:
return(line,0)
else:
return(line,1)
```

### 1.26 Check for parsing failure

Check how many lines successfully parsed with the parsing function

```n_logs = rdd.count()
failed = rdd.map(lambda line: parse_log1(line)).filter(lambda line: line[1] == 0).count()
print('Out of a total of {} logs, {} failed to parse'.format(n_logs,failed))
# Get the failed records line[1] == 0
failed1=rdd.map(lambda line: parse_log1(line)).filter(lambda line: line[1]==0)
failed1.take(3)```
Out of a total of 3461613 logs, 38768 failed to parse
Out[10]:
[(‘gw1.att.com – – [01/Aug/1995:00:03:53 -0400] “GET /shuttle/missions/sts-73/news HTTP/1.0” 302 -‘,
0),
(‘js002.cc.utsunomiya-u.ac.jp – – [01/Aug/1995:00:07:33 -0400] “GET /shuttle/resources/orbiters/discovery.gif HTTP/1.0” 404 -‘,
0),
(‘pipe1.nyc.pipeline.com – – [01/Aug/1995:00:12:37 -0400] “GET /history/apollo/apollo-13/apollo-13-patch-small.gif” 200 12859’,
0)]

### 1.26 The above rule is not enough to parse the logs

It can be seen that the single rule only parses part of the logs and we cannot group the regex separately. There is an error “AttributeError: ‘NoneType’ object has no attribute ‘group'” which shows up

#rdd.map(lambda line: re.search(‘^(\S+)((\s)(-))+\s(\[\S+ -\d{4}\])\s(“\w+\s+([^\s]+)\s+HTTP.*”)\s(\d{3}\s(\d*)\$)’,line[0]).group()).take(4)

File “/databricks/spark/python/pyspark/util.py”, line 99, in wrapper
return f(*args, **kwargs)
File “<command-1348022240961444>”, line 1, in <lambda>
AttributeError: ‘NoneType’ object has no attribute ‘group’

### 1.27 Add rule for parsing failed records

One of the issues with the earlier rule is the content_size has “-” for some logs

import re
```def parse_failed(line):
match = re.search('^(\S+)((\s)(-))+\s(\[\S+ -\d{4}\])\s("\w+\s+([^\s]+)\s+HTTP.*")\s(\d{3}\s-\$)',line)
if match is None:
return(line,0)
else:
return(line,1)```

### 1.28 Parse records which fail

Parse the records that fails with the new rule

failed2=rdd.map(lambda line: parse_failed(line)).filter(lambda line: line[1]==1)
```failed2.take(5)
```
Out[13]:
[(‘gw1.att.com – – [01/Aug/1995:00:03:53 -0400] “GET /shuttle/missions/sts-73/news HTTP/1.0” 302 -‘,
1),
(‘js002.cc.utsunomiya-u.ac.jp – – [01/Aug/1995:00:07:33 -0400] “GET /shuttle/resources/orbiters/discovery.gif HTTP/1.0” 404 -‘,
1),
(‘tia1.eskimo.com – – [01/Aug/1995:00:28:41 -0400] “GET /pub/winvn/release.txt HTTP/1.0” 404 -‘,
1),
(‘itws.info.eng.niigata-u.ac.jp – – [01/Aug/1995:00:38:01 -0400] “GET /ksc.html/facts/about_ksc.html HTTP/1.0” 403 -‘,
1),
(‘grimnet23.idirect.com – – [01/Aug/1995:00:50:12 -0400] “GET /www/software/winvn/winvn.html HTTP/1.0” 404 -‘,
1)]

Add both rules for parsing the log.

Note it can be shown that even with both rules all the logs are not parse.Further rules may need to be added

import re
```def parse_log2(line):
# Parse logs with the rule below
match = re.search('^(\S+)((\s)(-))+\s(\[\S+ -\d{4}\])\s("\w+\s+([^\s]+)\s+HTTP.*")\s(\d{3})\s(\d*)\$',line)
# If match failed then use the rule below
if match is None:
match = re.search('^(\S+)((\s)(-))+\s(\[\S+ -\d{4}\])\s("\w+\s+([^\s]+)\s+HTTP.*")\s(\d{3}\s-\$)',line)
if match is None:
return (line, 0) # Return 0 for failure
else:
return (line, 1) # Return 1 for success
```

### 1.29 Group the different regex to groups for handling

def map2groups(line):
```    match = re.search('^(\S+)((\s)(-))+\s(\[\S+ -\d{4}\])\s("\w+\s+([^\s]+)\s+HTTP.*")\s(\d{3})\s(\d*)\$',line)
if match is None:
match = re.search('^(\S+)((\s)(-))+\s(\[\S+ -\d{4}\])\s("\w+\s+([^\s]+)\s+HTTP.*")\s(\d{3})\s(-)\$',line)
return(match.groups())
```

### 1.30 Parse the logs and map the groups

parsed_rdd = rdd.map(lambda line: parse_log2(line)).filter(lambda line: line[1] == 1).map(lambda line : line[0])

parsed_rdd2 = parsed_rdd.map(lambda line: map2groups(line))

## 2. Parse Web server logs with Pyspark

### 2.1Read data into a Pyspark dataframe

import os
```logs_file_path="/FileStore/tables/" + os.path.join('NASA_access_log_*.gz')
from pyspark.sql.functions import split, regexp_extract
#base_df.show(truncate=False)
from pyspark.sql.functions import split, regexp_extract
split_df = base_df.select(regexp_extract('value', r'^([^\s]+\s)', 1).alias('host'),
regexp_extract('value', r'^.*\[(\d\d\/\w{3}\/\d{4}:\d{2}:\d{2}:\d{2} -\d{4})]', 1).alias('timestamp'),
regexp_extract('value', r'^.*"\w+\s+([^\s]+)\s+HTTP.*"', 1).alias('path'),
regexp_extract('value', r'^.*"\s+([^\s]+)', 1).cast('integer').alias('status'),
regexp_extract('value', r'^.*\s+(\d+)\$', 1).cast('integer').alias('content_size'))
split_df.show(5,truncate=False)
```
+———————+————————–+———————————————–+——+————+
|host |timestamp |path |status|content_size|
+———————+————————–+———————————————–+——+————+
|199.72.81.55 |01/Jul/1995:00:00:01 -0400|/history/apollo/ |200 |6245 |
|unicomp6.unicomp.net |01/Jul/1995:00:00:06 -0400|/shuttle/countdown/ |200 |3985 |
|199.120.110.21 |01/Jul/1995:00:00:09 -0400|/shuttle/missions/sts-73/mission-sts-73.html |200 |4085 |
|burger.letters.com |01/Jul/1995:00:00:11 -0400|/shuttle/countdown/liftoff.html |304 |0 |
|199.120.110.21 |01/Jul/1995:00:00:11 -0400|/shuttle/missions/sts-73/sts-73-patch-small.gif|200 |4179 |
+———————+————————–+———————————————–+——+————+
only showing top 5 rows

### 2.2 Check data

```                              split_df['timestamp'].isNull() |
split_df['path'].isNull() |
split_df['status'].isNull() |
split_df['content_size'].isNull())
Out[20]: 33905

### 2.3Check no of rows which do not have digits

We have already seen that the content_type field has ‘-‘ instead of digits in RDDs

```bad_content_size_df.count()
```
Out[21]: 33905

To identify the rows that are bad, concatenate ‘*’ to the content_size field where the field does not have digits. It can be seen that the content_size has ‘-‘ instead of a valid number

from pyspark.sql.functions import lit, concat
```bad_content_size_df.select(concat(bad_content_size_df['value'], lit('*'))).show(4,truncate=False)
```
+—————————————————————————————————————————————————+
|concat(value, *) |
+—————————————————————————————————————————————————+
|dd15-062.compuserve.com – – [01/Jul/1995:00:01:12 -0400] “GET /news/sci.space.shuttle/archive/sci-space-shuttle-22-apr-1995-40.txt HTTP/1.0” 404 -*|
|dynip42.efn.org – – [01/Jul/1995:00:02:14 -0400] “GET /software HTTP/1.0” 302 -* |
|ix-or10-06.ix.netcom.com – – [01/Jul/1995:00:02:40 -0400] “GET /software/winvn HTTP/1.0” 302 -* |
|ix-or10-06.ix.netcom.com – – [01/Jul/1995:00:03:24 -0400] “GET /software HTTP/1.0” 302 -* |
+—————————————————————————————————————————————————+

### 2.5 Fill NAs with 0s

# Replace all null content_size values with 0.

cleaned_df = split_df.na.fill({‘content_size’: 0})

## 3. Webserver  logs parsing with SparkR

library(SparkR)
```library(stringr)
file_location = "/FileStore/tables/NASA_access_log_Jul95.gz"
file_location = "/FileStore/tables/NASA_access_log_Aug95.gz"

# Initiate a SparkR session
sparkR.session()
sc <- sparkR.session()
sqlContext <- sparkRSQL.init(sc)

#df=SparkR::select(df, "value")
#m=regexp_extract(df\$value,'\\\\S+',1)

a=df %>%
withColumn('host', regexp_extract(df\$value, '^(\\S+)', 1)) %>%
withColumn('timestamp', regexp_extract(df\$value, "((\\S+ -\\d{4}))", 2)) %>%
withColumn('path', regexp_extract(df\$value, '(\\"\\w+\\s+([^\\s]+)\\s+HTTP.*")', 2))  %>%
withColumn('status', regexp_extract(df\$value, '(^.*"\\s+([^\\s]+))', 2)) %>%
withColumn('content_size', regexp_extract(df\$value, '(^.*\\s+(\\d+)\$)', 2))
#b=a%>% select(host,timestamp,path,status,content_type)
```

1 199.72.81.55 – – [01/Jul/1995:00:00:01 -0400] “GET /history/apollo/ HTTP/1.0” 200 6245
2 unicomp6.unicomp.net – – [01/Jul/1995:00:00:06 -0400] “GET /shuttle/countdown/ HTTP/1.0” 200 3985
3 199.120.110.21 – – [01/Jul/1995:00:00:09 -0400] “GET /shuttle/missions/sts-73/mission-sts-73.html HTTP/1.0” 200 4085
4 burger.letters.com – – [01/Jul/1995:00:00:11 -0400] “GET /shuttle/countdown/liftoff.html HTTP/1.0” 304 0
5 199.120.110.21 – – [01/Jul/1995:00:00:11 -0400] “GET /shuttle/missions/sts-73/sts-73-patch-small.gif HTTP/1.0” 200 4179
6 burger.letters.com – – [01/Jul/1995:00:00:12 -0400] “GET /images/NASA-logosmall.gif HTTP/1.0” 304 0
7 burger.letters.com – – [01/Jul/1995:00:00:12 -0400] “GET /shuttle/countdown/video/livevideo.gif HTTP/1.0” 200 0
8 205.212.115.106 – – [01/Jul/1995:00:00:12 -0400] “GET /shuttle/countdown/countdown.html HTTP/1.0” 200 3985
9 d104.aa.net – – [01/Jul/1995:00:00:13 -0400] “GET /shuttle/countdown/ HTTP/1.0” 200 3985
10 129.94.144.152 – – [01/Jul/1995:00:00:13 -0400] “GET / HTTP/1.0” 200 7074
host timestamp
1 199.72.81.55 [01/Jul/1995:00:00:01 -0400
2 unicomp6.unicomp.net [01/Jul/1995:00:00:06 -0400
3 199.120.110.21 [01/Jul/1995:00:00:09 -0400
4 burger.letters.com [01/Jul/1995:00:00:11 -0400
5 199.120.110.21 [01/Jul/1995:00:00:11 -0400
6 burger.letters.com [01/Jul/1995:00:00:12 -0400
7 burger.letters.com [01/Jul/1995:00:00:12 -0400
8 205.212.115.106 [01/Jul/1995:00:00:12 -0400
9 d104.aa.net [01/Jul/1995:00:00:13 -0400
10 129.94.144.152 [01/Jul/1995:00:00:13 -0400
path status content_size
1 /history/apollo/ 200 6245
2 /shuttle/countdown/ 200 3985
3 /shuttle/missions/sts-73/mission-sts-73.html 200 4085
4 /shuttle/countdown/liftoff.html 304 0
5 /shuttle/missions/sts-73/sts-73-patch-small.gif 200 4179
6 /images/NASA-logosmall.gif 304 0
7 /shuttle/countdown/video/livevideo.gif 200 0
8 /shuttle/countdown/countdown.html 200 3985
9 /shuttle/countdown/ 200 3985
10 / 200 7074

## 4 Webserver logs parsing with SparklyR

```install.packages("sparklyr")
library(sparklyr)
library(dplyr)
library(stringr)
#sc <- spark_connect(master = "local", version = "2.1.0")
sc <- spark_connect(method = "databricks")
sdf <-spark_read_text(sc, name="df", path = "/FileStore/tables/NASA_access_log*.gz")
sdf
```
Installing package into ‘/databricks/spark/R/lib’
```# Source: spark [?? x 1]
line

1 "199.72.81.55 - - [01/Jul/1995:00:00:01 -0400] \"GET /history/apollo/ HTTP/1…
2 "unicomp6.unicomp.net - - [01/Jul/1995:00:00:06 -0400] \"GET /shuttle/countd…
3 "199.120.110.21 - - [01/Jul/1995:00:00:09 -0400] \"GET /shuttle/missions/sts…
4 "burger.letters.com - - [01/Jul/1995:00:00:11 -0400] \"GET /shuttle/countdow…
5 "199.120.110.21 - - [01/Jul/1995:00:00:11 -0400] \"GET /shuttle/missions/sts…
6 "burger.letters.com - - [01/Jul/1995:00:00:12 -0400] \"GET /images/NASA-logo…
7 "burger.letters.com - - [01/Jul/1995:00:00:12 -0400] \"GET /shuttle/countdow…
8 "205.212.115.106 - - [01/Jul/1995:00:00:12 -0400] \"GET /shuttle/countdown/c…
9 "d104.aa.net - - [01/Jul/1995:00:00:13 -0400] \"GET /shuttle/countdown/ HTTP…
10 "129.94.144.152 - - [01/Jul/1995:00:00:13 -0400] \"GET / HTTP/1.0\" 200 7074"
# … with more rows```
#install.packages(“sparklyr”)
```library(sparklyr)
library(dplyr)
library(stringr)
#sc <- spark_connect(master = "local", version = "2.1.0")
sc <- spark_connect(method = "databricks")
sdf <-spark_read_text(sc, name="df", path = "/FileStore/tables/NASA_access_log*.gz")
sdf <- sdf %>% mutate(host = regexp_extract(line, '^(\\\\S+)',1)) %>%
mutate(timestamp = regexp_extract(line, '((\\\\S+ -\\\\d{4}))',2)) %>%
mutate(path = regexp_extract(line, '(\\\\"\\\\w+\\\\s+([^\\\\s]+)\\\\s+HTTP.*")',2)) %>%
mutate(status = regexp_extract(line, '(^.*"\\\\s+([^\\\\s]+))',2)) %>%
mutate(content_size = regexp_extract(line, '(^.*\\\\s+(\\\\d+)\$)',2))

```

## 5 Hosts

### 5.1  RDD

#### 5.11 Parse and map to hosts to groups

parsed_rdd = rdd.map(lambda line: parse_log2(line)).filter(lambda line: line[1] == 1).map(lambda line : line[0])
```parsed_rdd2 = parsed_rdd.map(lambda line: map2groups(line))

# Create tuples of (host,1) and apply reduceByKey() and order by descending
rslt=(parsed_rdd2.map(lambda x😦x[0],1))
.reduceByKey(lambda a,b:a+b)
.takeOrdered(10, lambda x: -x[1]))
rslt
```
Out[18]:
[(‘piweba3y.prodigy.com’, 21988),
(‘piweba4y.prodigy.com’, 16437),
(‘piweba1y.prodigy.com’, 12825),
(‘edams.ksc.nasa.gov’, 11962),
(‘163.206.89.4’, 9697),
(‘news.ti.com’, 8161),
(‘www-d1.proxy.aol.com’, 8047),
(‘alyssa.prodigy.com’, 8037),
(‘siltb10.orl.mmc.com’, 7573),
(‘www-a2.proxy.aol.com’, 7516)]

#### 5.12Plot counts of hosts

import seaborn as sns

import pandas as pd import matplotlib.pyplot as plt df=pd.DataFrame(rslt,columns=[‘host’,‘count’]) sns.barplot(x=‘host’,y=‘count’,data=df) plt.subplots_adjust(bottom=0.6, right=0.8, top=0.9) plt.xticks(rotation=“vertical”,fontsize=8) display()

### 5.2 PySpark

#### 5.21 Compute counts of hosts

df= (cleaned_df
```     .groupBy('host')
.count()
.orderBy('count',ascending=False))
df.show(5)
```
+——————–+—–+
| host|count|
+——————–+—–+
|piweba3y.prodigy….|21988|
|piweba4y.prodigy….|16437|
|piweba1y.prodigy….|12825|
| edams.ksc.nasa.gov |11964|
| 163.206.89.4 | 9697|
+——————–+—–+
only showing top 5 rows

### 5.22 Plot count of hosts

import matplotlib.pyplot as plt
```import pandas as pd
import seaborn as sns
df1=df.toPandas()
df2.count()
sns.barplot(x='host',y='count',data=df2)
plt.xlabel("Hosts")
plt.ylabel('Count')
plt.xticks(rotation="vertical",fontsize=10)
display()```

### 5.31 Compute count of hosts

c <- SparkR::select(a,a\$host)
```df=SparkR::summarize(SparkR::groupBy(c, a\$host), noHosts = count(a\$host))
```
host noHosts
```1 piweba3y.prodigy.com   17572
2 piweba4y.prodigy.com   11591
3 piweba1y.prodigy.com    9868
4   alyssa.prodigy.com    7852
5  siltb10.orl.mmc.com    7573
6 piweba2y.prodigy.com    5922```

### 5.32 Plot count of hosts

library(ggplot2)
```p <-ggplot(data=df1, aes(x=host, y=noHosts,fill=host)) +   geom_bar(stat="identity") + theme(axis.text.x = element_text(angle = 90, hjust = 1)) + xlab('Host') + ylab('Count')
p

```

### 5.4 SparklyR

#### 5.41 Compute count of Hosts

df <- sdf %>% select(host,timestamp,path,status,content_size)
```df1 <- df %>% select(host) %>% group_by(host) %>% summarise(noHosts=n()) %>% arrange(desc(noHosts))
```

### 5.42 Plot count of hosts

library(ggplot2)

p <-ggplot(data=df2, aes(x=host, y=noHosts,fill=host)) + geom_bar(stat=identity”)+ theme(axis.text.x = element_text(angle = 90, hjust = 1)) + xlab(Host’) + ylab(Count’)

p

## 6 Paths

### 6.11 Parse and map to hosts to groups

parsed_rdd = rdd.map(lambda line: parse_log2(line)).filter(lambda line: line[1] == 1).map(lambda line : line[0])
```parsed_rdd2 = parsed_rdd.map(lambda line: map2groups(line))
rslt=(parsed_rdd2.map(lambda x😦x[5],1))
.reduceByKey(lambda a,b:a+b)
.takeOrdered(10, lambda x: -x[1]))
rslt
```
[(‘”GET /images/NASA-logosmall.gif HTTP/1.0″‘, 207520),
(‘”GET /images/KSC-logosmall.gif HTTP/1.0″‘, 164487),
(‘”GET /images/MOSAIC-logosmall.gif HTTP/1.0″‘, 126933),
(‘”GET /images/USA-logosmall.gif HTTP/1.0″‘, 126108),
(‘”GET /images/WORLD-logosmall.gif HTTP/1.0″‘, 124972),
(‘”GET /images/ksclogo-medium.gif HTTP/1.0″‘, 120704),
(‘”GET /ksc.html HTTP/1.0″‘, 83209),
(‘”GET /images/launch-logo.gif HTTP/1.0″‘, 75839),
(‘”GET /history/apollo/images/apollo-logo1.gif HTTP/1.0″‘, 68759),
(‘”GET /shuttle/countdown/ HTTP/1.0″‘, 64467)]

#### 6.12 Plot counts of HTTP Requests

import seaborn as sns

df=pd.DataFrame(rslt,columns=[‘path’,‘count’]) sns.barplot(x=‘path’,y=‘count’,data=df) plt.subplots_adjust(bottom=0.7, right=0.8, top=0.9) plt.xticks(rotation=“vertical”,fontsize=8)

display()

### 6.2 Pyspark

#### 6.21 Compute count of HTTP Requests

```df= (cleaned_df
.groupBy('path')
.count()
.orderBy('count',ascending=False))
df.show(5)
```
Out[20]:
+——————–+——+
| path| count|
+——————–+——+
|/images/NASA-logo…|208362|
|/images/KSC-logos…|164813|
|/images/MOSAIC-lo…|127656|
|/images/USA-logos…|126820|
|/images/WORLD-log…|125676|
+——————–+——+
only showing top 5 rows

#### 6.22 Plot count of HTTP Requests

import matplotlib.pyplot as plt

import pandas as pd import seaborn as sns df1=df.toPandas() df2 = df1.head(10) df2.count() sns.barplot(x=‘path’,y=‘count’,data=df2)

plt.subplots_adjust(bottom=0.7, right=0.8, top=0.9) plt.xlabel(“HTTP Requests”) plt.ylabel(‘Count’) plt.xticks(rotation=90,fontsize=8)

display()

### 6.3 SparkR

#### 6.31Compute count of HTTP requests

library(SparkR)
```c <- SparkR::select(a,a\$path)
df=SparkR::summarize(SparkR::groupBy(c, a\$path), numRequest = count(a\$path))
```

#### 3.14 Plot count of HTTP Requests

library(ggplot2)
```p <-ggplot(data=df1, aes(x=path, y=numRequest,fill=path)) +   geom_bar(stat="identity") + theme(axis.text.x = element_text(angle = 90, hjust = 1))+ xlab('Path') + ylab('Count')
p

```

### 6.4 SparklyR

#### 6.41 Compute count of paths

```df <- sdf %>% select(host,timestamp,path,status,content_size)
df1 <- df %>% select(path) %>% group_by(path) %>% summarise(noPaths=n()) %>% arrange(desc(noPaths))
df2
```
# Source: spark [?? x 2]
```# Ordered by: desc(noPaths)
path                                    noPaths

1 /images/NASA-logosmall.gif               208362
2 /images/KSC-logosmall.gif                164813
3 /images/MOSAIC-logosmall.gif             127656
4 /images/USA-logosmall.gif                126820
5 /images/WORLD-logosmall.gif              125676
6 /images/ksclogo-medium.gif               121286
7 /ksc.html                                 83685
8 /images/launch-logo.gif                   75960
9 /history/apollo/images/apollo-logo1.gif   68858
10 /shuttle/countdown/                       64695```

#### 6.42 Plot count of Paths

library(ggplot2)
```p <-ggplot(data=df2, aes(x=path, y=noPaths,fill=path)) +   geom_bar(stat="identity")+ theme(axis.text.x = element_text(angle = 90, hjust = 1)) + xlab('Path') + ylab('Count')
p

```

### 7.1 RDD

#### 7.11 Compute count of HTTP Status

parsed_rdd = rdd.map(lambda line: parse_log2(line)).filter(lambda line: line[1] == 1).map(lambda line : line[0])

```parsed_rdd2 = parsed_rdd.map(lambda line: map2groups(line))
rslt=(parsed_rdd2.map(lambda x😦x[7],1))
.reduceByKey(lambda a,b:a+b)
.takeOrdered(10, lambda x: -x[1]))
rslt
```
Out[22]:
[(‘200’, 3095682),
(‘304’, 266764),
(‘302’, 72970),
(‘404’, 20625),
(‘403’, 225),
(‘500’, 65),
(‘501’, 41)]

#### 1.37 Plot counts of HTTP response status’

import seaborn as sns

df=pd.DataFrame(rslt,columns=[‘status’,‘count’]) sns.barplot(x=‘status’,y=‘count’,data=df) plt.subplots_adjust(bottom=0.4, right=0.8, top=0.9) plt.xticks(rotation=“vertical”,fontsize=8)

display()

### 7.2 Pyspark

#### 7.21 Compute count of HTTP status

status_count=(cleaned_df
```                .groupBy('status')
.count()
.orderBy('count',ascending=False))
status_count.show()```
+——+——-+
|status| count|
+——+——-+
| 200|3100522|
| 304| 266773|
| 302| 73070|
| 404| 20901|
| 403| 225|
| 500| 65|
| 501| 41|
| 400| 15|
| null| 1|

### 7.22 Plot count of HTTP status

Plot the HTTP return status vs the counts

df1=status_count.toPandas()

df2 = df1.head(10) df2.count() sns.barplot(x=‘status’,y=‘count’,data=df2) plt.subplots_adjust(bottom=0.5, right=0.8, top=0.9) plt.xlabel(“HTTP Status”) plt.ylabel(‘Count’) plt.xticks(rotation=“vertical”,fontsize=10) display()

### 7.3 SparkR

#### 7.31 Compute count of HTTP Response status

library(SparkR)
```c <- SparkR::select(a,a\$status)
df=SparkR::summarize(SparkR::groupBy(c, a\$status), numStatus = count(a\$status))

## 3.16 Plot count of HTTP Response status

library(ggplot2)
```p <-ggplot(data=df1, aes(x=status, y=numStatus,fill=status)) +   geom_bar(stat="identity") + theme(axis.text.x = element_text(angle = 90, hjust = 1)) + xlab('Status') + ylab('Count')
p

```

#### 7.41 Compute count of status

```df <- sdf %>% select(host,timestamp,path,status,content_size)
df1 <- df %>% select(status) %>% group_by(status) %>% summarise(noStatus=n()) %>% arrange(desc(noStatus))
df2
```
# Source: spark [?? x 2]
```# Ordered by: desc(noStatus)
status noStatus

1 200     3100522
2 304      266773
3 302       73070
4 404       20901
5 403         225
6 500          65
7 501          41
8 400          15
9 ""            1```

#### 7.42 Plot count of status

library(ggplot2)

p <-ggplot(data=df2, aes(x=status, y=noStatus,fill=status)) + geom_bar(stat=identity”)+ theme(axis.text.x = element_text(angle = 90, hjust = 1)) + xlab(Status’) + ylab(Count’) p

### 8.1 RDD

#### 8.12 Compute count of content size

```parsed_rdd = rdd.map(lambda line: parse_log2(line)).filter(lambda line: line[1] == 1).map(lambda line : line[0])
parsed_rdd2 = parsed_rdd.map(lambda line: map2groups(line))
rslt=(parsed_rdd2.map(lambda x😦x[8],1))
.reduceByKey(lambda a,b:a+b)
.takeOrdered(10, lambda x: -x[1]))
rslt
```
Out[24]:
[(‘0’, 280017),
(‘786’, 167281),
(‘1204’, 140505),
(‘363’, 111575),
(‘234’, 110824),
(‘669’, 110056),
(‘5866’, 107079),
(‘1713’, 66904),
(‘1173’, 63336),
(‘3635’, 55528)]

#### 8.21 Plot content size

import seaborn as sns

df=pd.DataFrame(rslt,columns=[‘content_size’,‘count’]) sns.barplot(x=‘content_size’,y=‘count’,data=df) plt.subplots_adjust(bottom=0.4, right=0.8, top=0.9) plt.xticks(rotation=“vertical”,fontsize=8) display()

### 8.2 Pyspark

#### 8.21 Compute count of content_size

size_counts=(cleaned_df
```                .groupBy('content_size')
.count()
.orderBy('count',ascending=False))
size_counts.show(10)
+------------+------+
|content_size| count|
+------------+------+
|           0|313932|
|         786|167709|
|        1204|140668|
|         363|111835|
|         234|111086|
|         669|110313|
|        5866|107373|
|        1713| 66953|
|        1173| 63378|
|        3635| 55579|
+------------+------+
only showing top 10 rows```

#### 8.22 Plot counts of content size

Plot the path access versus the counts

df1=size_counts.toPandas()

df2 = df1.head(10) df2.count() sns.barplot(x=‘content_size’,y=‘count’,data=df2) plt.subplots_adjust(bottom=0.5, right=0.8, top=0.9) plt.xlabel(“content_size”) plt.ylabel(‘Count’) plt.xticks(rotation=“vertical”,fontsize=10) display()

### 8.31 Compute count of content size

```library(SparkR)
c <- SparkR::select(a,a\$content_size)
df=SparkR::summarize(SparkR::groupBy(c, a\$content_size), numContentSize = count(a\$content_size))
df1
```
content_size numContentSize
```1        28426           1414
2        78382            293
3        60053              4
4        36067              2
5        13282            236
6        41785            174```
8.32 Plot count of content sizes
library(ggplot2)

p <-ggplot(data=df1, aes(x=content_size, y=numContentSize,fill=content_size)) + geom_bar(stat=identity”) + theme(axis.text.x = element_text(angle = 90, hjust = 1)) + xlab(Content Size’) + ylab(Count’)

p

### 8.4 SparklyR

#### 8.41Compute count of content_size

df <- sdf %>% select(host,timestamp,path,status,content_size)
```df1 <- df %>% select(content_size) %>% group_by(content_size) %>% summarise(noContentSize=n()) %>% arrange(desc(noContentSize))
df2
```
# Source: spark [?? x 2]
```# Ordered by: desc(noContentSize)
content_size noContentSize

1 0                   280027
2 786                 167709
3 1204                140668
4 363                 111835
5 234                 111086
6 669                 110313
7 5866                107373
8 1713                 66953
9 1173                 63378
10 3635                 55579```

#### 8.42 Plot count of content_size

library(ggplot2)
```p <-ggplot(data=df2, aes(x=content_size, y=noContentSize,fill=content_size)) +   geom_bar(stat="identity")+ theme(axis.text.x = element_text(angle = 90, hjust = 1)) + xlab('Content size') + ylab('Count')
p

```

Conclusion: I spent many,many hours struggling with Regex and getting RDDs,Pyspark to work. Also had to spend a lot of time trying to work out the syntax for SparkR and SparklyR for parsing. After you parse the logs plotting and analysis is a piece of cake! This is definitely worth a try!

Watch this space!!

To see all posts click Index of posts

# Revisiting World Bank data analysis with WDI and gVisMotionChart

Note: I had written a post about 3 years back on World Bank Data Analysis using World Development Indicators (WDI) & gVisMotionCharts. But the motion charts stopped working  some time ago. I have always been wanting to fix this and I now got to actually doing it. The issue was 2 of the WDI indicators had changed. After I fixed this I was able to host the generated motion chart using github.io pages. Please make sure that you enable flash player if you open the motion charts with Google Chrome. You may also have to enable flash if using Firefox, IE etc

Please check out the 2 motions charts with World Bank data

If you are using Chrome please enable (Allow)  ‘flash player’ by

a) Clicking on the lock sign in the URL as shown and if Flash is shown set it to ‘Allow’ and press ‘Reload’

b) Or click  the lock and then click on site settings and set Flash to ‘Allow’ as below and then press ‘Reload’

## Introduction

Recently I was surfing the web, when I came across a real cool post New R package to access World Bank data, by Markus Gesmann on using googleVis and motion charts with World Bank Data. The post also introduced me to Hans Rosling, Professor of Sweden’s Karolinska Institute. Hans Rosling, the creator of the famous Gapminder chart, the “Heath and Wealth of Nations” displays global trends through animated charts (A must see!!!). As they say, in Hans Rosling’s hands, data dances and sings. Take a look at  his Ted talks for e.g. Hans Rosling:New insights on poverty. Prof Rosling developed the breakthrough software behind the visualizations, in the Gapminder. The free software, which can be loaded with any data – was purchased by Google in March 2007.

In this post, I recreate some of the Gapminder charts with the help of R packages WDI and googleVis. The WDI  package of  Vincent Arel-Bundock, provides a set of really useful functions to get to data based on the World Bank Data indicators.  googleVis provides motion charts with which you can animate the data.

You can clone/download the code from Github at worldBankAnalysis which is in the form of an Rmd file.

``````library(WDI)
library(ggplot2)
library(plyr)``````

## 1.Get the data from 1960 to 2019 for the following

1. Population – SP.POP.TOTL
2. GDP in US \$ – NY.GDP.MKTP.CD
3. Life Expectancy at birth (Years) – SP.DYN.LE00.IN
4. GDP Per capita income – NY.GDP.PCAP.PP.CD
5. Fertility rate (Births per woman) – SP.DYN.TFRT.IN
6. Poverty headcount ratio – SI.POV.NAHC
``````# World population total
population = WDI(indicator='SP.POP.TOTL', country="all",start=1960, end=2019)
# GDP in US \$
gdp= WDI(indicator='NY.GDP.MKTP.CD', country="all",start=1960, end=2019)
# Life expectancy at birth (Years)
lifeExpectancy= WDI(indicator='SP.DYN.LE00.IN', country="all",start=1960, end=2019)
# GDP Per capita
income = WDI(indicator='NY.GDP.PCAP.PP.CD', country="all",start=1960, end=2019)
# Fertility rate (births per woman)
fertility = WDI(indicator='SP.DYN.TFRT.IN', country="all",start=1960, end=2019)
poverty= WDI(indicator='SI.POV.NAHC', country="all",start=1960, end=2019)``````

## 2.Rename the columns

``````names(population)[3]="Total population"
names(lifeExpectancy)[3]="Life Expectancy (Years)"
names(gdp)[3]="GDP (US\$)"
names(income)[3]="GDP per capita income"
names(fertility)[3]="Fertility (Births per woman)"

## 3.Join the data frames

Join the individual data frames to one large wide data frame with all the indicators for the countries``` j1 <- join(population, gdp) ```

```j2 <- join(j1,lifeExpectancy) ```

```j3 <- join(j2,income) ```

```j4 <- join(j3,poverty) ```

```wbData <- join(j4,fertility) ```

## 4.Use WDI_data

Use WDI_data to get the list of indicators and the countries. Join the countries and region

``````#This returns  list of 2 matrixes
wdi_data =WDI_data
# The 1st matrix is the list is the set of all World Bank Indicators
indicators=wdi_data[[1]]
# The 2nd  matrix gives the set of countries and regions
countries=wdi_data[[2]]
df = as.data.frame(countries)
aa <- df\$region != "Aggregates"
# Remove the aggregates
countries_df <- df[aa,]
# Subset from the development data only those corresponding to the countries
bb = subset(wbData, country %in% countries_df\$country)
cc = join(bb,countries_df)``````
``````dd = complete.cases(cc)
developmentDF = cc[dd,]``````

## 5.Create and display the motion chart

``````gg<- gvisMotionChart(cc,
idvar = "country",
timevar = "year",
xvar = "GDP",
yvar = "Life Expectancy",
sizevar ="Population",
colorvar = "region")
plot(gg)
cat(gg\$html\$chart, file="chart1.html")
``````

Note: Unfortunately it is not possible to embed the motion chart in WordPress. It is has to hosted on a server as a Webpage. After exploring several possibilities I came up with the following process to display the animation graph. The plot is saved as a html file using ‘cat’ as shown above. The WorldBank_chart1.html page is then hosted as a Github page (gh-page) on Github.

Here is the ggvisMotionChart

Do give  World Bank Motion Chart1  a spin.  Here is how the Motion Chart has to be used

You can select Life Expectancy, Population, Fertility etc by clicking the black arrows. The blue arrow shows the ‘play’ button to set animate the motion chart. You can also select the countries and change the size of the circles. Do give it a try. Here are some quick analysis by playing around with the motion charts with different parameters chosen

The set of charts below are screenshots captured by running the motion chart World Bank Motion Chart1

a. Life Expectancy vs Fertility chart

This chart is used by Hans Rosling in his Ted talk. The left chart shows low life expectancy and high fertility rate for several sub Saharan and East Asia Pacific countries in the early 1960’s. Today the fertility has dropped and the life expectancy has increased overall. However the sub Saharan countries still have a high fertility rate

b. Population vs GDP

The chart below shows that GDP of India and China have the same GDP from 1973-1994 with US and Japan well ahead.

From 1998- 2014 China really pulls away from India and Japan as seen below

c. Per capita income vs Life Expectancy

In the 1990’s the per capita income and life expectancy of the sub -saharan countries are low (42-50). Japan and US have a good life expectancy in 1990’s. In 2014 the per capita income of the sub-saharan countries are still low though the life expectancy has marginally improved.

In the early 1990’s China had a higher poverty head count ratio than India. By 2004 China had this all figured out and the poverty head count ratio drops significantly. This can also be seen in the chart below.

In the chart above China shows a drastic reduction in poverty headcount ratio vs India. Strangely Zambia shows an increase in the poverty head count ratio.

## 6.Get the data for the 2nd set of indicators

1. Total population  – SP.POP.TOTL
2. GDP in US\$ – NY.GDP.MKTP.CD
4. Electricity consumption KWh per capita -EG.USE.ELEC.KH.PC
5. CO2 emissions -EN.ATM.CO2E.KT
6. Basic Sanitation Access – SH.STA.BASS.ZS
``````# World population
population = WDI(indicator='SP.POP.TOTL', country="all",start=1960, end=2016)
# GDP in US \$
gdp= WDI(indicator='NY.GDP.MKTP.CD', country="all",start=1960, end=2016)
elecAccess= WDI(indicator='EG.ELC.ACCS.ZS', country="all",start=1960, end=2016)
# Electric power consumption Kwh per capita
elecConsumption= WDI(indicator='EG.USE.ELEC.KH.PC', country="all",start=1960, end=2016)
#CO2 emissions
co2Emissions= WDI(indicator='EN.ATM.CO2E.KT', country="all",start=1960, end=2016)
sanitationAccess= WDI(indicator='SH.STA.ACSN', country="all",start=1960, end=2016)``````

## 7.Rename the columns

``````names(population)[3]="Total population"
names(gdp)[3]="GDP US(\$)"
names(elecConsumption)[3]="Electric power consumption (KWH per capita)"
names(co2Emissions)[3]="CO2 emisions"

## 8.Join the individual data frames

Join the individual data frames to one large wide data frame with all the indicators for the countries

``````
j1 <- join(population, gdp)
j2 <- join(j1,elecAccess)
j3 <- join(j2,elecConsumption)
j4 <- join(j3,co2Emissions)
wbData1 <- join(j3,sanitationAccess)
``````

## 9.Use WDI_data

Use WDI_data to get the list of indicators and the countries. Join the countries and region

``````#This returns  list of 2 matrixes
wdi_data =WDI_data
# The 1st matrix is the list is the set of all World Bank Indicators
indicators=wdi_data[[1]]
# The 2nd  matrix gives the set of countries and regions
countries=wdi_data[[2]]
df = as.data.frame(countries)
aa <- df\$region != "Aggregates"
# Remove the aggregates
countries_df <- df[aa,]
# Subset from the development data only those corresponding to the countries
ee = subset(wbData1, country %in% countries_df\$country)
ff = join(ee,countries_df)``````
``## Joining by: iso2c, country``

## 10.Create and display the motion chart

``````gg1<- gvisMotionChart(ff,
idvar = "country",
timevar = "year",
xvar = "GDP",
sizevar ="Population",
colorvar = "region")
plot(gg1)
cat(gg1\$html\$chart, file="chart2.html")
``````

This is World Bank Motion Chart2  which has a different set of parameters like Access to Energy, CO2 emissions etc

The set of charts below are screenshots of the motion chart World Bank Motion Chart 2

The above chart shows that in China 100% population have access to electricity. India has made decent progress from 50% in 1990 to 79% in 2012. However Pakistan seems to have been much better in providing access to electricity. Pakistan moved from 59% to close 98% access to electricity

b. Power consumption vs population

The above chart shows the Power consumption vs Population. China and India have proportionally much lower consumption that Norway, US, Canada

c. CO2 emissions vs Population

In 1963 the CO2 emissions were fairly low and about comparable for all countries. US, India have shown a steady increase while China shows a steep increase. Interestingly UK shows a drop in CO2 emissions

India shows an improvement but it has a long way to go with only 40% of population with access to sanitation. China has made much better strides with 80% having access to sanitation in 2015. Strangely Nigeria shows a drop in sanitation by almost about 20% of population.

The code is available at Github at worldBankAnalysis

Conclusion: So there you have it. I have shown some screenshots of some sample parameters of the World indicators. Please try to play around with World Bank Motion Chart1 & World Bank Motion Chart 2  with your own set of parameters and countries.  You can also create your own motion chart from the 100s of WDI indicators avaialable at  World Bank Data indicator.

To see all posts Index of posts

# Analyzing performances of cricketers using cricketr template

This post includes a template which you can use for analyzing the performances of cricketers, both batsmen and bowlers in Test, ODI and Twenty 20 cricket using my R package cricketr. To see actual usage of functions in the R package cricketr see Introducing cricketr! : An R package to analyze performances of cricketers.

The ‘cricketr’ package uses the statistics info available in ESPN Cricinfo Statsguru. The current version of this package supports all formats of the game including Test, ODI and Twenty20 versions.

You should be able to install the package from GitHub and use the many functions available in the package. Please mindful of the ESPN Cricinfo Terms of Use

Take a look at my short video tutorial on my R package cricketr on Youtube – R package cricketr – A short tutorial

Do check out my interactive Shiny app implementation using the cricketr package – Sixer – R package cricketr’s new Shiny avatar

Important note 1: The latest release of ‘cricketr’ now includes the ability to analyze performances of teams now!!  See Cricketr adds team analytics to its repertoire!!!

Important note 2 : Cricketr can now do a more fine-grained analysis of players, see Cricketr learns new tricks : Performs fine-grained analysis of players

Important note 3: Do check out the python avatar of cricketr, ‘cricpy’ in my post ‘Introducing cricpy:A python package to analyze performances of cricketers

# The cricketr package

The cricketr package has several functions that perform several different analyses on both batsman and bowlers. The package has function that plot percentage frequency runs or wickets, runs likelihood for a batsman, relative run/strike rates of batsman and relative performance/economy rate for bowlers are available.

Other interesting functions include batting performance moving average, forecast and a function to check whether the batsmans in in-form or out-of-form.

The data for a particular player can be obtained with the getPlayerData() function. To do you will need to go to ESPN CricInfo Player and type in the name of the player for e.g Ricky Ponting, Sachin Tendulkar etc. This will bring up a page which have the profile number for the player e.g. for Sachin Tendulkar this would be http://www.espncricinfo.com/india/content/player/35320.html. Hence, Sachin’s profile is 35320. This can be used to get the data for Tendulkar as shown below

The cricketr package is now available from CRAN!!! You should be able to install directly with

### 1. Install the cricketr package

``````if (!require("cricketr")){
install.packages("cricketr",lib = "c:/test")
}
library(cricketr)``````

The cricketr package includes some pre-packaged sample (.csv) files. You can use these sample to test functions as shown below

``````# Retrieve the file path of a data file installed with cricketr
#pathToFile <- system.file("data", "tendulkar.csv", package = "cricketr")
#batsman4s(pathToFile, "Sachin Tendulkar")

# The general format is pkg-function(pathToFile,par1,...)
#batsman4s(<path-To-File>,"Sachin Tendulkar")``````

“` The pre-packaged files can be accessed as shown above. To get the data of any player use the function in Test, ODI and Twenty20 use the following

### 2. For Test cricket

``#tendulkar <- getPlayerData(35320,dir="..",file="tendulkar.csv",type="batting",homeOrAway=c(1,2), result=c(1,2,4))``

### 2a. For ODI cricket

``#tendulkarOD <- getPlayerDataOD(35320,dir="..",file="tendulkarOD.csv",type="batting")``

### 2b For Twenty 20 cricket

``#tendulkarT20 <- getPlayerDataTT(35320,dir="..",file="tendulkarT20.csv",type="batting")``

## Analysis of batsmen

Important Note This needs to be done only once for a player. This function stores the player’s data in a CSV file (for e.g. tendulkar.csv as above) which can then be reused for all other functions. Once we have the data for the players many analyses can be done. This post will use the stored CSV file obtained with a prior getPlayerData for all subsequent analyses

## Sachin Tendulkar’s performance – Basic Analyses

The 3 plots below provide the following for Tendulkar

1. Frequency percentage of runs in each run range over the whole career
2. Mean Strike Rate for runs scored in the given range
3. A histogram of runs frequency percentages in runs ranges For example

### 3. Basic analyses

``````par(mfrow=c(1,3))
par(mar=c(4,4,2,2))
#batsmanRunsFreqPerf("./tendulkar.csv","Tendulkar")
#batsmanMeanStrikeRate("./tendulkar.csv","Tendulkar")
#batsmanRunsRanges("./tendulkar.csv","Tendulkar")
dev.off()``````
``````## null device
##           1``````
1. Player 1
2. Player 2
3. Player 3
4. Player 4

### 4. More analyses

``````par(mfrow=c(1,3))
par(mar=c(4,4,2,2))
#batsman4s("./player1.csv","Player1")
#batsman6s("./player1.csv","Player1")
#batsmanMeanStrikeRate("./player1.csv","Player1")

# For ODI and T20
#batsmanScoringRateODTT("./player1.csv","Player1")
dev.off()``````
``````## null device
##           1``````
``````par(mfrow=c(1,3))
par(mar=c(4,4,2,2))
#batsman4s("./player2.csv","Player2")
#batsman6s("./player2.csv","Player2")
#batsmanMeanStrikeRate("./player2.csv","Player2")
# For ODI and T20
#batsmanScoringRateODTT("./player1.csv","Player1")
dev.off()``````
``````## null device
##           1``````
``````par(mfrow=c(1,3))
par(mar=c(4,4,2,2))
#batsman4s("./player3.csv","Player3")
#batsman6s("./player3.csv","Player3")
#batsmanMeanStrikeRate("./player3.csv","Player3")
# For ODI and T20
#batsmanScoringRateODTT("./player1.csv","Player1")

dev.off()``````
``````## null device
##           1``````
``````par(mfrow=c(1,3))
par(mar=c(4,4,2,2))
#batsman4s("./player4.csv","Player4")
#batsman6s("./player4.csv","Player4")
#batsmanMeanStrikeRate("./player4.csv","Player4")
# For ODI and T20
#batsmanScoringRateODTT("./player1.csv","Player1")
dev.off()``````
``````## null device
##           1``````

Note: For mean strike rate in ODI and Twenty20 use the function batsmanScoringRateODTT()

### 5.Boxplot histogram plot

This plot shows a combined boxplot of the Runs ranges and a histogram of the Runs Frequency

``````#batsmanPerfBoxHist("./player1.csv","Player1")
#batsmanPerfBoxHist("./player2.csv","Player2")
#batsmanPerfBoxHist("./player3.csv","Player3")
#batsmanPerfBoxHist("./player4.csv","Player4")``````

### 6. Contribution to won and lost matches

For the 2 functions below you will have to use the getPlayerDataSp() function. I have commented this as I already have these files. This function can only be used for Test matches

``````#player1sp <- getPlayerDataSp(xxxx,tdir=".",tfile="player1sp.csv",ttype="batting")
#player2sp <- getPlayerDataSp(xxxx,tdir=".",tfile="player2sp.csv",ttype="batting")
#player3sp <- getPlayerDataSp(xxxx,tdir=".",tfile="player3sp.csv",ttype="batting")
#player4sp <- getPlayerDataSp(xxxx,tdir=".",tfile="player4sp.csv",ttype="batting")``````
``````par(mfrow=c(2,2))
par(mar=c(4,4,2,2))
#batsmanContributionWonLost("player1sp.csv","Player1")
#batsmanContributionWonLost("player2sp.csv","Player2")
#batsmanContributionWonLost("player3sp.csv","Player3")
#batsmanContributionWonLost("player4sp.csv","Player4")
dev.off()``````
``````## null device
##           1``````

### 7, Performance at home and overseas

This function also requires the use of getPlayerDataSp() as shown above. This can only be used for Test matches

``````par(mfrow=c(2,2))
par(mar=c(4,4,2,2))
#batsmanPerfHomeAway("player1sp.csv","Player1")
#batsmanPerfHomeAway("player2sp.csv","Player2")
#batsmanPerfHomeAway("player3sp.csv","Player3")
#batsmanPerfHomeAway("player4sp.csv","Player4")
dev.off()``````
``````## null device
##           1``````

### 8. Batsman average at different venues

``````par(mfrow=c(2,2))
par(mar=c(4,4,2,2))
#batsmanAvgRunsGround("./player1.csv","Player1")
#batsmanAvgRunsGround("./player2.csv","Player2")
#batsmanAvgRunsGround("./player3.csv","Ponting")
#batsmanAvgRunsGround("./player4.csv","Player4")
dev.off()``````
``````## null device
##           1``````

### 9. Batsman average against different opposition

``````par(mfrow=c(2,2))
par(mar=c(4,4,2,2))
#batsmanAvgRunsOpposition("./player1.csv","Player1")
#batsmanAvgRunsOpposition("./player2.csv","Player2")
#batsmanAvgRunsOpposition("./player3.csv","Ponting")
#batsmanAvgRunsOpposition("./player4.csv","Player4")
dev.off()``````
``````## null device
##           1``````

### 10. Runs Likelihood of batsman

``````par(mfrow=c(2,2))
par(mar=c(4,4,2,2))
#batsmanRunsLikelihood("./player1.csv","Player1")
#batsmanRunsLikelihood("./player2.csv","Player2")
#batsmanRunsLikelihood("./player3.csv","Ponting")
#batsmanRunsLikelihood("./player4.csv","Player4")
dev.off()``````
``````## null device
##           1``````

### 11. Moving Average of runs in career

``````par(mfrow=c(2,2))
par(mar=c(4,4,2,2))
#batsmanMovingAverage("./player1.csv","Player1")
#batsmanMovingAverage("./player2.csv","Player2")
#batsmanMovingAverage("./player3.csv","Ponting")
#batsmanMovingAverage("./player4.csv","Player4")
dev.off()``````
``````## null device
##           1``````

### 12. Cumulative Average runs of batsman in career

``````par(mfrow=c(2,2))
par(mar=c(4,4,2,2))
#batsmanCumulativeAverageRuns("./player1.csv","Player1")
#batsmanCumulativeAverageRuns("./player2.csv","Player2")
#batsmanCumulativeAverageRuns("./player3.csv","Ponting")
#batsmanCumulativeAverageRuns("./player4.csv","Player4")
dev.off()``````
``````## null device
##           1``````

### 13. Cumulative Average strike rate of batsman in career

``````par(mfrow=c(2,2))
par(mar=c(4,4,2,2))
#batsmanCumulativeStrikeRate("./player1.csv","Player1")
#batsmanCumulativeStrikeRate("./player2.csv","Player2")
#batsmanCumulativeStrikeRate("./player3.csv","Ponting")
#batsmanCumulativeStrikeRate("./player4.csv","Player4")
dev.off()``````
``````## null device
##           1``````

### 14. Future Runs forecast

Here are plots that forecast how the batsman will perform in future. In this case 90% of the career runs trend is uses as the training set. the remaining 10% is the test set.

A Holt-Winters forecating model is used to forecast future performance based on the 90% training set. The forecated runs trend is plotted. The test set is also plotted to see how close the forecast and the actual matches

Take a look at the runs forecasted for the batsman below.

``````par(mfrow=c(2,2))
par(mar=c(4,4,2,2))
#batsmanPerfForecast("./player1.csv","Player1")
#batsmanPerfForecast("./player2.csv","Player2")
#batsmanPerfForecast("./player3.csv","Player3")
#batsmanPerfForecast("./player4.csv","Player4")
dev.off()``````
``````## null device
##           1``````

### 15. Relative Mean Strike Rate plot

The plot below compares the Mean Strike Rate of the batsman for each of the runs ranges of 10 and plots them. The plot indicate the following

``````frames <- list("./player1.csv","./player2.csv","player3.csv","player4.csv")
names <- list("Player1","Player2","Player3","Player4")
#relativeBatsmanSR(frames,names)``````

### 16. Relative Runs Frequency plot

The plot below gives the relative Runs Frequency Percetages for each 10 run bucket. The plot below show

``````frames <- list("./player1.csv","./player2.csv","player3.csv","player4.csv")
names <- list("Player1","Player2","Player3","Player4")
#relativeRunsFreqPerf(frames,names)``````

### 17. Relative cumulative average runs in career

``````frames <- list("./player1.csv","./player2.csv","player3.csv","player4.csv")
names <- list("Player1","Player2","Player3","Player4")
#relativeBatsmanCumulativeAvgRuns(frames,names)``````

### 18. Relative cumulative average strike rate in career

``````frames <- list("./player1.csv","./player2.csv","player3.csv","player4.csv")
names <- list("Player1","Player2","Player3","player4")
#relativeBatsmanCumulativeStrikeRate(frames,names)``````

### 19. Check Batsman In-Form or Out-of-Form

The below computation uses Null Hypothesis testing and p-value to determine if the batsman is in-form or out-of-form. For this 90% of the career runs is chosen as the population and the mean computed. The last 10% is chosen to be the sample set and the sample Mean and the sample Standard Deviation are caculated.

The Null Hypothesis (H0) assumes that the batsman continues to stay in-form where the sample mean is within 95% confidence interval of population mean The Alternative (Ha) assumes that the batsman is out of form the sample mean is beyond the 95% confidence interval of the population mean.

A significance value of 0.05 is chosen and p-value us computed If p-value >= .05 – Batsman In-Form If p-value < 0.05 – Batsman Out-of-Form

Note Ideally the p-value should be done for a population that follows the Normal Distribution. But the runs population is usually left skewed. So some correction may be needed. I will revisit this later

This is done for the Top 4 batsman

``````#checkBatsmanInForm("./player1.csv","Player1")
#checkBatsmanInForm("./player2.csv","Player2")
#checkBatsmanInForm("./player3.csv","Player3")
#checkBatsmanInForm("./player4.csv","Player4")``````

### 20. 3D plot of Runs vs Balls Faced and Minutes at Crease

The plot is a scatter plot of Runs vs Balls faced and Minutes at Crease. A prediction plane is fitted

``````par(mfrow=c(1,2))
par(mar=c(4,4,2,2))
#battingPerf3d("./player1.csv","Player1")
#battingPerf3d("./player2.csv","Player2")``````
``````par(mfrow=c(1,2))
par(mar=c(4,4,2,2))
#battingPerf3d("./player3.csv","Player3")
#battingPerf3d("./player4.csv","player4")
dev.off()``````
``````## null device
##           1``````

### 21. Predicting Runs given Balls Faced and Minutes at Crease

A multi-variate regression plane is fitted between Runs and Balls faced +Minutes at crease.

``````BF <- seq( 10, 400,length=15)
Mins <- seq(30,600,length=15)
newDF <- data.frame(BF,Mins)
#Player1 <- batsmanRunsPredict("./player1.csv","Player1",newdataframe=newDF)
#Player2 <- batsmanRunsPredict("./player2.csv","Player2",newdataframe=newDF)
#ponting <- batsmanRunsPredict("./player3.csv","Player3",newdataframe=newDF)
#sangakkara <- batsmanRunsPredict("./player4.csv","Player4",newdataframe=newDF)``````
``````#batsmen <-cbind(round(Player1\$Runs),round(Player2\$Runs),round(Player3\$Runs),round(Player4\$Runs))
#colnames(batsmen) <- c("Player1","Player2","Player3","Player4")
#newDF <- data.frame(round(newDF\$BF),round(newDF\$Mins))
#colnames(newDF) <- c("BallsFaced","MinsAtCrease")
#predictedRuns <- cbind(newDF,batsmen)
#predictedRuns``````

## Analysis of bowlers

1. Bowler1
2. Bowler2
3. Bowler3
4. Bowler4

player1 <- getPlayerData(xxxx,dir=“..”,file=“player1.csv”,type=“bowling”) Note For One day you will have to use getPlayerDataOD() and for Twenty20 it is getPlayerDataTT()

### 21. Wicket Frequency Plot

This plot below computes the percentage frequency of number of wickets taken for e.g 1 wicket x%, 2 wickets y% etc and plots them as a continuous line

``````par(mfrow=c(1,3))
par(mar=c(4,4,2,2))
#bowlerWktsFreqPercent("./bowler1.csv","Bowler1")
#bowlerWktsFreqPercent("./bowler2.csv","Bowler2")
#bowlerWktsFreqPercent("./bowler3.csv","Bowler3")
dev.off()``````
``````## null device
##           1``````

### 22. Wickets Runs plot

``````par(mfrow=c(1,3))
par(mar=c(4,4,2,2))
#bowlerWktsRunsPlot("./bowler1.csv","Bowler1")
#bowlerWktsRunsPlot("./bowler2.csv","Bowler2")
#bowlerWktsRunsPlot("./bowler3.csv","Bowler3")
dev.off()``````
``````## null device
##           1``````

### 23. Average wickets at different venues

``#bowlerAvgWktsGround("./bowler3.csv","Bowler3")``

### 24. Average wickets against different opposition

``#bowlerAvgWktsOpposition("./bowler3.csv","Bowler3")``

### 25. Wickets taken moving average

``````par(mfrow=c(1,3))
par(mar=c(4,4,2,2))
#bowlerMovingAverage("./bowler1.csv","Bowler1")
#bowlerMovingAverage("./bowler2.csv","Bowler2")
#bowlerMovingAverage("./bowler3.csv","Bowler3")

dev.off()``````
``````## null device
##           1``````

### 26. Cumulative Wickets taken

``````par(mfrow=c(1,3))
par(mar=c(4,4,2,2))
#bowlerCumulativeAvgWickets("./bowler1.csv","Bowler1")
#bowlerCumulativeAvgWickets("./bowler2.csv","Bowler2")
#bowlerCumulativeAvgWickets("./bowler3.csv","Bowler3")
dev.off()``````
``````## null device
##           1``````

### 27. Cumulative Economy rate

``````par(mfrow=c(1,3))
par(mar=c(4,4,2,2))
#bowlerCumulativeAvgEconRate("./bowler1.csv","Bowler1")
#bowlerCumulativeAvgEconRate("./bowler2.csv","Bowler2")
#bowlerCumulativeAvgEconRate("./bowler3.csv","Bowler3")
dev.off()``````
``````## null device
##           1``````

### 28. Future Wickets forecast

Here are plots that forecast how the bowler will perform in future. In this case 90% of the career wickets trend is used as the training set. the remaining 10% is the test set.

A Holt-Winters forecating model is used to forecast future performance based on the 90% training set. The forecated wickets trend is plotted. The test set is also plotted to see how close the forecast and the actual matches

``````par(mfrow=c(1,3))
par(mar=c(4,4,2,2))
#bowlerPerfForecast("./bowler1.csv","Bowler1")
#bowlerPerfForecast("./bowler2.csv","Bowler2")
#bowlerPerfForecast("./bowler3.csv","Bowler3")
dev.off()``````
``````## null device
##           1``````

### 29. Contribution to matches won and lost

As discussed above the next 2 charts require the use of getPlayerDataSp(). This can only be done for Test matches

``````#bowler1sp <- getPlayerDataSp(xxxx,tdir=".",tfile="bowler1sp.csv",ttype="bowling")
#bowler2sp <- getPlayerDataSp(xxxx,tdir=".",tfile="bowler2sp.csv",ttype="bowling")
#bowler3sp <- getPlayerDataSp(xxxx,tdir=".",tfile="bowler3sp.csv",ttype="bowling")``````
``````par(mfrow=c(1,3))
par(mar=c(4,4,2,2))
#bowlerContributionWonLost("bowler1sp","Bowler1")
#bowlerContributionWonLost("bowler2sp","Bowler2")
#bowlerContributionWonLost("bowler3sp","Bowler3")
dev.off()``````
``````## null device
##           1``````

### 30. Performance home and overseas.

This can only be done for Test matches

``````par(mfrow=c(1,3))
par(mar=c(4,4,2,2))
#bowlerPerfHomeAway("bowler1sp","Bowler1")
#bowlerPerfHomeAway("bowler2sp","Bowler2")
#bowlerPerfHomeAway("bowler3sp","Bowler3")
dev.off()``````
``````## null device
##           1``````

### 31 Relative Wickets Frequency Percentage

``````frames <- list("./bowler1.csv","./bowler3.csv","bowler2.csv")
names <- list("Bowler1","Bowler3","Bowler2")
#relativeBowlingPerf(frames,names)``````

### 32 Relative Economy Rate against wickets taken

``````frames <- list("./bowler1.csv","./bowler3.csv","bowler2.csv")
names <- list("Bowler1","Bowler3","Bowler2")
#relativeBowlingER(frames,names)``````

### 33 Relative cumulative average wickets of bowlers in career

``````frames <- list("./bowler1.csv","./bowler3.csv","bowler2.csv")
names <- list("Bowler1","Bowler3","Bowler2")
#relativeBowlerCumulativeAvgWickets(frames,names)``````

### 34 Relative cumulative average economy rate of bowlers

``````frames <- list("./bowler1.csv","./bowler3.csv","bowler2.csv")
names <- list("Bowler1","Bowler3","Bowler2")
#relativeBowlerCumulativeAvgEconRate(frames,names)``````

### 35 Check for bowler in-form/out-of-form

The below computation uses Null Hypothesis testing and p-value to determine if the bowler is in-form or out-of-form. For this 90% of the career wickets is chosen as the population and the mean computed. The last 10% is chosen to be the sample set and the sample Mean and the sample Standard Deviation are caculated.

The Null Hypothesis (H0) assumes that the bowler continues to stay in-form where the sample mean is within 95% confidence interval of population mean The Alternative (Ha) assumes that the bowler is out of form the sample mean is beyond the 95% confidence interval of the population mean.

A significance value of 0.05 is chosen and p-value us computed If p-value >= .05 – Batsman In-Form If p-value < 0.05 – Batsman Out-of-Form

Note Ideally the p-value should be done for a population that follows the Normal Distribution. But the runs population is usually left skewed. So some correction may be needed. I will revisit this later

Note: The check for the form status of the bowlers indicate

``````#checkBowlerInForm("./bowler1.csv","Bowler1")
#checkBowlerInForm("./bowler2.csv","Bowler2")
#checkBowlerInForm("./bowler3.csv","Bowler3")
dev.off()``````
``````## null device
##           1``````

# Big Data-2: Move into the big league:Graduate from R to SparkR

This post is a continuation of my earlier post Big Data-1: Move into the big league:Graduate from Python to Pyspark. While the earlier post discussed parallel constructs in Python and Pyspark, this post elaborates similar and key constructs in R and SparkR. While this post just focuses on the programming part of R and SparkR it is essential to understand and fully grasp the concept of Spark, RDD and how data is distributed across the clusters. This post like the earlier post shows how if you already have a good handle of R, you can easily graduate to Big Data with SparkR

Note 1: This notebook has also been published at Databricks community site Big Data-2: Move into the big league:Graduate from R to SparkR

Note 2: You can download this RMarkdown file from Github at Big Data- Python to Pyspark and R to SparkR

Note: To upload the CSV to databricks see the video Upload Flat File to Databricks Table

```# Read CSV file
#Check the dimensions of the dataframe
dim(tendulkar)
```
`[1] 347  12`
```# Load the SparkR library
library(SparkR)
# Initiate a SparkR session
sparkR.session()
delimiter = ",",
source = "csv",
inferSchema = "true",
na.strings = "")

# Check the dimensions of the dataframe
dim(tendulkar1)
```
`[1] 347  12`
2a. Data frame shape – R
```# Get the shape of the dataframe in R
dim(tendulkar)
```
`[1] 347  12`
2b. Dataframe shape – SparkR

The same ‘dim’ command works in SparkR too!

```dim(tendulkar1)
```
`[1] 347  12`
3a . Dataframe columns – R
```# Get the names
names(tendulkar) # Also colnames(tendulkar)
```
``` [1] "Runs"       "Mins"       "BF"         "X4s"        "X6s"
[6] "SR"         "Pos"        "Dismissal"  "Inns"       "Opposition"
[11] "Ground"     "Start.Date"```
3b. Dataframe columns – SparkR
```names(tendulkar1)
```
``` [1] "Runs"       "Mins"       "BF"         "4s"         "6s"
[6] "SR"         "Pos"        "Dismissal"  "Inns"       "Opposition"
[11] "Ground"     "Start Date"```
4a. Rename columns – R
```names(tendulkar)=c('Runs','Minutes','BallsFaced','Fours','Sixes','StrikeRate','Position','Dismissal','Innings','Opposition','Ground','StartDate')
names(tendulkar)
```
``` [1] "Runs"       "Minutes"    "BallsFaced" "Fours"      "Sixes"
[6] "StrikeRate" "Position"   "Dismissal"  "Innings"    "Opposition"
[11] "Ground"     "StartDate"```
4b. Rename columns – SparkR
```names(tendulkar1)=c('Runs','Minutes','BallsFaced','Fours','Sixes','StrikeRate','Position','Dismissal','Innings','Opposition','Ground','StartDate')
names(tendulkar1)
```
``` [1] "Runs"       "Minutes"    "BallsFaced" "Fours"      "Sixes"
[6] "StrikeRate" "Position"   "Dismissal"  "Innings"    "Opposition"
[11] "Ground"     "StartDate"```
5a. Summary – R
```summary(tendulkar)
```
```     Runs              Minutes        BallsFaced         Fours
Length:347         Min.   :  1.0   Min.   :  0.00   Min.   : 0.000
Class :character   1st Qu.: 33.0   1st Qu.: 22.00   1st Qu.: 1.000
Mode  :character   Median : 82.0   Median : 58.50   Median : 4.000
Mean   :125.5   Mean   : 89.75   Mean   : 6.274
3rd Qu.:181.0   3rd Qu.:133.25   3rd Qu.: 9.000
Max.   :613.0   Max.   :436.00   Max.   :35.000
NA's   :18      NA's   :19       NA's   :19
Sixes          StrikeRate        Position     Dismissal
Min.   :0.0000   Min.   :  0.00   Min.   :2.00   Length:347
1st Qu.:0.0000   1st Qu.: 38.09   1st Qu.:4.00   Class :character
Median :0.0000   Median : 52.25   Median :4.00   Mode  :character
Mean   :0.2097   Mean   : 51.79   Mean   :4.24
3rd Qu.:0.0000   3rd Qu.: 65.09   3rd Qu.:4.00
Max.   :4.0000   Max.   :166.66   Max.   :7.00
NA's   :18       NA's   :20       NA's   :18
Innings       Opposition           Ground           StartDate
Min.   :1.000   Length:347         Length:347         Length:347
1st Qu.:1.000   Class :character   Class :character   Class :character
Median :2.000   Mode  :character   Mode  :character   Mode  :character
Mean   :2.376
3rd Qu.:3.000
Max.   :4.000
NA's   :1```
5b. Summary – SparkR
```summary(tendulkar1)
```
`SparkDataFrame[summary:string, Runs:string, Minutes:string, BallsFaced:string, Fours:string, Sixes:string, StrikeRate:string, Position:string, Dismissal:string, Innings:string, Opposition:string, Ground:string, StartDate:string]`
6a. Displaying details of dataframe with str() – R
```str(tendulkar)
```
```'data.frame':	347 obs. of  12 variables:
\$ Runs      : chr  "15" "DNB" "59" "8" ...
\$ Minutes   : int  28 NA 254 24 124 74 193 1 50 324 ...
\$ BallsFaced: int  24 NA 172 16 90 51 134 1 44 266 ...
\$ Fours     : int  2 NA 4 1 5 5 6 0 3 5 ...
\$ Sixes     : int  0 NA 0 0 0 0 0 0 0 0 ...
\$ StrikeRate: num  62.5 NA 34.3 50 45.5 ...
\$ Position  : int  6 NA 6 6 7 6 6 6 6 6 ...
\$ Dismissal : chr  "bowled" NA "lbw" "run out" ...
\$ Innings   : int  2 4 1 3 1 1 3 2 3 1 ...
\$ Opposition: chr  "v Pakistan" "v Pakistan" "v Pakistan" "v Pakistan" ...
\$ StartDate : chr  "15-Nov-89" "15-Nov-89" "23-Nov-89" "23-Nov-89" ...```
6b. Displaying details of dataframe with str() – SparkR
```str(tendulkar1)
```
```'SparkDataFrame': 12 variables:
\$ Runs      : chr "15" "DNB" "59" "8" "41" "35"
\$ Minutes   : chr "28" "-" "254" "24" "124" "74"
\$ BallsFaced: chr "24" "-" "172" "16" "90" "51"
\$ Fours     : chr "2" "-" "4" "1" "5" "5"
\$ Sixes     : chr "0" "-" "0" "0" "0" "0"
\$ StrikeRate: chr "62.5" "-" "34.3" "50" "45.55" "68.62"
\$ Position  : chr "6" "-" "6" "6" "7" "6"
\$ Dismissal : chr "bowled" "-" "lbw" "run out" "bowled" "lbw"
\$ Innings   : chr "2" "4" "1" "3" "1" "1"
\$ Opposition: chr "v Pakistan" "v Pakistan" "v Pakistan" "v Pakistan" "v Pakistan" "v Pakistan"
\$ StartDate : chr "15-Nov-89" "15-Nov-89" "23-Nov-89" "23-Nov-89" "1-Dec-89" "9-Dec-89"```
```print(head(tendulkar),3)
print(tail(tendulkar),3)
```
``` Runs Minutes BallsFaced Fours Sixes StrikeRate Position Dismissal Innings
1   15      28         24     2     0      62.50        6    bowled       2
2  DNB      NA         NA    NA    NA         NA       NA             4
3   59     254        172     4     0      34.30        6       lbw       1
4    8      24         16     1     0      50.00        6   run out       3
5   41     124         90     5     0      45.55        7    bowled       1
6   35      74         51     5     0      68.62        6       lbw       1
Opposition     Ground StartDate
1 v Pakistan    Karachi 15-Nov-89
2 v Pakistan    Karachi 15-Nov-89
5 v Pakistan     Lahore  1-Dec-89
6 v Pakistan    Sialkot  9-Dec-89
Runs Minutes BallsFaced Fours Sixes StrikeRate Position Dismissal Innings
342   37     125         81     5     0      45.67        4    caught       2
343   21      71         23     2     0      91.30        4   run out       4
344   32      99         53     5     0      60.37        4       lbw       2
345    1       8          5     0     0      20.00        4       lbw       4
346   10      41         24     2     0      41.66        4       lbw       2
347   74     150        118    12     0      62.71        4    caught       2
Opposition  Ground StartDate
342   v Australia  Mohali 14-Mar-13
343   v Australia  Mohali 14-Mar-13
344   v Australia   Delhi 22-Mar-13
345   v Australia   Delhi 22-Mar-13
346 v West Indies Kolkata  6-Nov-13
347 v West Indies  Mumbai 14-Nov-13```
```head(tendulkar1,3)
```
```  Runs Minutes BallsFaced Fours Sixes StrikeRate Position Dismissal Innings
1   15      28         24     2     0       62.5        6    bowled       2
2  DNB       -          -     -     -          -        -         -       4
3   59     254        172     4     0       34.3        6       lbw       1
Opposition     Ground StartDate
1 v Pakistan    Karachi 15-Nov-89
2 v Pakistan    Karachi 15-Nov-89
8a. Determining the column types with sapply -R
```sapply(tendulkar,class)
```
```       Runs     Minutes  BallsFaced       Fours       Sixes  StrikeRate
"character"   "integer"   "integer"   "integer"   "integer"   "numeric"
Position   Dismissal     Innings  Opposition      Ground   StartDate
"integer" "character"   "integer" "character" "character" "character"```
8b. Determining the column types with printSchema – SparkR
```printSchema(tendulkar1)
```
```root
|-- Runs: string (nullable = true)
|-- Minutes: string (nullable = true)
|-- BallsFaced: string (nullable = true)
|-- Fours: string (nullable = true)
|-- Sixes: string (nullable = true)
|-- StrikeRate: string (nullable = true)
|-- Position: string (nullable = true)
|-- Dismissal: string (nullable = true)
|-- Innings: string (nullable = true)
|-- Opposition: string (nullable = true)
|-- Ground: string (nullable = true)
|-- StartDate: string (nullable = true)```
9a. Selecting columns – R
```library(dplyr)
df=select(tendulkar,Runs,BallsFaced,Minutes)
```
```  Runs BallsFaced Minutes
1   15         24      28
2  DNB         NA      NA
3   59        172     254
4    8         16      24
5   41         90     124```
9b. Selecting columns – SparkR
```library(SparkR)
Sys.setenv(SPARK_HOME="/usr/hdp/2.6.0.3-8/spark")
.libPaths(c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"), .libPaths()))
# Initiate a SparkR session
sparkR.session()
delimiter = ",",
source = "csv",
inferSchema = "true",
na.strings = "")
df=SparkR::select(tendulkar1, "Runs", "BF","Mins")
```
```  Runs  BF Mins
1   15  24   28
2  DNB   -    -
3   59 172  254
4    8  16   24
5   41  90  124
6   35  51   74```
10a. Filter rows by criteria – R
```library(dplyr)
df=tendulkar %>% filter(Runs > 50)
```
```  Runs Minutes BallsFaced Fours Sixes StrikeRate Position Dismissal Innings
1  DNB      NA         NA    NA    NA         NA       NA             4
2   59     254        172     4     0      34.30        6       lbw       1
3    8      24         16     1     0      50.00        6   run out       3
4   57     193        134     6     0      42.53        6    caught       3
5   88     324        266     5     0      33.08        6    caught       1
Opposition     Ground StartDate
1    v Pakistan    Karachi 15-Nov-89
4    v Pakistan    Sialkot  9-Dec-89
5 v New Zealand     Napier  9-Feb-90```
10b. Filter rows by criteria – SparkR
```df=SparkR::filter(tendulkar1, tendulkar1\$Runs > 50)
```
```  Runs Mins  BF 4s 6s    SR Pos Dismissal Inns     Opposition       Ground
1   59  254 172  4  0  34.3   6       lbw    1     v Pakistan   Faisalabad
2   57  193 134  6  0 42.53   6    caught    3     v Pakistan      Sialkot
3   88  324 266  5  0 33.08   6    caught    1  v New Zealand       Napier
4   68  216 136  8  0    50   6    caught    2      v England   Manchester
5  114  228 161 16  0  70.8   4    caught    2    v Australia        Perth
6  111  373 270 19  0 41.11   4    caught    2 v South Africa Johannesburg
Start Date
1  23-Nov-89
2   9-Dec-89
3   9-Feb-90
4   9-Aug-90
5   1-Feb-92
6  26-Nov-92```
11a. Unique values -R
```unique(tendulkar\$Runs)
```
```  [1] "15"   "DNB"  "59"   "8"    "41"   "35"   "57"   "0"    "24"   "88"
[11] "5"    "10"   "27"   "68"   "119*" "21"   "11"   "16"   "7"    "40"
[21] "148*" "6"    "17"   "114"  "111"  "1"    "73"   "50"   "9*"   "165"
[31] "78"   "62"   "TDNB" "28"   "104*" "71"   "142"  "96"   "43"   "11*"
[41] "34"   "85"   "179"  "54"   "4"    "0*"   "52*"  "2"    "122"  "31"
[51] "177"  "74"   "42"   "18"   "61"   "36"   "169"  "9"    "15*"  "92"
[61] "83"   "143"  "139"  "23"   "148"  "13"   "155*" "79"   "47"   "113"
[71] "67"   "136"  "29"   "53"   "124*" "126*" "44*"  "217"  "116"  "52"
[81] "45"   "97"   "20"   "39"   "201*" "76"   "65"   "126"  "36*"  "69"
[91] "155"  "22*"  "103"  "26"   "90"   "176"  "117"  "86"   "12"   "193"
[101] "16*"  "51"   "32"   "55"   "37"   "44"   "241*" "60*"  "194*" "3"
[111] "32*"  "248*" "94"   "22"   "109"  "19"   "14"   "28*"  "63"   "64"
[121] "101"  "122*" "91"   "82"   "56*"  "154*" "153"  "49"   "10*"  "103*"
[131] "160"  "100*" "105*" "100"  "106"  "84"   "203"  "98"   "38"   "214"
[141] "53*"  "111*" "146"  "14*"  "56"   "80"   "25"   "81"   "13*"```
11b. Unique values – SparkR
```head(SparkR::distinct(tendulkar1[,"Runs"]),5)
```
```  Runs
1 119*
2    7
3   51
4  169
5  32*```
12a. Aggregate – Mean, min and max – R
```library(dplyr)
library(magrittr)
a <- tendulkar\$Runs != "DNB"
tendulkar <- tendulkar[a,]
dim(tendulkar)

# Remove rows with 'TDNB'
c <- tendulkar\$Runs != "TDNB"
tendulkar <- tendulkar[c,]

# Remove rows with absent
d <- tendulkar\$Runs != "absent"
tendulkar <- tendulkar[d,]
dim(tendulkar)

# Remove the "* indicating not out
tendulkar\$Runs <- as.numeric(gsub("\\*","",tendulkar\$Runs))
c <- complete.cases(tendulkar)

#Subset the rows which are complete
tendulkar <- tendulkar[c,]
print(dim(tendulkar))
df <-tendulkar %>%  group_by(Ground) %>% summarise(meanRuns= mean(Runs), minRuns=min(Runs), maxRuns=max(Runs))
#names(tendulkar)
```
```[1] 327  12
# A tibble: 6 x 4
Ground       meanRuns minRuns maxRuns

3 Auckland         5.00      5.      5.
4 Bangalore       57.9       4.    214.
5 Birmingham      46.8       1.    122.
6 Bloemfontein    85.0      15.    155.```
12b. Aggregate- Mean, Min, Max – SparkR
```sparkR.session()

delimiter = ",",
source = "csv",
inferSchema = "true",
na.strings = "")

print(dim(tendulkar1))
tendulkar1 <-SparkR::filter(tendulkar1,tendulkar1\$Runs != "DNB")
print(dim(tendulkar1))
tendulkar1<-SparkR::filter(tendulkar1,tendulkar1\$Runs != "TDNB")
print(dim(tendulkar1))
tendulkar1<-SparkR::filter(tendulkar1,tendulkar1\$Runs != "absent")
print(dim(tendulkar1))

# Cast the string type Runs to double
withColumn(tendulkar1, "Runs", cast(tendulkar1\$Runs, "double"))
# Remove the "* indicating not out
tendulkar1\$Runs=SparkR::regexp_replace(tendulkar1\$Runs, "\\*", "")
df=SparkR::summarize(SparkR::groupBy(tendulkar1, tendulkar1\$Ground), mean = mean(tendulkar1\$Runs), minRuns=min(tendulkar1\$Runs),maxRuns=max(tendulkar1\$Runs))
```
```[1] 347  12
[1] 330  12
[1] 329  12
[1] 329  12
Ground       mean minRuns maxRuns
1      Bangalore  54.312500       0      96
3  Colombo (PSS)  37.200000      14      71
4   Christchurch  12.000000       0      24
5       Auckland   5.000000       5       5
6        Chennai  60.625000       0      81
7      Centurion  73.500000     111      36
8       Brisbane   7.666667       0       7
9     Birmingham  46.750000       1      40
11 Colombo (RPS) 143.000000     143     143
12    Chittagong  57.800000     101      36
13     Cape Town  69.857143      14       9
14    Bridgetown  26.000000       0      92
15      Bulawayo  55.000000      36      74
16         Delhi  39.947368       0      76
17    Chandigarh  11.000000      11      11
18  Bloemfontein  85.000000      15     155
19 Colombo (SSC)  77.555556     104       8
20       Cuttack   2.000000       2       2```
13a Using SQL with SparkR
```sparkR.session()
delimiter = ",",
source = "csv",
inferSchema = "true",
na.strings = "")

# Register this SparkDataFrame as a temporary view.
createOrReplaceTempView(tendulkar1, "tendulkar2")

# SQL statements can be run by using the sql method
df=SparkR::sql("SELECT * FROM tendulkar2 WHERE Ground='Karachi'")

```  Runs Mins BF 4s 6s    SR Pos Dismissal Inns Opposition  Ground Start Date