Cricpy adds team analytics to its arsenal!!

I can’t sit still and see another man slaving and working. I want to get up and superintend, and walk round with my hands in my pockets, and tell him what to do. It is my energetic nature. I can’t help it.

It always does seem to me that I am doing more work than I should do. It is not that I object to the work, mind you; I like work: it fascinates me. I can sit and look at it for hours. I love to keep it by me: the idea of getting rid of it nearly breaks my heart.

Let your boat of life be light, packed with only what you need – a homely home and simple pleasures, one or two friends, worth the name, someone to love and someone to love you, a cat, a dog, and a pipe or two, enough to eat and enough to wear, and a little more than enough to drink; for thirst is a dangerous thing.

                Three Men in a boat by Jerome K Jerome
                

Introduction

Cricpy, the python avatar of my R package was born about a 9 months back see Introducing cricpy:A python package to analyze performances of cricketers. Cricpy, like its R twin, can analyze performance of batsmen & bowlers in Test, ODI and T20 formats. About a week and a half back, I added team analytics to my R package cricketr see Cricketr adds team analytics to its repertoire!!!. If cricketr has team analysis functions, then can cricpy be far behind? So, I have included the same 8 functions which can perform Team analytics into cricpy also. Team performance analysis can be done for Test, ODI and T20 matches.

This package uses the statistics info available in ESPN Cricinfo Statsguru. The current version of this package can handle all formats of the game including Test, ODI and Twenty20 cricket.

You should be able to install the package using pip install cricpy. Please be mindful of ESPN Cricinfo Terms of Use

There are 5 functions which are used internally 1) getTeamData b) getTeamNumber c) getMatchType d) getTeamDataHomeAway e) cleanTeamData

and the external functions which a) teamWinLossStatusVsOpposition b) teamWinLossStatusAtGrounds c) plotTimelineofhttps://drive.google.com/file/d/1l4nQsRZ0C2FyPosigZmo0t-kC2xZZ_wl/view?usp=sharingWinsLosses

All the above functions are common to Test, ODI and T20 teams

The data for a particular Team can be obtained with the getTeamDataHomeAway() function from the package. This will return a dataframe of the team’s win/loss status at home and away venues over a period of time. This can be saved as a CSV file. Once this is done, you can use this CSV file for all subsequent analysis

This post has been published at Rpubs at teamAnalyticsCricpy You can download the PDF version of this post at teamAnalyticsCricpy

As before you can get the help for any of the cricpy functions as below

import cricpy.analytics as ca
help(ca.teamWinLossStatusAtGrounds)
## Help on function teamWinLossStatusAtGrounds in module cricpy.analytics:
## 
## teamWinLossStatusAtGrounds(file, teamName, opposition=['all'], homeOrAway=['all'], matchType='Test', plot=False)
##     Compute the wins/losses/draw/tied etc for a Team in Test, ODI or T20 at venues
##     
##     Description
##     
##     This function computes the won,lost,draw,tied or no result for a team against other teams in home/away or neutral venues and either returns a dataframe or plots it for grounds
##     
##     Usage
##     
##     teamWinLossStatusAtGrounds(file,teamName,opposition=["all"],homeOrAway=["all"],
##                   matchType="Test",plot=FALSE)
##     Arguments
##     
##     file        
##     The CSV file for which the plot is required
##     teamName    
##     The name of the team for which plot is required
##     opposition  
##     Opposition is a vector namely ["all")] or ["Australia", "India", "England"]
##     homeOrAway  
##     This parameter is a vector which is either ["all")] or a vector of venues ["home","away","neutral"]
##     matchType   
##     Match type - Test, ODI or T20
##     plot        
##     If plot=FALSE then a data frame is returned, If plot=TRUE then a plot is generated
##     Value
##     
##     None
##     
##     Note
##     
##     Maintainer: Tinniam V Ganesh tvganesh.85@gmail.com
##     
##     Author(s)
##     
##     Tinniam V Ganesh
##     
##     References
##     
##     http://www.espncricinfo.com/ci/content/stats/index.html
##     https://gigadom.in/
##     See Also
##     
##     teamWinLossStatusVsOpposition teamWinLossStatusAtGrounds plotTimelineofWinsLosses
##     
##     Examples
##     
##     ## Not run: 
##     #Get the team data for India for Tests
##     
##     df =getTeamDataHomeAway(teamName="India",file="indiaOD.csv",matchType="ODI")
##     ca.teamWinLossStatusAtGrounds("india.csv",teamName="India",opposition=c("Australia","England","India"),
##                               homeOrAway=c("home","away"),plot=TRUE)
##     
##     ## End(Not run)

1. Get team data

1a. Test

The teams in Test cricket are included below

  1. Afghanistan 2.Bangladesh 3. England 4. World 5. India 6. Ireland 7. New Zealand 8. Pakistan 9. South Africa 10.Sri Lanka 11. West Indies 12.Zimbabwe

You can use this for the teamName paramater. This will return a dataframe and also save the file as a CSV , if save=True

Note: – Since I have already got the data as CSV files I am not executing the lines below

import cricpy.analytics as ca
# Get the data for the teams. Save as CSV
#indiaTest= ca.getTeamDataHomeAway(dir=".",teamView="bat",matchType="Test",file="indiaTest.csv",save=True,teamName="India")
#ca.getTeamDataHomeAway(teamName="South Africa", matchType="Test", file="southafricaTest.csv", save=True)
#ca.getTeamDataHomeAway(teamName="West Indies", matchType="Test", file="westindiesTest.csv", save=True)
#newzealandTest = ca.getTeamDataHomeAway(matchType="Test",file="newzealandTest.csv",save=True,teamName="New Zealand")

1b. ODI

The ODI teams in the world are below. The data for these teams can be got by names as shown below

  1. Afghanistan 2. Africa XI 3. Asia XI 4.Australia 5.Bangladesh
  2. Bermuda 7. England 8. ICC World X1 9. India 11.Ireland 12. New Zealand 13. Pakistan       14. South Africa 15.Sri Lanka 17. West Indies 18. Zimbabwe 19 Canada    21. East Africa        22. Hong Kong 23.Ireland 24. Kenya 25. Namibia 26.Nepal 27.Netherlands 28. Oman 29.Papua New Guinea 30. Scotland 31 United Arab Emirates 32. United States of America
import cricpy.analytics as ca
#indiaODI=  ca.getTeamDataHomeAway(dir=".",matchType="ODI",file="indiaODI.csv",save=True,teamName="India")
#englandODI =  ca.getTeamDataHomeAway(matchType="ODI",file="englandODI.csv",save=True,teamName="England")
#westindiesODI = ca.getTeamDataHomeAway(matchType="ODI",file="westindiesODI.csv",save=True,teamName="West Indies")
#irelandODI <- ca.getTeamDataHomeAway(matchType="ODI",file="irelandODI.csv",save=True,teamName="Ireland")

1c T20

The T20 teams in the world are

  1. Afghanistan 2. Australia 3. Bahrain 4. Bangladesh 5. Belgium 6. Belize
  2. Bermuda 8.Botswana 9. Canada 11. Costa Rica 12. Germany 13. Ghana
  3. Guernsey 15. Hong Kong 16. ICC World X1 17.India 18. Ireland 19.Italy
  4. Jersey 21. Kenya 22.Kuwait 23.Maldives 24.Malta 25.Mexico 26.Namibia
    27.Nepal 28.Netherlands 29. New Zealand 30.Nigeria 31.Oman 32. Pakistan
    33.Panama 34.Papua New Guinea 35. Philippines 36.Qatar 37.Saudi Arabia
    38.Scotland 39.South Africa 40.Spain 41.Sri Lanka 42.Uganda 43.United Arab Emirates United States of America 44.Vanuatu 45.West Indies
import cricpy.analytics as ca
#southafricaT20 = ca.getTeamDataHomeAway(matchType="T20",file="southafricaT20.csv",save=True,teamName="South Africa")
#srilankaT20 = ca.getTeamDataHomeAway(matchType="T20",file="srilankaT20.csv",save=True,teamName="Sri Lanka")
#canadaT20 = ca.getTeamDataHomeAway(matchType="T20",file="canadaT20.csv",save=True,teamName="Canada")
#afghanistanT20 = ca.getTeamDataHomeAway(matchType="T20",file="afghanistanT20.csv",save=True,teamName="Afghanistan")

2 Analysis of Test matches

The functions below perform analysis of Test teams

2a. Wins vs Loss against opposition

This function performs analysis of Test teams against other teams at home/away or neutral venue. Note:- The opposition can be a list of opposition teams. Similarly homeOrAway can also be a list of home/away/neutral venues.

import cricpy.analytics as ca
# Get the performance of Indian test team against all teams at all venues as a dataframe
df =ca.teamWinLossStatusVsOpposition("indiaTest.csv",teamName="India",opposition=["all"], homeOrAway=["all"], matchType="Test", plot=False)
print(df)
## ha                   away  home
## Opposition   Result            
## Afghanistan  won      0.0   1.0
## Australia    draw    20.0  23.0
##              lost    58.0  26.0
##              tied     0.0   2.0
##              won     13.0  39.0
## Bangladesh   draw     3.0   0.0
##              won      9.0   2.0
## England      draw    35.0  48.0
##              lost    68.0  26.0
##              won     13.0  33.0
## New Zealand  draw    18.0  28.0
##              lost    16.0   4.0
##              won     10.0  28.0
## Pakistan     draw    29.0  34.0
##              lost    14.0  10.0
##              won      2.0  13.0
## South Africa draw    13.0   3.0
##              lost    20.0  10.0
##              won      6.0  15.0
## Sri Lanka    draw    11.0  14.0
##              lost    14.0   0.0
##              won     16.0  13.0
## West Indies  draw    39.0  35.0
##              lost    32.0  28.0
##              won     13.0  21.0
## Zimbabwe     draw     1.0   1.0
##              lost     4.0   0.0
##              won      5.0   6.0
# Plot the performance of Indian Test team  against all teams at all venues
ca.teamWinLossStatusVsOpposition("indiaTest.csv",teamName="India",opposition=["all"],homeOrAway=["all"],matchType="Test",plot=True)















# Get the performance of Australia against India, England and New Zealand at all venues in Tests
df =ca.teamWinLossStatusVsOpposition("southafricaTest.csv",teamName="South Africa",opposition=["India","England","New Zealand"],homeOrAway=["all"],matchType="Test",plot=False)
print(df)

#Plot the performance of Australia against England, India and New Zealand only at home (Australia) 
## ha                  away  home
## Opposition  Result            
## England     draw      43    55
##             lost      60    62
##             won       26    34
## India       draw       5    14
##             lost      16     6
##             won        7    19
## New Zealand draw      20     7
##             lost       2     6
##             won       14    29
ca.teamWinLossStatusVsOpposition("southafricaTest.csv",teamName="South Africa",opposition=["India","England","New Zealand"],homeOrAway=["home","away"],matchType="Test",plot=True)

 

2b Wins vs losses of Test teams against opposition at different venues

import cricpy.analytics as ca
# Get the  performance of Pakistan against India, West Indies, South Africa at all venues in Tests and show performances at the venues
df = ca.teamWinLossStatusAtGrounds("westindiesTest.csv",teamName="West Indies",opposition=["India","Sri Lanka","South Africa"],homeOrAway=["all"],matchType="Test",plot=False)
print(df)

# Plot the performance of New Zealand Test team against England, Sri Lanka and Bangladesh at all grounds playes 
## ha                         away  home
## Ground             Result            
## Ahmedabad          won      2.0   0.0
## Basseterre         draw     0.0   3.0
## Bengaluru          draw     2.0   0.0
##                    won      2.0   0.0
## Bridgetown         draw     0.0   6.0
##                    lost     0.0   6.0
##                    won      0.0  14.0
## Cape Town          draw     2.0   0.0
##                    lost     6.0   0.0
## Centurion          lost     6.0   0.0
## Chennai            draw     4.0   0.0
##                    lost     8.0   0.0
##                    won      3.0   0.0
## Colombo (PSS)      lost     2.0   0.0
## Colombo (RPS)      draw     2.0   0.0
## Colombo (SSC)      lost     4.0   0.0
## Delhi              draw     6.0   0.0
##                    lost     2.0   0.0
##                    won      3.0   0.0
## Durban             lost     6.0   0.0
## Galle              draw     1.0   0.0
##                    lost     4.0   0.0
## Georgetown         draw     0.0  10.0
## Gros Islet         draw     0.0   5.0
##                    lost     0.0   2.0
## Hyderabad (Deccan) lost     2.0   0.0
## Johannesburg       lost     4.0   0.0
## Kandy              lost     4.0   0.0
## Kanpur             draw     1.0   0.0
##                    won      3.0   0.0
## Kingston           draw     0.0   8.0
##                    lost     0.0   4.0
##                    won      0.0  15.0
## Kingstown          draw     0.0   2.0
## Kolkata            draw     7.0   0.0
##                    lost     6.0   0.0
##                    won      3.0   0.0
## Mohali             won      2.0   0.0
## Moratuwa           draw     1.0   0.0
## Mumbai             draw     7.0   0.0
##                    lost     6.0   0.0
##                    won      2.0   0.0
## Mumbai (BS)        draw     5.0   0.0
##                    won      2.0   0.0
## Nagpur             draw     2.0   0.0
## North Sound        lost     0.0   2.0
## Pallekele          draw     1.0   0.0
## Port Elizabeth     draw     1.0   0.0
##                    lost     2.0   0.0
##                    won      2.0   0.0
## Port of Spain      draw     0.0  12.0
##                    lost     0.0  12.0
##                    won      0.0  10.0
## Providence         lost     0.0   2.0
## Rajkot             lost     2.0   0.0
## Roseau             draw     0.0   2.0
## St John's          draw     0.0   6.0
##                    lost     0.0   2.0
##                    won      0.0   2.0
ca. teamWinLossStatusAtGrounds("newzealandTest.csv",teamName="New Zealand",opposition=["England","Sri Lanka","Bangladesh"],homeOrAway=["all"],matchType="Test",plot=True)

 

2c. Plot the time line of wins vs losses of Test teams against opposition at different venues during an interval

import cricpy.analytics as ca
# Plot the time line of wins/losses of India against Australia, West Indies, South Africa in away/neutral venues
#from 2000-01-01 to 2017-01-01
ca.plotTimelineofWinsLosses("indiaTest.csv",teamName="India",opposition=["Australia","West Indies","South Africa"],
                         homeOrAway=["away","neutral"], startDate="2000-01-01",endDate="2017-01-01")
#Plot the time line of wins/losses of Indian Test team from 1970 onwards

ca.plotTimelineofWinsLosses("indiaTest.csv",teamName="India",startDate="1970-01-01",endDate="2017-01-01")

3 ODI

The functions below perform analysis of ODI teams listed above

3a. Wins vs Loss against opposition ODI teams

This function performs analysis of ODI teams against other teams at home/away or neutral venue. Note:- The opposition can be a vector of opposition teams. Similarly homeOrAway can also be a vector of home/away/neutral venues.

import cricpy.analytics as ca
# Get the performance of West Indies in ODIs against all other ODI teams at all venues and retirn as a dataframe
df = ca.teamWinLossStatusVsOpposition("westindiesODI.csv",teamName="West Indies",opposition=["all"],homeOrAway=["all"],matchType="ODI",plot=False)
print(df)

# Plot the performance of West Indies in ODIs against Sri Lanka, India at all venues
## ha                   away  home  neutral
## Opposition   Result                     
## Afghanistan  lost     0.0   1.0      2.0
##              won      0.0   1.0      0.0
## Australia    lost    41.0  25.0      8.0
##              n/r      3.0   0.0      0.0
##              tied     1.0   2.0      0.0
##              won     35.0  18.0      7.0
## Bangladesh   lost     6.0   5.0      3.0
##              n/r      1.0   0.0      1.0
##              won     10.0   8.0      3.0
## Bermuda      won      0.0   0.0      1.0
## Canada       won      2.0   1.0      1.0
## England      lost    22.0  17.0     12.0
##              n/r      0.0   3.0      0.0
##              won     15.0  23.0      6.0
## India        lost    27.0  14.0     18.0
##              n/r      0.0   1.0      0.0
##              tied     1.0   0.0      1.0
##              won     27.0  20.0     15.0
## Ireland      lost     0.0   0.0      1.0
##              won      2.0   3.0      2.0
## Kenya        lost     0.0   0.0      1.0
##              won      3.0   0.0      2.0
## Netherlands  won      0.0   0.0      2.0
## New Zealand  lost    19.0   5.0      3.0
##              n/r      2.0   0.0      2.0
##              won     10.0  15.0      5.0
## P.N.G.       won      0.0   0.0      1.0
## Pakistan     lost    11.0  15.0     34.0
##              tied     1.0   2.0      0.0
##              won     14.0  16.0     41.0
## Scotland     won      0.0   0.0      3.0
## South Africa lost    20.0  17.0      7.0
##              n/r      1.0   0.0      0.0
##              tied     0.0   0.0      1.0
##              won      5.0   7.0      3.0
## Sri Lanka    lost     9.0   5.0     11.0
##              n/r      2.0   1.0      0.0
##              won      3.0   5.0     20.0
## U.A.E.       won      0.0   0.0      2.0
## Zimbabwe     lost     4.0   1.0      5.0
##              n/r      0.0   1.0      0.0
##              tied     1.0   0.0      0.0
##              won      9.0  15.0     12.0
ca.teamWinLossStatusVsOpposition("westindiesODI.csv",teamName="West Indies",opposition=["Sri Lanka", "India"],homeOrAway=["all"],matchType="ODI",plot=True)















#Plot the performance of Ireland in ODIs against Zimbabwe, Kenya, bermuda, UAE, Oman and Scotland at all venues
ca.teamWinLossStatusVsOpposition("irelandODI.csv",teamName="Ireland",opposition=["Zimbabwe","Kenya","Bermuda","U.A.E.","Oman","Scotland"],homeOrAway=["all"],matchType="ODI",plot=True)

 

3b Wins vs losses of ODI teams against opposition at different venues

import cricpy.analytics as ca
# Plot the performance of England ODI team against Bangladesh, West Indies and Australia at all venues
ca.teamWinLossStatusAtGrounds("englandODI.csv",teamName="England",opposition=["West Indies"],homeOrAway=["all"],matchType="ODI",plot=True)
























#Plot the performance of India against South Africa, West Indies and Australia at 'home' venues
ca.teamWinLossStatusAtGrounds("indiaODI.csv",teamName="India",opposition=["South Africa"],homeOrAway=["home"],matchType="ODI",plot=True)

 

3c. Plot the time line of wins vs losses of ODI teams against opposition at different venues during an interval


import cricpy.analytics as ca
#Plot the time line of wins/losses of Bangladesh ODI team between 2015 and 2019 against all other teams and at
# all venues
ca.plotTimelineofWinsLosses("bangladeshOD.csv",teamName="Bangladesh",startDate="2015-01-01",endDate="2019-01-01",matchType="ODI")























#Plot the time line of wins/losses of India ODI against Sri Lanka, Bangladesh from 2016 to 2019
ca.plotTimelineofWinsLosses("indiaODI.csv",teamName="India",opposition=["Sri Lanka","Bangladesh"],startDate="2016-01-01",endDate="2019-01-01",matchType="ODI")

 

4 Twenty 20

The functions below perform analysis of Twenty 20 teams listed above

4a. Wins vs Loss against opposition ODI teams

This function performs analysis of T20 teams against other T20 teams at home/away or neutral venue. Note:- The opposition can be a list of opposition teams. Similarly homeOrAway can also be a list of home/away/neutral venues.

import cricpy.analytics as ca
# Get the performance of South Africa T20 team against England, India and Sri Lanka at home grounds at England
df = ca.teamWinLossStatusVsOpposition("southafricaT20.csv",teamName="South Africa",opposition=["England","India","Sri Lanka"], homeOrAway=["home"], matchType="T20", plot=False)
print(df)

#Plot the performance of South Africa T20 against England, India and Sri Lanka at all venues
## ha                 home
## Opposition Result      
## England    lost       1
##            won        4
## India      lost       5
##            won        2
## Sri Lanka  lost       2
##            tied       1
##            won        3
ca.teamWinLossStatusVsOpposition("southafricaT20.csv",teamName="South Africa", opposition=["England","India","Sri Lanka"],homeOrAway=["all"],matchType="T20",plot=True)

























#Plot the performance of Afghanistan T20 teams against all oppositions

 

ca.teamWinLossStatusVsOpposition("afghanistanT20.csv",teamName="Afghanistan",opposition=["all"],homeOrAway=["all"],matchType="T20",plot=True)

 

4b Wins vs losses of T20 teams against opposition at different venues

# Compute the performance of Canada against all opposition at all venues and show by grounds. Return as dataframe
df =ca.teamWinLossStatusAtGrounds("canadaT20.csv",teamName="Canada",opposition=["all"],homeOrAway=["all"],matchType="T20",plot=False)
print(df)

# Plot the performance of Sri Lanka T20 team against India and Bangladesh in different venues at home/away and neutral
## ha                     home  neutral
## Ground         Result               
## Abu Dhabi      lost     0.0      1.0
## Belfast        lost     0.0      1.0
##                won      0.0      2.0
## Colombo (SSC)  lost     0.0      1.0
##                won      0.0      1.0
## Dubai (DSC)    lost     0.0      5.0
## ICCA Dubai     lost     0.0      2.0
##                won      0.0      1.0
## King City (NW) lost     3.0      0.0
##                tied     1.0      0.0
## Sharjah        lost     0.0      1.0
ca.teamWinLossStatusAtGrounds("srilankaT20.csv",teamName="Sri Lanka",opposition=["India", "Bangladesh"], homeOrAway=["all"], matchType="T20", plot=True)

 

4c. Plot the time line of wins vs losses of T20 teams against opposition at different venues during an interval

#Plot the time line of Sri Lanka T20 team agaibst all opposition
ca.plotTimelineofWinsLosses("srilankaT20.csv",teamName="Sri Lanka",opposition=["Australia", "Pakistan"], startDate="2013-01-01", endDate="2019-01-01",  matchType="T20")





















# Plot the time line of South Africa T20 between 2010 and 2015 against West Indies and Pakistan
ca.plotTimelineofWinsLosses("southafricaT20.csv",teamName="South Africa",opposition=["West Indies", "Pakistan"], startDate="2010-01-01", endDate="2015-01-01",  matchType="T20")

Conclusion

With the above additional functions cricpy can now analyze batsmen, bowlers and teams in all formats of the game (Test, ODI and T20).

Have fun with cricpy!!!

You may also like

  1. My book ‘Deep Learning from first principles:Second Edition’ now on Amazon
  2. Practical Machine Learning with R and Python – Part 3
  3. Big Data-4: Webserver log analysis with RDDs, Pyspark, SparkR and SparklyR
  4. Revisiting World Bank data analysis with WDI and gVisMotionChart
  5. The Clash of the Titans in Test and ODI cricket
  6. Simulating the domino effect in Android using Box2D and AndEngine
  7. Presentation on Wireless Technologies – Part 1 8.De-blurring revisited with Wiener filter using OpenCV
  8. Cloud Computing – Design Considerations

To see all posts click Index of posts

Big Data-4: Webserver log analysis with RDDs, Pyspark, SparkR and SparklyR

“There’s something so paradoxical about pi. On the one hand, it represents order, as embodied by the shape of a circle, long held to be a symbol of perfection and eternity. On the other hand, pi is unruly, disheveled in appearance, its digits obeying no obvious rule, or at least none that we can perceive. Pi is elusive and mysterious, forever beyond reach. Its mix of order and disorder is what makes it so bewitching. ” 

From  Infinite Powers by Steven Strogatz

Anybody who wants to be “anybody” in Big Data must necessarily be able to work on both large structured and unstructured data.  Log analysis is critical in any enterprise which is usually unstructured. As I mentioned in my previous post Big Data: On RDDs, Dataframes,Hive QL with Pyspark and SparkR-Part 3 RDDs are typically used to handle unstructured data. Spark has the Dataframe abstraction over RDDs which performs better as it is optimized with the Catalyst optimization engine. Nevertheless, it is important to be able to process with RDDs.  This post is a continuation of my 3 earlier posts on Big Data namely

1. Big Data-1: Move into the big league:Graduate from Python to Pyspark
2. Big Data-2: Move into the big league:Graduate from R to SparkR
3. Big Data: On RDDs, Dataframes,Hive QL with Pyspark and SparkR-Part 3

This post uses publicly available Webserver logs from NASA. The logs are for the months Jul 95 and Aug 95 and are a good place to start unstructured text analysis/log analysis. I highly recommend parsing these publicly available logs with regular expressions. It is only when you do that the truth of Jamie Zawinski’s pearl of wisdom

“Some people, when confronted with a problem, think “I know, I’ll use regular expressions.” Now they have two problems.” – Jamie Zawinksi

hits home. I spent many hours struggling with regex!!

For this post for the RDD part,  I had to refer to Dr. Fisseha Berhane’s blog post Webserver Log Analysis and for the Pyspark part, to the Univ. of California Specialization which I had done 3 years back Big Data Analysis with Apache Spark. Once I had played around with the regex for RDDs and PySpark I managed to get SparkR and SparklyR versions to work.

The notebooks used in this post have been published and are available at

  1. logsAnalysiswithRDDs
  2. logsAnalysiswithPyspark
  3. logsAnalysiswithSparkRandSparklyR

You can also download all the notebooks from Github at WebServerLogsAnalysis

An essential and unavoidable aspect of Big Data processing is the need to process unstructured text.Web server logs are one such area which requires Big Data techniques to process massive amounts of logs. The Common Log Format also known as the NCSA Common log format, is a standardized text file format used by web servers when generating server log files. Because the format is standardized, the files can be readily analyzed.

A publicly available webserver logs is the NASA-HTTP Web server logs. This is good dataset with which we can play around to get familiar to handling web server logs. The logs can be accessed at NASA-HTTP

Description These two traces contain two month’s worth of all HTTP requests to the NASA Kennedy Space Center WWW server in Florida.

Format The logs are an ASCII file with one line per request, with the following columns:

-host making the request. A hostname when possible, otherwise the Internet address if the name could not be looked up.

-timestamp in the format “DAY MON DD HH:MM:SS YYYY”, where DAY is the day of the week, MON is the name of the month, DD is the day of the month, HH:MM:SS is the time of day using a 24-hour clock, and YYYY is the year. The timezone is -0400.

-request given in quotes.

-HTTP reply code.

-bytes in the reply.

1 Parse Web server logs with RDDs

1.1 Read NASA Web server logs

Read the logs files from NASA for the months Jul 95 and Aug 95

from pyspark import SparkContext, SparkConf
from pyspark.sql import SQLContext

conf = SparkConf().setAppName("Spark-Logs-Handling").setMaster("local[*]")
sc = SparkContext.getOrCreate(conf)

sqlcontext = SQLContext(sc)
rdd = sc.textFile("/FileStore/tables/NASA_access_log_*.gz")
rdd.count()
Out[1]: 3461613

1.2Check content

Check the logs to identify the parsing rules required for the logs

i=0
for line in rdd.sample(withReplacement = False, fraction = 0.00001, seed = 100).collect():
    i=i+1
    print(line)
    if i >5:
      break
ix-stp-fl2-19.ix.netcom.com – – [03/Aug/1995:23:03:09 -0400] “GET /images/faq.gif HTTP/1.0” 200 263
slip183-1.kw.jp.ibm.net – – [04/Aug/1995:18:42:17 -0400] “GET /shuttle/missions/sts-70/images/DSC-95EC-0001.gif HTTP/1.0” 200 107133
piweba4y.prodigy.com – – [05/Aug/1995:19:17:41 -0400] “GET /icons/menu.xbm HTTP/1.0” 200 527
ruperts.bt-sys.bt.co.uk – – [07/Aug/1995:04:44:10 -0400] “GET /shuttle/countdown/video/livevideo2.gif HTTP/1.0” 200 69067
dal06-04.ppp.iadfw.net – – [07/Aug/1995:21:10:19 -0400] “GET /images/NASA-logosmall.gif HTTP/1.0” 200 786
p15.ppp-1.directnet.com – – [10/Aug/1995:01:22:54 -0400] “GET /images/KSC-logosmall.gif HTTP/1.0” 200 1204

1.3 Write the parsing rule for each of the fields

  • host
  • timestamp
  • path
  • status
  • content_bytes

1.21 Get IP address/host name

This regex is at the start of the log and includes any non-white characted

import re
rslt=(rdd.map(lambda line: re.search('\S+',line)
   .group(0))
   .take(3)) # Get the IP address \host name
rslt
Out[3]: [‘in24.inetnebr.com’, ‘uplherc.upl.com’, ‘uplherc.upl.com’]

1.22 Get timestamp

Get the time stamp

rslt=(rdd.map(lambda line: re.search(‘(\S+ -\d{4})’,line)
    .groups())
    .take(3))  #Get the  date
rslt
[(‘[01/Aug/1995:00:00:01 -0400’,),
(‘[01/Aug/1995:00:00:07 -0400’,),
(‘[01/Aug/1995:00:00:08 -0400’,)]

1.23 HTTP request

Get the HTTP request sent to Web server \w+ {GET}

# Get the REST call with ” “
rslt=(rdd.map(lambda line: re.search('"\w+\s+([^\s]+)\s+HTTP.*"',line)
    .groups())
    .take(3)) # Get the REST call
rslt
[(‘/shuttle/missions/sts-68/news/sts-68-mcc-05.txt’,),
(‘/’,),
(‘/images/ksclogo-medium.gif’,)]

1.23Get HTTP response status

Get the HTTP response to the request

rslt=(rdd.map(lambda line: re.search('"\s(\d{3})',line)
    .groups())
    .take(3)) #Get the status
rslt
Out[6]: [(‘200’,), (‘304’,), (‘304’,)]

1.24 Get content size

Get the HTTP response in bytes

rslt=(rdd.map(lambda line: re.search(‘^.*\s(\d*)$’,line)
    .groups())
    .take(3)) # Get the content size
rslt
Out[7]: [(‘1839’,), (‘0’,), (‘0’,)]

1.24 Putting it all together

Now put all the individual pieces together into 1 big regular expression and assign to the groups

  1. Host 2. Timestamp 3. Path 4. Status 5. Content_size
rslt=(rdd.map(lambda line: re.search('^(\S+)((\s)(-))+\s(\[\S+ -\d{4}\])\s("\w+\s+([^\s]+)\s+HTTP.*")\s(\d{3}\s(\d*)$)',line)
    .groups())
    .take(3))
rslt
[(‘in24.inetnebr.com’,
‘ -‘,
‘ ‘,
‘-‘,
‘[01/Aug/1995:00:00:01 -0400]’,
‘”GET /shuttle/missions/sts-68/news/sts-68-mcc-05.txt HTTP/1.0″‘,
‘/shuttle/missions/sts-68/news/sts-68-mcc-05.txt’,
‘200 1839’,
‘1839’),
(‘uplherc.upl.com’,
‘ -‘,
‘ ‘,
‘-‘,
‘[01/Aug/1995:00:00:07 -0400]’,
‘”GET / HTTP/1.0″‘,
‘/’,
‘304 0’,
‘0’),
(‘uplherc.upl.com’,
‘ -‘,
‘ ‘,
‘-‘,
‘[01/Aug/1995:00:00:08 -0400]’,
‘”GET /images/ksclogo-medium.gif HTTP/1.0″‘,
‘/images/ksclogo-medium.gif’,
‘304 0’,
‘0’)]

1.25 Add a log parsing function

import re
def parse_log1(line):
    match = re.search('^(\S+)((\s)(-))+\s(\[\S+ -\d{4}\])\s("\w+\s+([^\s]+)\s+HTTP.*")\s(\d{3}\s(\d*)$)',line)
    if match is None:    
        return(line,0)
    else:
        return(line,1)

1.26 Check for parsing failure

Check how many lines successfully parsed with the parsing function

n_logs = rdd.count()
failed = rdd.map(lambda line: parse_log1(line)).filter(lambda line: line[1] == 0).count()
print('Out of a total of {} logs, {} failed to parse'.format(n_logs,failed))
# Get the failed records line[1] == 0
failed1=rdd.map(lambda line: parse_log1(line)).filter(lambda line: line[1]==0)
failed1.take(3)
Out of a total of 3461613 logs, 38768 failed to parse
Out[10]:
[(‘gw1.att.com – – [01/Aug/1995:00:03:53 -0400] “GET /shuttle/missions/sts-73/news HTTP/1.0” 302 -‘,
0),
(‘js002.cc.utsunomiya-u.ac.jp – – [01/Aug/1995:00:07:33 -0400] “GET /shuttle/resources/orbiters/discovery.gif HTTP/1.0” 404 -‘,
0),
(‘pipe1.nyc.pipeline.com – – [01/Aug/1995:00:12:37 -0400] “GET /history/apollo/apollo-13/apollo-13-patch-small.gif” 200 12859’,
0)]

1.26 The above rule is not enough to parse the logs

It can be seen that the single rule only parses part of the logs and we cannot group the regex separately. There is an error “AttributeError: ‘NoneType’ object has no attribute ‘group'” which shows up

#rdd.map(lambda line: re.search(‘^(\S+)((\s)(-))+\s(\[\S+ -\d{4}\])\s(“\w+\s+([^\s]+)\s+HTTP.*”)\s(\d{3}\s(\d*)$)’,line[0]).group()).take(4)

File “/databricks/spark/python/pyspark/util.py”, line 99, in wrapper
return f(*args, **kwargs)
File “<command-1348022240961444>”, line 1, in <lambda>
AttributeError: ‘NoneType’ object has no attribute ‘group’

at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:490)

1.27 Add rule for parsing failed records

One of the issues with the earlier rule is the content_size has “-” for some logs

import re
def parse_failed(line):
    match = re.search('^(\S+)((\s)(-))+\s(\[\S+ -\d{4}\])\s("\w+\s+([^\s]+)\s+HTTP.*")\s(\d{3}\s-$)',line)
    if match is None:        
        return(line,0)
    else:
        return(line,1)

1.28 Parse records which fail

Parse the records that fails with the new rule

failed2=rdd.map(lambda line: parse_failed(line)).filter(lambda line: line[1]==1)
failed2.take(5)
Out[13]:
[(‘gw1.att.com – – [01/Aug/1995:00:03:53 -0400] “GET /shuttle/missions/sts-73/news HTTP/1.0” 302 -‘,
1),
(‘js002.cc.utsunomiya-u.ac.jp – – [01/Aug/1995:00:07:33 -0400] “GET /shuttle/resources/orbiters/discovery.gif HTTP/1.0” 404 -‘,
1),
(‘tia1.eskimo.com – – [01/Aug/1995:00:28:41 -0400] “GET /pub/winvn/release.txt HTTP/1.0” 404 -‘,
1),
(‘itws.info.eng.niigata-u.ac.jp – – [01/Aug/1995:00:38:01 -0400] “GET /ksc.html/facts/about_ksc.html HTTP/1.0” 403 -‘,
1),
(‘grimnet23.idirect.com – – [01/Aug/1995:00:50:12 -0400] “GET /www/software/winvn/winvn.html HTTP/1.0” 404 -‘,
1)]

1.28 Add both rules

Add both rules for parsing the log.

Note it can be shown that even with both rules all the logs are not parse.Further rules may need to be added

import re
def parse_log2(line):
    # Parse logs with the rule below
    match = re.search('^(\S+)((\s)(-))+\s(\[\S+ -\d{4}\])\s("\w+\s+([^\s]+)\s+HTTP.*")\s(\d{3})\s(\d*)$',line)
    # If match failed then use the rule below
    if match is None:
        match = re.search('^(\S+)((\s)(-))+\s(\[\S+ -\d{4}\])\s("\w+\s+([^\s]+)\s+HTTP.*")\s(\d{3}\s-$)',line)
    if match is None:
        return (line, 0) # Return 0 for failure
    else:
        return (line, 1) # Return 1 for success

1.29 Group the different regex to groups for handling

def map2groups(line):
    match = re.search('^(\S+)((\s)(-))+\s(\[\S+ -\d{4}\])\s("\w+\s+([^\s]+)\s+HTTP.*")\s(\d{3})\s(\d*)$',line)
    if match is None:
        match = re.search('^(\S+)((\s)(-))+\s(\[\S+ -\d{4}\])\s("\w+\s+([^\s]+)\s+HTTP.*")\s(\d{3})\s(-)$',line)    
    return(match.groups())

1.30 Parse the logs and map the groups

parsed_rdd = rdd.map(lambda line: parse_log2(line)).filter(lambda line: line[1] == 1).map(lambda line : line[0])

parsed_rdd2 = parsed_rdd.map(lambda line: map2groups(line))

2. Parse Web server logs with Pyspark

2.1Read data into a Pyspark dataframe

import os
logs_file_path="/FileStore/tables/" + os.path.join('NASA_access_log_*.gz')
from pyspark.sql.functions import split, regexp_extract
base_df = sqlContext.read.text(logs_file_path)
#base_df.show(truncate=False)
from pyspark.sql.functions import split, regexp_extract
split_df = base_df.select(regexp_extract('value', r'^([^\s]+\s)', 1).alias('host'),
                          regexp_extract('value', r'^.*\[(\d\d\/\w{3}\/\d{4}:\d{2}:\d{2}:\d{2} -\d{4})]', 1).alias('timestamp'),
                          regexp_extract('value', r'^.*"\w+\s+([^\s]+)\s+HTTP.*"', 1).alias('path'),
                          regexp_extract('value', r'^.*"\s+([^\s]+)', 1).cast('integer').alias('status'),
                          regexp_extract('value', r'^.*\s+(\d+)$', 1).cast('integer').alias('content_size'))
split_df.show(5,truncate=False)
+———————+————————–+———————————————–+——+————+
|host |timestamp |path |status|content_size|
+———————+————————–+———————————————–+——+————+
|199.72.81.55 |01/Jul/1995:00:00:01 -0400|/history/apollo/ |200 |6245 |
|unicomp6.unicomp.net |01/Jul/1995:00:00:06 -0400|/shuttle/countdown/ |200 |3985 |
|199.120.110.21 |01/Jul/1995:00:00:09 -0400|/shuttle/missions/sts-73/mission-sts-73.html |200 |4085 |
|burger.letters.com |01/Jul/1995:00:00:11 -0400|/shuttle/countdown/liftoff.html |304 |0 |
|199.120.110.21 |01/Jul/1995:00:00:11 -0400|/shuttle/missions/sts-73/sts-73-patch-small.gif|200 |4179 |
+———————+————————–+———————————————–+——+————+
only showing top 5 rows

2.2 Check data

bad_rows_df = split_df.filter(split_df[‘host’].isNull() |
                              split_df['timestamp'].isNull() |
                              split_df['path'].isNull() |
                              split_df['status'].isNull() |
                             split_df['content_size'].isNull())
bad_rows_df.count()
Out[20]: 33905

2.3Check no of rows which do not have digits

We have already seen that the content_type field has ‘-‘ instead of digits in RDDs

#bad_content_size_df = base_df.filter(~ base_df[‘value’].rlike(r’\d+$’))
bad_content_size_df.count()
Out[21]: 33905

2.4 Add ‘*’ to identify bad rows

To identify the rows that are bad, concatenate ‘*’ to the content_size field where the field does not have digits. It can be seen that the content_size has ‘-‘ instead of a valid number

from pyspark.sql.functions import lit, concat
bad_content_size_df.select(concat(bad_content_size_df['value'], lit('*'))).show(4,truncate=False)
+—————————————————————————————————————————————————+
|concat(value, *) |
+—————————————————————————————————————————————————+
|dd15-062.compuserve.com – – [01/Jul/1995:00:01:12 -0400] “GET /news/sci.space.shuttle/archive/sci-space-shuttle-22-apr-1995-40.txt HTTP/1.0” 404 -*|
|dynip42.efn.org – – [01/Jul/1995:00:02:14 -0400] “GET /software HTTP/1.0” 302 -* |
|ix-or10-06.ix.netcom.com – – [01/Jul/1995:00:02:40 -0400] “GET /software/winvn HTTP/1.0” 302 -* |
|ix-or10-06.ix.netcom.com – – [01/Jul/1995:00:03:24 -0400] “GET /software HTTP/1.0” 302 -* |
+—————————————————————————————————————————————————+

2.5 Fill NAs with 0s

# Replace all null content_size values with 0.

cleaned_df = split_df.na.fill({‘content_size’: 0})

3. Webserver  logs parsing with SparkR

library(SparkR)
library(stringr)
file_location = "/FileStore/tables/NASA_access_log_Jul95.gz"
file_location = "/FileStore/tables/NASA_access_log_Aug95.gz"
# Load the SparkR library


# Initiate a SparkR session
sparkR.session()
sc <- sparkR.session()
sqlContext <- sparkRSQL.init(sc)
df <- read.text(sqlContext,"/FileStore/tables/NASA_access_log_Jul95.gz")

#df=SparkR::select(df, "value")
#head(SparkR::collect(df))
#m=regexp_extract(df$value,'\\\\S+',1)

a=df %>% 
  withColumn('host', regexp_extract(df$value, '^(\\S+)', 1)) %>%
  withColumn('timestamp', regexp_extract(df$value, "((\\S+ -\\d{4}))", 2)) %>%
  withColumn('path', regexp_extract(df$value, '(\\"\\w+\\s+([^\\s]+)\\s+HTTP.*")', 2))  %>%
  withColumn('status', regexp_extract(df$value, '(^.*"\\s+([^\\s]+))', 2)) %>%
  withColumn('content_size', regexp_extract(df$value, '(^.*\\s+(\\d+)$)', 2))
#b=a%>% select(host,timestamp,path,status,content_type)
head(SparkR::collect(a),10)

1 199.72.81.55 – – [01/Jul/1995:00:00:01 -0400] “GET /history/apollo/ HTTP/1.0” 200 6245
2 unicomp6.unicomp.net – – [01/Jul/1995:00:00:06 -0400] “GET /shuttle/countdown/ HTTP/1.0” 200 3985
3 199.120.110.21 – – [01/Jul/1995:00:00:09 -0400] “GET /shuttle/missions/sts-73/mission-sts-73.html HTTP/1.0” 200 4085
4 burger.letters.com – – [01/Jul/1995:00:00:11 -0400] “GET /shuttle/countdown/liftoff.html HTTP/1.0” 304 0
5 199.120.110.21 – – [01/Jul/1995:00:00:11 -0400] “GET /shuttle/missions/sts-73/sts-73-patch-small.gif HTTP/1.0” 200 4179
6 burger.letters.com – – [01/Jul/1995:00:00:12 -0400] “GET /images/NASA-logosmall.gif HTTP/1.0” 304 0
7 burger.letters.com – – [01/Jul/1995:00:00:12 -0400] “GET /shuttle/countdown/video/livevideo.gif HTTP/1.0” 200 0
8 205.212.115.106 – – [01/Jul/1995:00:00:12 -0400] “GET /shuttle/countdown/countdown.html HTTP/1.0” 200 3985
9 d104.aa.net – – [01/Jul/1995:00:00:13 -0400] “GET /shuttle/countdown/ HTTP/1.0” 200 3985
10 129.94.144.152 – – [01/Jul/1995:00:00:13 -0400] “GET / HTTP/1.0” 200 7074
host timestamp
1 199.72.81.55 [01/Jul/1995:00:00:01 -0400
2 unicomp6.unicomp.net [01/Jul/1995:00:00:06 -0400
3 199.120.110.21 [01/Jul/1995:00:00:09 -0400
4 burger.letters.com [01/Jul/1995:00:00:11 -0400
5 199.120.110.21 [01/Jul/1995:00:00:11 -0400
6 burger.letters.com [01/Jul/1995:00:00:12 -0400
7 burger.letters.com [01/Jul/1995:00:00:12 -0400
8 205.212.115.106 [01/Jul/1995:00:00:12 -0400
9 d104.aa.net [01/Jul/1995:00:00:13 -0400
10 129.94.144.152 [01/Jul/1995:00:00:13 -0400
path status content_size
1 /history/apollo/ 200 6245
2 /shuttle/countdown/ 200 3985
3 /shuttle/missions/sts-73/mission-sts-73.html 200 4085
4 /shuttle/countdown/liftoff.html 304 0
5 /shuttle/missions/sts-73/sts-73-patch-small.gif 200 4179
6 /images/NASA-logosmall.gif 304 0
7 /shuttle/countdown/video/livevideo.gif 200 0
8 /shuttle/countdown/countdown.html 200 3985
9 /shuttle/countdown/ 200 3985
10 / 200 7074

4 Webserver logs parsing with SparklyR

install.packages("sparklyr")
library(sparklyr)
library(dplyr)
library(stringr)
#sc <- spark_connect(master = "local", version = "2.1.0")
sc <- spark_connect(method = "databricks")
sdf <-spark_read_text(sc, name="df", path = "/FileStore/tables/NASA_access_log*.gz")
sdf
Installing package into ‘/databricks/spark/R/lib’
# Source: spark [?? x 1]
   line                                                                         
                                                                           
 1 "199.72.81.55 - - [01/Jul/1995:00:00:01 -0400] \"GET /history/apollo/ HTTP/1…
 2 "unicomp6.unicomp.net - - [01/Jul/1995:00:00:06 -0400] \"GET /shuttle/countd…
 3 "199.120.110.21 - - [01/Jul/1995:00:00:09 -0400] \"GET /shuttle/missions/sts…
 4 "burger.letters.com - - [01/Jul/1995:00:00:11 -0400] \"GET /shuttle/countdow…
 5 "199.120.110.21 - - [01/Jul/1995:00:00:11 -0400] \"GET /shuttle/missions/sts…
 6 "burger.letters.com - - [01/Jul/1995:00:00:12 -0400] \"GET /images/NASA-logo…
 7 "burger.letters.com - - [01/Jul/1995:00:00:12 -0400] \"GET /shuttle/countdow…
 8 "205.212.115.106 - - [01/Jul/1995:00:00:12 -0400] \"GET /shuttle/countdown/c…
 9 "d104.aa.net - - [01/Jul/1995:00:00:13 -0400] \"GET /shuttle/countdown/ HTTP…
10 "129.94.144.152 - - [01/Jul/1995:00:00:13 -0400] \"GET / HTTP/1.0\" 200 7074"
# … with more rows
#install.packages(“sparklyr”)
library(sparklyr)
library(dplyr)
library(stringr)
#sc <- spark_connect(master = "local", version = "2.1.0")
sc <- spark_connect(method = "databricks")
sdf <-spark_read_text(sc, name="df", path = "/FileStore/tables/NASA_access_log*.gz")
sdf <- sdf %>% mutate(host = regexp_extract(line, '^(\\\\S+)',1)) %>% 
               mutate(timestamp = regexp_extract(line, '((\\\\S+ -\\\\d{4}))',2)) %>%
               mutate(path = regexp_extract(line, '(\\\\"\\\\w+\\\\s+([^\\\\s]+)\\\\s+HTTP.*")',2)) %>%
               mutate(status = regexp_extract(line, '(^.*"\\\\s+([^\\\\s]+))',2)) %>%
               mutate(content_size = regexp_extract(line, '(^.*\\\\s+(\\\\d+)$)',2))

5 Hosts

5.1  RDD

5.11 Parse and map to hosts to groups

parsed_rdd = rdd.map(lambda line: parse_log2(line)).filter(lambda line: line[1] == 1).map(lambda line : line[0])
parsed_rdd2 = parsed_rdd.map(lambda line: map2groups(line))

# Create tuples of (host,1) and apply reduceByKey() and order by descending
rslt=(parsed_rdd2.map(lambda x😦x[0],1))
                 .reduceByKey(lambda a,b:a+b)
                 .takeOrdered(10, lambda x: -x[1]))
rslt
Out[18]:
[(‘piweba3y.prodigy.com’, 21988),
(‘piweba4y.prodigy.com’, 16437),
(‘piweba1y.prodigy.com’, 12825),
(‘edams.ksc.nasa.gov’, 11962),
(‘163.206.89.4’, 9697),
(‘news.ti.com’, 8161),
(‘www-d1.proxy.aol.com’, 8047),
(‘alyssa.prodigy.com’, 8037),
(‘siltb10.orl.mmc.com’, 7573),
(‘www-a2.proxy.aol.com’, 7516)]

5.12Plot counts of hosts

import seaborn as sns

import pandas as pd import matplotlib.pyplot as plt df=pd.DataFrame(rslt,columns=[‘host’,‘count’]) sns.barplot(x=‘host’,y=‘count’,data=df) plt.subplots_adjust(bottom=0.6, right=0.8, top=0.9) plt.xticks(rotation=“vertical”,fontsize=8) display()

5.2 PySpark

5.21 Compute counts of hosts

df= (cleaned_df
     .groupBy('host')
     .count()
     .orderBy('count',ascending=False))
df.show(5)
+——————–+—–+
| host|count|
+——————–+—–+
|piweba3y.prodigy….|21988|
|piweba4y.prodigy….|16437|
|piweba1y.prodigy….|12825|
| edams.ksc.nasa.gov |11964|
| 163.206.89.4 | 9697|
+——————–+—–+
only showing top 5 rows

5.22 Plot count of hosts

import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
df1=df.toPandas()
df2 = df1.head(10)
df2.count()
sns.barplot(x='host',y='count',data=df2)
plt.subplots_adjust(bottom=0.5, right=0.8, top=0.9)
plt.xlabel("Hosts")
plt.ylabel('Count')
plt.xticks(rotation="vertical",fontsize=10)
display()

5.3 SparkR

5.31 Compute count of hosts

c <- SparkR::select(a,a$host)
df=SparkR::summarize(SparkR::groupBy(c, a$host), noHosts = count(a$host))
df1 =head(arrange(df,desc(df$noHosts)),10)
head(df1)
                  host noHosts
1 piweba3y.prodigy.com   17572
2 piweba4y.prodigy.com   11591
3 piweba1y.prodigy.com    9868
4   alyssa.prodigy.com    7852
5  siltb10.orl.mmc.com    7573
6 piweba2y.prodigy.com    5922

5.32 Plot count of hosts

library(ggplot2)
p <-ggplot(data=df1, aes(x=host, y=noHosts,fill=host)) +   geom_bar(stat="identity") + theme(axis.text.x = element_text(angle = 90, hjust = 1)) + xlab('Host') + ylab('Count')
p

5.4 SparklyR

5.41 Compute count of Hosts

df <- sdf %>% select(host,timestamp,path,status,content_size)
df1 <- df %>% select(host) %>% group_by(host) %>% summarise(noHosts=n()) %>% arrange(desc(noHosts))
df2 <-head(df1,10)

5.42 Plot count of hosts

library(ggplot2)

p <-ggplot(data=df2, aes(x=host, y=noHosts,fill=host)) + geom_bar(stat=identity”)+ theme(axis.text.x = element_text(angle = 90, hjust = 1)) + xlab(Host’) + ylab(Count’)

p

6 Paths

6.1 RDD

6.11 Parse and map to hosts to groups

parsed_rdd = rdd.map(lambda line: parse_log2(line)).filter(lambda line: line[1] == 1).map(lambda line : line[0])
parsed_rdd2 = parsed_rdd.map(lambda line: map2groups(line))
rslt=(parsed_rdd2.map(lambda x😦x[5],1))
                 .reduceByKey(lambda a,b:a+b)
                 .takeOrdered(10, lambda x: -x[1]))
rslt
[(‘”GET /images/NASA-logosmall.gif HTTP/1.0″‘, 207520),
(‘”GET /images/KSC-logosmall.gif HTTP/1.0″‘, 164487),
(‘”GET /images/MOSAIC-logosmall.gif HTTP/1.0″‘, 126933),
(‘”GET /images/USA-logosmall.gif HTTP/1.0″‘, 126108),
(‘”GET /images/WORLD-logosmall.gif HTTP/1.0″‘, 124972),
(‘”GET /images/ksclogo-medium.gif HTTP/1.0″‘, 120704),
(‘”GET /ksc.html HTTP/1.0″‘, 83209),
(‘”GET /images/launch-logo.gif HTTP/1.0″‘, 75839),
(‘”GET /history/apollo/images/apollo-logo1.gif HTTP/1.0″‘, 68759),
(‘”GET /shuttle/countdown/ HTTP/1.0″‘, 64467)]

6.12 Plot counts of HTTP Requests

import seaborn as sns

df=pd.DataFrame(rslt,columns=[‘path’,‘count’]) sns.barplot(x=‘path’,y=‘count’,data=df) plt.subplots_adjust(bottom=0.7, right=0.8, top=0.9) plt.xticks(rotation=“vertical”,fontsize=8)

display()

6.2 Pyspark

6.21 Compute count of HTTP Requests

df= (cleaned_df
     .groupBy('path')
     .count()
     .orderBy('count',ascending=False))
df.show(5)
Out[20]:
+——————–+——+
| path| count|
+——————–+——+
|/images/NASA-logo…|208362|
|/images/KSC-logos…|164813|
|/images/MOSAIC-lo…|127656|
|/images/USA-logos…|126820|
|/images/WORLD-log…|125676|
+——————–+——+
only showing top 5 rows

6.22 Plot count of HTTP Requests

import matplotlib.pyplot as plt

import pandas as pd import seaborn as sns df1=df.toPandas() df2 = df1.head(10) df2.count() sns.barplot(x=‘path’,y=‘count’,data=df2)

plt.subplots_adjust(bottom=0.7, right=0.8, top=0.9) plt.xlabel(“HTTP Requests”) plt.ylabel(‘Count’) plt.xticks(rotation=90,fontsize=8)

display()

 

6.3 SparkR

6.31Compute count of HTTP requests

library(SparkR)
c <- SparkR::select(a,a$path)
df=SparkR::summarize(SparkR::groupBy(c, a$path), numRequest = count(a$path))
df1=head(df)

3.14 Plot count of HTTP Requests

library(ggplot2)
p <-ggplot(data=df1, aes(x=path, y=numRequest,fill=path)) +   geom_bar(stat="identity") + theme(axis.text.x = element_text(angle = 90, hjust = 1))+ xlab('Path') + ylab('Count')
p

6.4 SparklyR

6.41 Compute count of paths

df <- sdf %>% select(host,timestamp,path,status,content_size)
df1 <- df %>% select(path) %>% group_by(path) %>% summarise(noPaths=n()) %>% arrange(desc(noPaths))
df2 <-head(df1,10)
df2
# Source: spark [?? x 2]
# Ordered by: desc(noPaths)
   path                                    noPaths
                                        
 1 /images/NASA-logosmall.gif               208362
 2 /images/KSC-logosmall.gif                164813
 3 /images/MOSAIC-logosmall.gif             127656
 4 /images/USA-logosmall.gif                126820
 5 /images/WORLD-logosmall.gif              125676
 6 /images/ksclogo-medium.gif               121286
 7 /ksc.html                                 83685
 8 /images/launch-logo.gif                   75960
 9 /history/apollo/images/apollo-logo1.gif   68858
10 /shuttle/countdown/                       64695

6.42 Plot count of Paths

library(ggplot2)
p <-ggplot(data=df2, aes(x=path, y=noPaths,fill=path)) +   geom_bar(stat="identity")+ theme(axis.text.x = element_text(angle = 90, hjust = 1)) + xlab('Path') + ylab('Count')
p

7.1 RDD

7.11 Compute count of HTTP Status

parsed_rdd = rdd.map(lambda line: parse_log2(line)).filter(lambda line: line[1] == 1).map(lambda line : line[0])

parsed_rdd2 = parsed_rdd.map(lambda line: map2groups(line))
rslt=(parsed_rdd2.map(lambda x😦x[7],1))
                 .reduceByKey(lambda a,b:a+b)
                 .takeOrdered(10, lambda x: -x[1]))
rslt
Out[22]:
[(‘200’, 3095682),
(‘304’, 266764),
(‘302’, 72970),
(‘404’, 20625),
(‘403’, 225),
(‘500’, 65),
(‘501’, 41)]

1.37 Plot counts of HTTP response status’

import seaborn as sns

df=pd.DataFrame(rslt,columns=[‘status’,‘count’]) sns.barplot(x=‘status’,y=‘count’,data=df) plt.subplots_adjust(bottom=0.4, right=0.8, top=0.9) plt.xticks(rotation=“vertical”,fontsize=8)

display()

7.2 Pyspark

7.21 Compute count of HTTP status

status_count=(cleaned_df
                .groupBy('status')
                .count()
                .orderBy('count',ascending=False))
status_count.show()
+——+——-+
|status| count|
+——+——-+
| 200|3100522|
| 304| 266773|
| 302| 73070|
| 404| 20901|
| 403| 225|
| 500| 65|
| 501| 41|
| 400| 15|
| null| 1|

7.22 Plot count of HTTP status

Plot the HTTP return status vs the counts

df1=status_count.toPandas()

df2 = df1.head(10) df2.count() sns.barplot(x=‘status’,y=‘count’,data=df2) plt.subplots_adjust(bottom=0.5, right=0.8, top=0.9) plt.xlabel(“HTTP Status”) plt.ylabel(‘Count’) plt.xticks(rotation=“vertical”,fontsize=10) display()

7.3 SparkR

7.31 Compute count of HTTP Response status

library(SparkR)
c <- SparkR::select(a,a$status)
df=SparkR::summarize(SparkR::groupBy(c, a$status), numStatus = count(a$status))
df1=head(df)

3.16 Plot count of HTTP Response status

library(ggplot2)
p <-ggplot(data=df1, aes(x=status, y=numStatus,fill=status)) +   geom_bar(stat="identity") + theme(axis.text.x = element_text(angle = 90, hjust = 1)) + xlab('Status') + ylab('Count')
p

7.4 SparklyR

7.41 Compute count of status

df <- sdf %>% select(host,timestamp,path,status,content_size)
df1 <- df %>% select(status) %>% group_by(status) %>% summarise(noStatus=n()) %>% arrange(desc(noStatus))
df2 <-head(df1,10)
df2
# Source: spark [?? x 2]
# Ordered by: desc(noStatus)
  status noStatus
       
1 200     3100522
2 304      266773
3 302       73070
4 404       20901
5 403         225
6 500          65
7 501          41
8 400          15
9 ""            1

7.42 Plot count of status

library(ggplot2)

p <-ggplot(data=df2, aes(x=status, y=noStatus,fill=status)) + geom_bar(stat=identity”)+ theme(axis.text.x = element_text(angle = 90, hjust = 1)) + xlab(Status’) + ylab(Count’) p

8.1 RDD

8.12 Compute count of content size

parsed_rdd = rdd.map(lambda line: parse_log2(line)).filter(lambda line: line[1] == 1).map(lambda line : line[0])
parsed_rdd2 = parsed_rdd.map(lambda line: map2groups(line))
rslt=(parsed_rdd2.map(lambda x😦x[8],1))
                 .reduceByKey(lambda a,b:a+b)
                 .takeOrdered(10, lambda x: -x[1]))
rslt
Out[24]:
[(‘0’, 280017),
(‘786’, 167281),
(‘1204’, 140505),
(‘363’, 111575),
(‘234’, 110824),
(‘669’, 110056),
(‘5866’, 107079),
(‘1713’, 66904),
(‘1173’, 63336),
(‘3635’, 55528)]

8.21 Plot content size

import seaborn as sns

df=pd.DataFrame(rslt,columns=[‘content_size’,‘count’]) sns.barplot(x=‘content_size’,y=‘count’,data=df) plt.subplots_adjust(bottom=0.4, right=0.8, top=0.9) plt.xticks(rotation=“vertical”,fontsize=8) display()

8.2 Pyspark

8.21 Compute count of content_size

size_counts=(cleaned_df
                .groupBy('content_size')
                .count()
                .orderBy('count',ascending=False))
size_counts.show(10)
+------------+------+
|content_size| count|
+------------+------+
|           0|313932|
|         786|167709|
|        1204|140668|
|         363|111835|
|         234|111086|
|         669|110313|
|        5866|107373|
|        1713| 66953|
|        1173| 63378|
|        3635| 55579|
+------------+------+
only showing top 10 rows

8.22 Plot counts of content size

Plot the path access versus the counts

df1=size_counts.toPandas()

df2 = df1.head(10) df2.count() sns.barplot(x=‘content_size’,y=‘count’,data=df2) plt.subplots_adjust(bottom=0.5, right=0.8, top=0.9) plt.xlabel(“content_size”) plt.ylabel(‘Count’) plt.xticks(rotation=“vertical”,fontsize=10) display()

8.3 SparkR

8.31 Compute count of content size

library(SparkR)
c <- SparkR::select(a,a$content_size)
df=SparkR::summarize(SparkR::groupBy(c, a$content_size), numContentSize = count(a$content_size))
df1=head(df)
df1
     content_size numContentSize
1        28426           1414
2        78382            293
3        60053              4
4        36067              2
5        13282            236
6        41785            174
8.32 Plot count of content sizes
library(ggplot2)

p <-ggplot(data=df1, aes(x=content_size, y=numContentSize,fill=content_size)) + geom_bar(stat=identity”) + theme(axis.text.x = element_text(angle = 90, hjust = 1)) + xlab(Content Size’) + ylab(Count’)

p

8.4 SparklyR

8.41Compute count of content_size

df <- sdf %>% select(host,timestamp,path,status,content_size)
df1 <- df %>% select(content_size) %>% group_by(content_size) %>% summarise(noContentSize=n()) %>% arrange(desc(noContentSize))
df2 <-head(df1,10)
df2
# Source: spark [?? x 2]
# Ordered by: desc(noContentSize)
   content_size noContentSize
                   
 1 0                   280027
 2 786                 167709
 3 1204                140668
 4 363                 111835
 5 234                 111086
 6 669                 110313
 7 5866                107373
 8 1713                 66953
 9 1173                 63378
10 3635                 55579

8.42 Plot count of content_size

library(ggplot2)
p <-ggplot(data=df2, aes(x=content_size, y=noContentSize,fill=content_size)) +   geom_bar(stat="identity")+ theme(axis.text.x = element_text(angle = 90, hjust = 1)) + xlab('Content size') + ylab('Count')
p

Conclusion: I spent many,many hours struggling with Regex and getting RDDs,Pyspark to work. Also had to spend a lot of time trying to work out the syntax for SparkR and SparklyR for parsing. After you parse the logs plotting and analysis is a piece of cake! This is definitely worth a try!

Watch this space!!

Also see
1. Practical Machine Learning with R and Python – Part 3
2. Deep Learning from first principles in Python, R and Octave – Part 5
3. My book ‘Cricket analytics with cricketr and cricpy’ is now on Amazon
4. Latency, throughput implications for the Cloud
5. Modeling a Car in Android
6. Architecting a cloud based IP Multimedia System (IMS)
7. Dabbling with Wiener filter using OpenCV

To see all posts click Index of posts

My book ‘Cricket analytics with cricketr and cricpy’ is now on Amazon

‘Cricket analytics with cricketr and cricpy – Analytics harmony with R and Python’ is now available on Amazon in both paperback ($21.99) and kindle ($9.99/Rs 449) versions. The book includes analysis of cricketers using both my R package ‘cricketr’ and my python package ‘cricpy’ for all formats of the game namely Test, ODI and T20. Both packages use data from ESPN Cricinfo Statsguru. The paperback is available on Amazon for $21.99 and the kindle version is available for $9.99/Rs 449

Pick up your copy today!

The book includes the following chapters

CONTENTS

Introduction 7
1. Cricket analytics with cricketr 9
1.1. Introducing cricketr! : An R package to analyze performances of cricketers 10
1.2. Taking cricketr for a spin – Part 1 48
1.2. cricketr digs the Ashes! 69
1.3. cricketr plays the ODIs! 97
1.4. cricketr adapts to the Twenty20 International! 139
1.5. Sixer – R package cricketr’s new Shiny avatar 168
1.6. Re-introducing cricketr! : An R package to analyze performances of cricketers 178
1.7. cricketr sizes up legendary All-rounders of yesteryear 233
1.8. cricketr flexes new muscles: The final analysis 277
1.9. The Clash of the Titans in Test and ODI cricket 300
1.10. Analyzing performances of cricketers using cricketr template 338
2. Cricket analytics with cricpy 352
2.1 Introducing cricpy:A python package to analyze performances of cricketers 353
2.2 Cricpy takes a swing at the ODIs 405
Analysis of Top 4 batsman 448
2.3 Cricpy takes guard for the Twenty20s 449
2.4 Analyzing batsmen and bowlers with cricpy template 490
9. Average runs against different opposing teams 493
3. Other cricket posts in R 500
3.1 Analyzing cricket’s batting legends – Through the mirage with R 500
3.2 Mirror, mirror … the best batsman of them all? 527
4. Appendix 541
Cricket analysis with Machine Learning using Octave 541
4.1 Informed choices through Machine Learning – Analyzing Kohli, Tendulkar and Dravid 542
4.2 Informed choices through Machine Learning-2 Pitting together Kumble, Kapil, Chandra 555
Further reading 569
Important Links 570

Also see
1. My book “Deep Learning from first principles” now on Amazon
2. Practical Machine Learning with R and Python – Part 1
3. Revisiting World Bank data analysis with WDI and gVisMotionChart
4. Natural language processing: What would Shakespeare say?
5. Optimal Cloud Computing
6. Pitching yorkpy … short of good length to IPL – Part 1
7. Computer Vision: Ramblings on derivatives, histograms and contours

To see all posts click Index of posts

Analyzing T20 matches with yorkpy templates

1. Introduction

In this post I create yorkpy templates for end-to-end analysis of any T20 matches that are available on Cricsheet as yaml format. These templates can be used to analyze Intl. T20, IPL, BBL and Natwest T20. In fact they can be used for any T20 games which have been saved in the yaml format as specified by Cricsheet Cricheet.

Noteyorkpy is the clone of my R package yorkr see yorkr pads up for the Twenty20s: Part 1- Analyzing team”s match performance

With these templates you can convert all T20 match data which is in yaml format to Pandas dataframes and save them as CSV. Note The data for Intl T20, IPL, BBL and Natwest T20 have already been converted and are available at allYorkpyData. This templates is also available at Github at yorkpyTemplate. The template includes the following steps

  1. Template for conversion and setup
  2. Analysis of Any T20 match
  3. Analysis of a T20 team in all matches against another T20 team
  4. Analysis of a T20 team in all matches against all other teams
  5. Analysis of T20 batsmen and bowlers

You can recreate the files as more matches are added to Cricsheet site in IPL 2017 and future seasons. This post contains all the steps needed for detailed analysis of IPL matches, teams and IPL player. This will also be my reference in future if I decide to analyze IPL in future!

Install yorkpy with pip install yorkpy

Data conversion of the yaml files have to be done before any analysis of T20 batsmen, bowlers, any T20 match matches between any 2 T20 team or analysis of a teams performance against all other team can be done

The first step is To convert the YAML files that are available for the different T20 leagues namely Intl. T20, IPL, BBL, Natwest T20 which are available in yaml format in Cricsheet. For initial data setup we need to use slighly different functions for each of the T20 leagues since the teams are different. The function to convert yaml to Pandas dataframe and save as CSV is common for all leagues

A. For International T20

import yorkpy.analytics as yka
# COnvert yaml to pandas and save as CSV
#yka.convertAllYaml2PandasDataframesT20(".", "..\\data1")

# Save all matches between any 2 Intl T20 countries
#yka.saveAllMatchesBetween2IntlT20s(dir1)

#Save all matches between an Intl.T20 country and all other countries
#yka.saveAllMatchesAllOppositionIntlT20(dir1)

# Get batting details for a country
#yka.getTeamBattingDetails(<country>,dir=dir1, save=True)

#Get bowling details
#yka.getTeamBowlingDetails(<country>,dir=dir1, save=True)

B. For Indian Premier League (IPL)

import yorkpy.analytics as yka
# COnvert yaml to pandas and save as CSV
#yka.convertAllYaml2PandasDataframesT20(".", "..\\data1")

# Save all matches between any 2 IPL teams
#yka.saveAllMatchesBetween2IPLTeams(dir1)

#Save all matches between an IPL team and all other teams
#yka.saveAllMatchesAllOppositionIPLT20(dir1)

# Get batting details for an IPL team
#yka.getTeamBattingDetails(<team1>,dir=dir1, save=True)

#Get bowling details for an IPL team
#yka.getTeamBowlingDetails(<team1>>,dir=dir1, save=True)

C. For Big Bash League (BBL)

import yorkpy.analytics as yka
# COnvert yaml to pandas and save as CSV
#yka.convertAllYaml2PandasDataframesT20(".", "..\\data1")

# Save all matches between any 2 BBL teams
#yka.saveAllMatchesBetween2BBLTeams(dir1)

#Save all matches between an BBL team and all other teams
#yka.saveAllMatchesAllOppositionBBLT20(dir1)

# Get batting details for an BBL team
#yka.getTeamBattingDetails(<team1>,dir=dir1, save=True)

#Get bowling details for an BBL team
#yka.getTeamBowlingDetails(<team1>>,dir=dir1, save=True)

D For Natwest T20

import yorkpy.analytics as yka
# COnvert yaml to pandas and save as CSV
#yka.convertAllYaml2PandasDataframesT20(".", "..\\data1")

# Save all matches between any 2 NWB teams
#yka.saveAllMatchesBetween2NWBTeams(dir1)

#Save all matches between an NWB team and all other teams
#yka.saveAllMatchesAllOppositionNWBT20(dir1)

# Get batting details for an NWB team
#yka.getTeamBattingDetails(<team1>,dir=dir1, save=True)

#Get bowling details for an NWB team
#yka.getTeamBowlingDetails(<team1>>,dir=dir1, save=True)

Once the conversion has been done and the data has been setup we can use any of the yorkpy functions for the the 4 leagues (Intl. T20, IPL, BBL or Natwest T20) There are four classes of functions. These functions can be used for any of the

  1. Class 1 – Functions that analyze a single T20 match
  2. Class 2 – Functions that analyze the performance of a T20 team in all matches against another T20 team
  3. Class 3 – Functions that analyze the performance of a T20 team against all other teams
  4. Class 4 – Functions that analyze individual T20 batsmen or bowler

2. Class 1 functions

These functions analyze a single T20 match (Intl T20, BBL, IPL or Natwest T20) To see actual usage of Class 1 function see Pitching yorkpy … short of good length to IPL – Part 1

import yorkpy.analytics as yka
# Get scorecard
#scorecard,extras=yka.teamBattingScorecardMatch(<team1>,"Name of Team")

#Get partnership
#match=pd.read_csv("<match.csv>")
#yka.teamBatsmenPartnershipMatch(match,<team1>,<team2>,plot=True/False)

#Batsmen vs bowler
#match=pd.read_csv("<match.csv>")
#yka.teamBatsmenVsBowlersMatch(match,<team1>,<team2>,plot=True/False)

#Bowling scorecard
#match=pd.read_csv("<match.csv>")
#a=yka.teamBowlingScorecardMatch(match,<team1>)

#Wicket Kind
#match=pd.read_csv("<match.csv>")
#yka.teamBowlingWicketKindMatch((match,<team1>,<team2>)

#Wicket Match
#match=pd.read_csv("<match.csv>")
#yka.teamBowlingWicketMatch(match,<team1>,<team2>,plot=True/False)

#Bowler vs Batsman
#match=pd.read_csv("<match.csv>")
#yka.teamBowlersVsBatsmenMatch(match,<team1>,<team2>)

#Match worm chart
#match=pd.read_csv("<match.csv>")
#yka.matchWormChart(match,<team1>,<team2>,)

3. Class 2 functions

These set of functions analyze the performance a T20 team for e.g. Intl T20, BBL or Natwest T20 in all matches against another T20 team (country or IPL, BBL or Natwest T20 team. To see usages of Class 2 functions see Pitching yorkpy…on the middle and outside off-stump to IPL – Part 2

import yorkpy.analytics as yka

# Batting partnerships - Table
#team1_team2_matches = pd.read_csv(<matches_between_2_teams.csv)
#m=yka.teamBatsmenPartnershiOppnAllMatches(team1_team2_matches,<team1/team2>,report="summary/detailed", top=<n>)

# Batting partnerships - Plot
#team1_team2_matches = pd.read_csv(<matches_between_2_teams.csv)
#yka.teamBatsmenPartnershipOppnAllMatchesChart(team1_team2_matches,<team1>,<team2> plot=<True/False>, top=<N>, partnershipRuns=<M>)

#Batsmen vs Bowlers
#team1_team2_matches = pd.read_csv(<matches_between_2_teams.csv)
#yka.teamBatsmenVsBowlersOppnAllMatches(team1_team2_matches,<team1>,<team2> plot=<True/False>, top=<N>,runsScored=<M>)

# Batting scorecard
#team1_team2_matches = pd.read_csv(<matches_between_2_teams.csv)
#scorecard=yka.teamBattingScorecardOppnAllMatches(team1_team2_matches,<team1>,<team2>)

#Bowling scorecard
#team1_team2_matches = pd.read_csv(<matches_between_2_teams.csv)
#scorecard=yka.teamBowlingScorecardOppnAllMatches(team1_team2_matches,<team1>,<team2>)

#Bowling wicket kind
#team1_team2_matches = pd.read_csv(<matches_between_2_teams.csv)
#yka.teamBowlingWicketKindOppositionAllMatches(team1_team2_matches,<team1>,<team2>,plot=<True/False>,top=<N>,wickets=<M>)

#Bowler vs batsman
#team1_team2_matches = pd.read_csv(<matches_between_2_teams.csv)
#yka.teamBowlersVsBatsmenOppnAllMatches(team1_team2_matches,<team1>,<team2>,plot=<True/False>,top=<N>,runsConceded=<M>)

# Wins vs losses
#team1_team2_matches = pd.read_csv(<matches_between_2_teams.csv)
#yka.plotWinLossBetweenTeams(team1_team2_matches,<team1>,<team2>)

#Wins by win type
#team1_team2_matches = pd.read_csv(<matches_between_2_teams.csv)
#yka.plotWinsByRunOrWickets(team1_team2_matches,<team1>)

#Wins by toss decision
#team1_team2_matches = pd.read_csv(<matches_between_2_teams.csv)
#yka.plotWinsbyTossDecision(team1_team2_matches,<team1>,tossDecision=<field/bat>)

4. Class 3 functions

This set of functions deals with analyzing the performance of a T20 team (Intl. T20, IPL, BBL or Natwest T20) in all matches against all other teams. To see usages of Class 3 functions see Pitching yorkpy…swinging away from the leg stump to IPL – Part 3. After the data is save all matches between all oppositions we can use this data

import yorkpy.analytics as yka
#Batsman partnerships
#allmatches = pd.read_csv("<allmatchesForteam")
#m=yka.teamBatsmenPartnershiAllOppnAllMatches(allmatches,<team1>,report=<"summary"/"detailed", top=<N>,partnershipRuns=<M>)

#Batsmen vs Bowlers
#allmatches = pd.read_csv("<allmatchesForteam")
#yka.teamBatsmenVsBowlersAllOppnAllMatches(allmatches,<team1>,plot=<True/False>,top=N>,runsScored=<M>)

#Batting scorecard
#allmatches = pd.read_csv("<allmatchesForteam")
#scorecard=yka.teamBattingScorecardAllOppnAllMatches(allmatches,<team1>)

#Bowling scorecard
#allmatches = pd.read_csv("<allmatchesForteam")
#scorecard=yka.teamBowlingScorecardAllOppnAllMatches(allmatches,<team1>)

#Bowling wicket kind
#allmatches = pd.read_csv("<allmatchesForteam")
#yka.teamBowlingWicketKindAllOppnAllMatches(allmatches,<team1>,plot=<True/False>,top=<N>,wickets=<M>)

# Bowler vs Batsmen
#allmatches = pd.read_csv("<allmatchesForteam")
#yka.teamBowlersVsBatsmenAllOppnAllMatches(allmatches,<team1>,plot=<True/False>,top=<N>,runsConceded=<M>)

# Wins vs losses
#allmatches = pd.read_csv("<allmatchesForteam")
#yka.plotWinLossByTeamAllOpposition(allmatches,<team1>,plot=<"summary"/"detailed">)

# Wins by win type
#allmatches = pd.read_csv("<allmatchesForteam")
#yka.plotWinsByRunOrWicketsAllOpposition(allmatches,<team1>)

# Wins by toss decision
#allmatches = pd.read_csv("<allmatchesForteam")
#yka.plotWinsbyTossDecisionAllOpposition(allmatches,<team1>,tossDecision='bat'/'field',plot='summary'/'detailed')

5. Class 4 functions

This set of functions are used for analyzing individual batsman/bowler. From the converted xxx-BattingDetails.csv and xxx-BowlingDetails.csv we can get the batsman and bowler details as shown below. Subsequenly we can perform analyses of the individual batsman and bowler. To see actual usages of Class 4 functions see Pitching yorkpy … in the block hole – Part 4

import yorkpy.analytics as yka

#Batsman analyses
#Get batsman Dataframe
#batsmanDF=yka.getBatsmanDetails(<team1>,<batsman>,dir=dir1)

#Batsman Runs vs Deliveries
#yka.batsmanRunsVsDeliveries(batsmanDF,<batsmanName>)

#Batsman fours and sixes
#yka.batsmanFoursSixes(batsmanDF,<batsmanName>)


#Batsman dismissals
#yka.batsmanDismissals(batsmanDF,<batsmanName>)

#Batsman Runs vs Strike Rate
#yka.batsmanRunsVsStrikeRate(batsmanDF,<batsmanName>)

#Batsman Moving average
#yka.batsmanMovingAverage(batsmanDF,<batsmanName>)


#Batsman Cumulative average
#yka.batsmanCumulativeAverageRuns(batsmanDF,<batsmanName>)

#Batsman Cumulative Strike rate
#yka.batsmanCumulativeStrikeRate(batsmanDF,<batsmanName>)

#Batsman Runs against opposition
#yka.batsmanRunsAgainstOpposition(batsmanDF,<batsmanName>)

#Batsman Runs against opposition
#yka.batsmanRunsVenue(batsmanDF,<batsmanName>)


#Bowler analyses
#Get bowler dataframe
#bowlerDF=yka.getBowlerWicketDetails(<team1>,<bowler>dir=dir1)

#Mean economy rate
#yka.bowlerMeanEconomyRate(bowlerDF,<bowlerName>)


#Mean Economy rate
#yka.bowlerMeanEconomyRate(bowlerDF,<bowlerName>)

#Mean Runs conceded
#yka.bowlerMeanRunsConceded(bowlerDF,<bowlerName>)

#Moving average of wickets
#yka.bowlerMovingAverage((bowlerDF,<bowlerName>)

# Cumulative average of wickets
#yka.bowlerCumulativeAvgWickets(bowlerDF,<bowlerName>)

# Cumulative economy rate
#yka.bowlerCumulativeAvgEconRate(bowlerDF,<bowlerName>)

# Wicket plot
#yka.bowlerWicketPlot(df,name)

# Wicket against opposition
#yka.bowlerWicketsAgainstOpposition(bowlerDF,<bowlerName>)

# Wickets at venue
#yka.bowlerWicketsVenue(bowlerDF,<bowlerName>)

Important note: Do check out my other posts using yorkpy at yorkpy-posts

Conclusion

With the above templates detailed analyis can be done on

  • A T20 match
  • Performance of a team in all matches against another team
  • Performance of a team in all matches against all other teams
  • Individual batting and bowling performances

See also

  1. Deep Learning from first principles in Python, R and Octave – Part 5
  2. My travels through the realms of Data Science, Machine Learning, Deep Learning and (AI)
  3. Practical Machine Learning with R and Python – Part 4
  4. Take 4+: Presentations on ‘Elements of Neural Networks and Deep Learning’ – Parts 1-8
  5. A method to crowd source pothole marking on (Indian) roads

To see all posts click Index of posts

yorkpy takes a hat-trick, bowls out Intl. T20s, BBL and Natwest T20!!!

“Dear, dear! How queer everything is to-day! And yesterday things went on just as usual. I wonder if I’ve been changed in the night? Let me think: was I the same when I got up this morning? I almost think I can remember feeling a little different. But if I’m not the same, the next question is ’Who in the world am I? Ah, that’s the great puzzle!”

             Alice's adventures  in Wonderland, Lewis Carroll

1. Introduction

In this post, yorkpy clean bowls the following T20 formats namely International T20s, Big Bash League and Natwest T20 Blast. I take yorkpy on a spin through these T20 leagues. In the post below,I choose a random set of about 10-12 of the overall 63 functions that yorkpy has, and execute them for each of the different T20 leagues – Intl T20s, BBL and Natwest T20s. yorkpy, is the python avatar of my R package yorkr, see Introducing cricket package yorkr: Part 1- Beaten by sheer pace!

There were a couple of new functions that needed to be added for each of the T20 leagues – Intl T20, BBL and Natwest T20 to take into account the different teams in each of these leagues. Further some bugs were also ironed out in tje latest version of yorkpy. yorkpy uses data from Cricsheet . The match data is in the form of YAML files. yorkpy converts these YAML files to dataframes. YAML files are very detailed and include a ball-by-ball account of the match.

– You can clone/fork the latest code for yorkpy from github yorkpy
– This post has also been published in RPubs at yorkpy takes a hat-trick
– You can download the PDF version of this post at yorkpy takes a hat-trick

The data for IPL, Intl. T20, BBL and Natwest T20 have already been converted into pandas dataframes and saved as CSVs. You can download the converted files from Github at [allYorkpyT20Data])(https://github.com/tvganesh/allYorkpyT20Data)

yorkpy has the following 4 main classes of functions

A.Functions analyzing individual T20 match (Class 1)

This was demonstrated in Pitching yorkpy . short of good length to IPL – Part 1 The functions deal with individual T20 matches. The functions are

  1. convertYaml2PandasDataframeT20()
  2. convertAllYaml2PandasDataframesT20()
  3. teamBattingScorecardMatch()
  4. teamBatsmenPartnershipMatch()
  5. teamBatsmenVsBowlersMatch()
  6. teamBowlingScorecardMatch()
  7. teamBowlingWicketKindMatch()
  8. teamBowlingWicketRunsMatch()
  9. teamBowlingWicketMatch()
  10. teamBowlersVsBatsmenMatch()
  11. matchWormChart()

B. Functions that analyze all matches between 2 T20 teams (Class 2

Pitching yorkpy.on the middle and outside off-stump to IPL – Part 2 included functions that analyze head-to-head confrontation between any 2 T20 teams The functions are

  1. getAllMatchesBetweenTeams()
  2. saveAllMatchesBetween2IPLTeams()
  3. getAllMatchesBetweenTeams()
  4. saveAllMatchesBetween2IPLTeams()
  5. teamBatsmenPartnershiOppnAllMatches()
  6. teamBatsmenPartnershipOppnAllMatchesChart()
  7. teamBatsmenVsBowlersOppnAllMatches()
  8. teamBattingScorecardOppnAllMatches()
  9. teamBowlingScorecardOppnAllMatches()
  10. teamBowlingWicketKindOppositionAllMatches()
  11. teamBowlersVsBatsmenOppnAllMatches()
  12. plotWinLossBetweenTeams()
  13. plotWinsByRunOrWickets() 23.plotWinsbyTossDecision()

C. Functions that analyze the performance of a T20 team against all other teams (Class 3)

The post Pitching yorkpy.swinging away from the leg stump to IPL – Part 3 is based on Class C set of functions shown below

  1. getAllMatchesAllOpposition()
  2. saveAllMatchesAllOppositionIPLT20(dir1)
  3. getAllMatchesAllOpposition()
  4. saveAllMatchesAllOppositionIPLT20()
  5. teamBatsmenPartnershiAllOppnAllMatches()
  6. teamBatsmenPartnershipAllOppnAllMatchesChart()
  7. teamBatsmenVsBowlersAllOppnAllMatches()
  8. teamBattingScorecardAllOppnAllMatches()
  9. teamBowlingScorecardAllOppnAllMatches()
  10. teamBowlingWicketKindAllOppnAllMatches()
  11. teamBowlersVsBatsmenAllOppnAllMatches()
  12. plotWinLossByTeamAllOpposition()
  13. plotWinsByRunOrWicketsAllOpposition()
  14. plotWinsbyTossDecisionAllOpposition()

D. Functions that analyze performances of T20 batsmen and bowlers (Class 4)

These set of functions analyze individual batsmen and bowlers and have been used in Pitching yorkpy . in the block hole – Part 4 The functions are

  1. getTeamBattingDetails()
  2. getBatsmanDetails()
  3. batsmanRunsVsDeliveries()
  4. batsmanFoursSixes()
  5. batsmanDismissals()
  6. batsmanRunsVsStrikeRate()
  7. batsmanMovingAverage()
  8. batsmanCumulativeAverageRuns()
  9. batsmanCumulativeStrikeRate()
  10. batsmanRunsAgainstOpposition()
  11. batsmanRunsVenue
  12. getTeamBowlingDetails()
  13. getBowlerWicketDetails()
  14. bowlerMeanEconomyRate()
  15. bowlerMeanRunsConceded()
  16. bowlerMovingAverage()
  17. bowlerCumulativeAvgWickets()
  18. bowlerCumulativeAvgEconRate()
  19. bowlerWicketPlot()
  20. bowlerWicketsAgainstOpposition()
  21. bowlerWicketsVenue()

Additional new functions were added to handle Intl T20s, Big Bash League and Natwest T20 Blast, since the teams are different. They are

59. saveAllMatchesBetween2IntlT20s()
60. saveAllMatchesAllOppositionIntlT20()
61. saveAllMatchesBetween2BBLTeams()
62 saveAllMatchesAllOppositionBBLT20()
63. saveAllMatchesBetween2NWBTeams()
64. saveAllMatchesAllOppositionNWBT20()

All other functions can be used as is! You can get the help of any function in yorkpy using

import yorkpy.analytics as yka
help(yka.teamBatsmenPartnershiOppnAllMatches)
## Help on function teamBatsmenPartnershiOppnAllMatches in module yorkpy.analytics:
## 
## teamBatsmenPartnershiOppnAllMatches(matches, theTeam, report='summary', top=5)
##     Team batting partnership against a opposition all IPL matches
##     
##     Description
##     
##     This function computes the performance of batsmen against all bowlers of an oppositions in 
##     all matches. This function returns a dataframe
##     
##     Usage
##     
##     teamBatsmenPartnershiOppnAllMatches(matches,theTeam,report="summary")
##     Arguments
##     
##     matches     
##     All the matches of the team against the oppositions
##     theTeam     
##     The team for which the the batting partnerships are sought
##     report      
##     If the report="summary" then the list of top batsmen with the highest partnerships 
##     is displayed. If report="detailed" then the detailed break up of partnership is returned 
##     as a dataframe
##     top
##     The number of players to be displayed from the top
##     Value
##     
##     partnerships The data frame of the partnerships
##     
##     Note
##     
##     Maintainer: Tinniam V Ganesh tvganesh.85@gmail.com
##     
##     Author(s)
##     
##     Tinniam V Ganesh
##     
##     References
##     
##     http://cricsheet.org/
##     https://gigadom.wordpress.com/
##     
##     
##     See Also
##     
##     teamBatsmenVsBowlersOppnAllMatchesPlot
##     teamBatsmenPartnershipOppnAllMatchesChart

As I mentioned above I will be randomly choosing a set of 12 functions from Class 1,2,3,4 for each of the T20 leagues (Intl T20, BBL and NWB T20) for analysis

2. International T20s

The following functions were added for handling Intl. T20s

  1. saveAllMatchesBetween2IntlT20s()
  2. saveAllMatchesAllOppositionIntlT20()

To handle the countries in Intl. T20s below

Afghanistan, Australia, Bangladesh, Bermuda, Canada, England,Hong Kong,India, Ireland, Kenya, Nepal, Netherlands, “New Zealand, Oman,Pakistan,Scotland,South Africa, Sri Lanka, United Arab Emirates,West Indies, Zimbabwe

import os
#os.chdir('C:\\software\\cricket-package\\yorkpyT20\\t20s')
#import yorkpy.analytics as yka
#1.  Convert all YAML files to dataframes and CSV
#yka.convertAllYaml2PandasDataframesT20(".", "..\\data1")
#dir1='C:\\software\\cricket-package\\yorkpyT20\\IntlT20-Matches'
#2. Save all matches between 2 T20 teams
#yka.saveAllMatchesBetween2IntlT20s(dir1)
#3. Save all matches between a T20 team and all other teams
#dir1='C:\\software\\cricket-package\\yorkpyT20\\IntlT20-Matches'
#yka.saveAllMatchesAllOppositionIntlT20(dir1)
#4. Get batting details
#dir1='C:\\software\\cricket-package\\yorkpyT20\\IntlT20-Matches
#yka.getTeamBattingDetails("Afghanistan",dir=dir1, save=True)
#yka.getTeamBattingDetails("Australia",dir=dir1,save=True)
#yka.getTeamBattingDetails("Bangladesh",dir=dir1,save=True)
#...
#5. Get bowling details
#dir1='C:\\software\\cricket-package\\yorkpyT20\\IntlT20-Matches
#yka.getTeamBowlingDetails("Afghanistan",dir=dir1, save=True)
#yka.getTeamBowlingDetails("Australia",dir=dir1,save=True)
#yka.getTeamBowlingDetails("Bangladesh",dir=dir1,save=True)
# ...

Once the data is converted you can use the yorkpy functions. The data has been converted for Intl T20 and is available at Github at IntlT20

To use the yorkpy functions for a new league we need to initial convert the YAML files into appropriate format for processing by yorkpy functions

This will create the necessary files which are are used in the functions below

2.2 2.1 Intl. T20 – Team score card  (Class 1)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\yorkpyT20\\IntlT20-Matches"
path=os.path.join(dir1,".\\India-New Zealand-2007-09-16.csv")
ind_nz=pd.read_csv(path)
scorecard,extras=yka.teamBattingScorecardMatch(ind_nz,"India")
print(scorecard)
##             batsman  runs  balls  4s  6s          SR
## 0         G Gambhir    51     34   5   2  150.000000
## 1          V Sehwag    40     18   6   2  222.222222
## 2        RV Uthappa     0      2   0   0    0.000000
## 3          MS Dhoni    24     20   2   0  120.000000
## 4      Yuvraj Singh     5      7   0   0   71.428571
## 5        KD Karthik    17     12   3   0  141.666667
## 6         IK Pathan    11     10   2   0  110.000000
## 7        AB Agarkar     1      2   0   0   50.000000
## 8   Harbhajan Singh     7      6   1   0  116.666667
## 9       S Sreesanth    19     10   4   0  190.000000
## 10         RP Singh     1      1   0   0  100.000000
print(extras)
##    total  wides  noballs  legbyes  byes  penalty  extras
## 0    370      6        0        8     0        0      14

2.2 Intl. T20 -Team batsmen partnership (Class 1)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\yorkpyT20\\IntlT20-Matches"
path=os.path.join(dir1,".\\South Africa-Australia-2009-03-27.csv")
sa_aus=pd.read_csv(path)
yka.teamBatsmenPartnershipMatch(sa_aus,'Australia','New Zealand',plot=True)

2.3 Intl. T20 -Team bowling scorecard match (Class 1)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\yorkpyT20\\IntlT20-Matches"
path=os.path.join(dir1,".\\Sri Lanka-West Indies-2012-09-28.csv")
sl_wi=pd.read_csv(path)
a=yka.teamBowlingScorecardMatch(sl_wi,'Sri Lanka')
print(a)
##          bowler  overs  runs  maidens  wicket  econrate
## 0    A Mohammed      2    13        0       0       6.5
## 1  SA Campbelle      1     8        0       1       8.0
## 2     SC Selman      1     3        0       0       3.0
## 3      SF Daley      2     5        0       1       2.5
## 4     SR Taylor      2     4        0       1       2.0
## 5     TD Smartt      2    17        0       0       8.5

2.4 Intl. T20 -Match Worm chart (Class 1)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\yorkpyT20\\IntlT20-Matches"
path=os.path.join(dir1,".\\England-India-2012-09-29.csv")
eng_ind=pd.read_csv(path)
yka.matchWormChart(eng_ind,"England", "India")

path=os.path.join(dir1,".\\Bangladesh-Ireland-2015-12-05.csv")
ban_ire=pd.read_csv(path)
yka.matchWormChart(ban_ire,"Bangladesh", "Ireland")

2.5 Intl. T20 -Team Batting partnerships all matches 2 teams (Class 2)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\yorkpyT20\\IntlT20-allMatchesBetween2Teams"
path=os.path.join(dir1,"India-England-allMatches.csv")
dc_mi_matches = pd.read_csv(path)
theTeam='India'
m=yka.teamBatsmenPartnershiOppnAllMatches(dc_mi_matches,theTeam,report="detailed", top=4)
print(m)
##      batsman  totalPartnershipRuns    non_striker  partnershipRuns
## 0   SK Raina                   265      G Gambhir                2
## 1   SK Raina                   265       KL Rahul               40
## 2   SK Raina                   265      MK Tiwary               24
## 3   SK Raina                   265       MS Dhoni              124
## 4   SK Raina                   265        P Kumar                0
## 5   SK Raina                   265      PP Chawla                4
## 6   SK Raina                   265       R Ashwin                1
## 7   SK Raina                   265      RG Sharma               16
## 8   SK Raina                   265        V Kohli               47
## 9   SK Raina                   265   Yuvraj Singh                7
## 10  MS Dhoni                   264       A Mishra                1
## 11  MS Dhoni                   264      AT Rayudu               18
## 12  MS Dhoni                   264      HH Pandya                8
## 13  MS Dhoni                   264      IK Pathan                2
## 14  MS Dhoni                   264      JJ Bumrah                2
## 15  MS Dhoni                   264      MK Pandey                3
## 16  MS Dhoni                   264  Parvez Rasool               21
## 17  MS Dhoni                   264       R Ashwin               11
## 18  MS Dhoni                   264      RA Jadeja               11
## 19  MS Dhoni                   264      RG Sharma                9
## 20  MS Dhoni                   264        RR Pant                6
## 21  MS Dhoni                   264     RV Uthappa                5
## 22  MS Dhoni                   264       SK Raina               98
## 23  MS Dhoni                   264      YK Pathan               36
## 24  MS Dhoni                   264   Yuvraj Singh               33
## 25   V Kohli                   236      AM Rahane                3
## 26   V Kohli                   236      G Gambhir               78
## 27   V Kohli                   236       KL Rahul               46
## 28   V Kohli                   236      RG Sharma                2
## 29   V Kohli                   236     RV Uthappa                4
## 30   V Kohli                   236       S Dhawan               45
## 31   V Kohli                   236       SK Raina               48
## 32   V Kohli                   236   Yuvraj Singh               10
## 33     M Raj                   176       A Sharma                2
## 34     M Raj                   176         H Kaur               18
## 35     M Raj                   176      J Goswami                6
## 36     M Raj                   176        KV Jain                5
## 37     M Raj                   176       L Kumari                5
## 38     M Raj                   176    N Niranjana                3
## 39     M Raj                   176       N Tanwar               17
## 40     M Raj                   176        PG Raut               41
## 41     M Raj                   176     R Malhotra                5
## 42     M Raj                   176     S Mandhana                8
## 43     M Raj                   176         S Naik               10
## 44     M Raj                   176       S Pandey               19
## 45     M Raj                   176       SK Naidu               37

2.6 Intl. T20 -Team Batsmen vs Bowlers all matches 2 teams (Class 2)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\yorkpyT20\\IntlT20-allMatchesBetween2Teams"
path=os.path.join(dir1,"Ireland-Netherlands-allMatches.csv")
ire_nl_matches = pd.read_csv(path)
yka.teamBatsmenVsBowlersOppnAllMatches(ire_nl_matches,'Ireland',"Netherlands",plot=True,top=3,runsScored=10)

2.7 Intl. T20 -Team Bowling scorecard all matches 2 teams (Class 2)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\yorkpyT20\\IntlT20-allMatchesBetween2Teams"
path=os.path.join(dir1,"Bangladesh-Nepal-allMatches.csv")
bang_nep_matches = pd.read_csv(path)
scorecard=yka.teamBowlingScorecardOppnAllMatches(bang_nep_matches,'Bangladesh',"Nepal")
print(scorecard)
##         bowler  overs  runs  maidens  wicket   econrate
## 0      B Regmi      3    14        0       1   4.666667
## 3   SP Gauchan      4    40        0       1  10.000000
## 1   JK Mukhiya      2    16        0       0   8.000000
## 2     P Khadka      3    23        0       0   7.666667
## 4    Sagar Pun      1    16        0       0  16.000000
## 5  Sompal Kami      2    21        0       0  10.500000

2.8 Intl. T20 -Team Batsmen vs Bowlers all Oppositions (Class 3)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\yorkpyT20\\\IntlT20-allMatchesAllOpposition\\"
path=os.path.join(dir1,"Australia-allMatchesAllOpposition.csv")
aus_matches = pd.read_csv(path)
yka.teamBatsmenVsBowlersAllOppnAllMatches(aus_matches,"Australia",plot=True,top=3,runsScored=40)

2.9 Intl. T20 -Wins vs Losses of a team against all other teams (Class 3)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\yorkpyT20\\\IntlT20-allMatchesAllOpposition\\"
path=os.path.join(dir1,"South Africa-allMatchesAllOpposition.csv")
sa_matches = pd.read_csv(path)
team1='South Africa'
yka.plotWinLossByTeamAllOpposition(sa_matches,team1,plot="detailed")

2.10 Intl. T20 -Batsmen analysis (Class 4)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\yorkpyT20\\\IntlT20-BattingBowlingDetails\\"
# Rohit Sharma
name="RG Sharma"
team='India'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanCumulativeAverageRuns(df,name)

# MJ Guptill
name="MJ Guptill"
team='New Zealand'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanCumulativeStrikeRate(df,name)

2.11 Intl. T20 -Bowler analysis (Class 4)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\yorkpyT20\\\IntlT20-BattingBowlingDetails\\"
# Shakib Al Hasan
name="Shakib Al Hasan"
team='Bangladesh'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerMeanEconomyRate(df,name)

# Rashid Khan
name="SL Malinga"
team='Sri Lanka'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerWicketsAgainstOpposition(df,name)

3. Big Bash League

The following functions for added to handle BBL teams

  1. saveAllMatchesBetween2BBLTeams()
  2. saveAllMatchesAllOppositionBBLT20

The BBL teams are included are Adelaide Strikers, Brisbane Heat, Hobart Hurricanes, Melbourne Renegades, Perth Scorchers, Sydney Sixers, Sydney Thunder

To use the yorkpy functions first the YAML files have to be converted into pandas dataframe and then saved as CSV as shown below

import os
import yorkpy.analytics as yka
os.chdir('C:\\software\\cricket-package\\yorkpyBBL\\bbl')
#1. Convert all YAML files to dataframes and save as CSV
#yka.convertAllYaml2PandasDataframesT20(".", "..\\BBLT20-Matches")
#2. Save all matches between 2 BBL teams
dir1='C:\\software\\cricket-package\\yorkpyBBL\\BBLT20-Matches'
#yka.saveAllMatchesBetween2BBLTeams(dir1)
#3. Save T20 matches between a BBL team and all other teams
dir1='C:\\software\\cricket-package\\yorkpyBBL\\BBLT20-Matches'
#yka.saveAllMatchesAllOppositionBBLT20(dir1)
#4. Get the batting details
dir1='C:\\software\\cricket-package\\yorkpyBBL\\BBLT20-Matches'
#yka.getTeamBattingDetails("Adelaide Strikers",dir=dir1, save=True)
#yka.getTeamBattingDetails("Brisbane Heat",dir=dir1,save=True)
#yka.getTeamBattingDetails("Hobart Hurricanes",dir=dir1,save=True)
#...
# Get the bowling details
dir1='C:\\software\\cricket-package\\yorkpyBBL\\BBLT20-Matches'
#yka.getTeamBowlingDetails("Adelaide Strikers",dir=dir1, save=True)
#yka.getTeamBowlingDetails("Brisbane Heat",dir=dir1,save=True)
#yka.getTeamBowlingDetails("Hobart Hurricanes",dir=dir1,save=True)
#...

The functions below perform analysis on the generated files from above. The YAML files have already been converted and are available at Github at BBL

3.1 Big Bash League – Team score card (Class 1)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\yorkpyBBL\\BBLT20-Matches"
path=os.path.join(dir1,".\\Adelaide Strikers-Brisbane Heat-2012-12-13.csv")
as_bh=pd.read_csv(path)
scorecard,extras=yka.teamBattingScorecardMatch(as_bh,"Brisbane Heat")
print(scorecard)
##          batsman  runs  balls  4s  6s          SR
## 0  LA Pomersbach    65     42   8   2  154.761905
## 1       JR Hopes     1      2   0   0   50.000000
## 2       JA Burns    37     31   2   2  119.354839
## 3   DT Christian    12     15   0   0   80.000000
## 4    NLTC Perera    12      4   0   2  300.000000
## 5        CA Lynn    19     18   1   1  105.555556
## 6    BCJ Cutting    13      5   0   2  260.000000
## 7     PJ Forrest    12      8   0   1  150.000000
## 8     CD Hartley     5      2   1   0  250.000000
print(extras)
##    total  wides  noballs  legbyes  byes  penalty  extras
## 0    371     10        2        5     0        0      17

3.2 Big Bash League -Team batsmen vs Bowlers (Class 1)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\yorkpyBBL\\BBLT20-Matches"
path=os.path.join(dir1,".\\Hobart Hurricanes-Melbourne Renegades-2012-01-18.csv")
hh_mr=pd.read_csv(path)
yka.teamBatsmenVsBowlersMatch(hh_mr,'Hobart Hurricanes','Melbourne Renegades',plot=True)

3.3 Big Bash League -Team bowling scorecard match (Class 1)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\yorkpyBBL\\BBLT20-Matches"
path=os.path.join(dir1,".\\Melbourne Stars-Sydney Thunder-2016-01-24.csv")
ms_st=pd.read_csv(path)
a=yka.teamBowlingScorecardMatch(ms_st,'Sydney Thunder')
print(a)
##           bowler  overs  runs  maidens  wicket   econrate
## 0        A Zampa      4    32        0       2   8.000000
## 1  BW Hilfenhaus      2    21        0       0  10.500000
## 2      DJ Hussey      1     9        0       1   9.000000
## 3     DJ Worrall      3    42        0       0  14.000000
## 4      EP Gulbis      2    19        0       0   9.500000
## 5        MA Beer      3    25        0       1   8.333333
## 6     MP Stoinis      4    30        0       3   7.500000

3.4 Big Bash League – Match Worm chart (Class 1)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\yorkpyBBL\\BBLT20-Matches"
path=os.path.join(dir1,".\\Sydney Sixers-Melbourne Stars-2011-12-27.csv")
ss_ms=pd.read_csv(path)
yka.matchWormChart(ss_ms,"Melbourne Stars", "Sydney Sixers")

path=os.path.join(dir1,".\\Hobart Hurricanes-Brisbane Heat-2015-01-02.csv")
hh_bh=pd.read_csv(path)
yka.matchWormChart(hh_bh,"Hobart Hurricanes", "Brisbane Heat")

3.5 Big Bash League -Team Batting partnerships all matches 2 teams (Class 2)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\yorkpyBBL\\BBLT20-allMatchesBetween2Teams"
path=os.path.join(dir1,"Brisbane Heat-Adelaide Strikers-allMatches.csv")
bh_as_matches = pd.read_csv(path)
yka.teamBatsmenPartnershipOppnAllMatchesChart(bh_as_matches,"Brisbane Heat","Adelaide Strikers",plot=True, top=4, partnershipRuns=20)

3.6 Big Bash League -Team Bowling wicket kind all matches 2 teams (Class 2)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\yorkpyBBL\\BBLT20-allMatchesBetween2Teams"
path=os.path.join(dir1,"Sydney Sixers-Perth Scorchers-allMatches.csv")
ss_ps_matches = pd.read_csv(path)
yka.teamBowlingWicketKindOppositionAllMatches(ss_ps_matches,'Perth Scorchers','Sydney Sixers',plot=True,top=5,wickets=1)

3.7 Big Bash League -Team Bowling scorecard all teams (Class 3)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\yorkpyBBL\\BBLT20-allMatchesAllOpposition"
path=os.path.join(dir1,"Hobart Hurricanes-allMatchesAllOpposition.csv")
hh_matches = pd.read_csv(path)
scorecard=yka.teamBowlingScorecardAllOppnAllMatches(hh_matches,"Hobart Hurricanes")
print(scorecard)
##              bowler  overs  runs  maidens  wicket   econrate
## 16            B Lee     20   132        0       9   6.600000
## 30         CJ McKay     13   110        0       9   8.461538
## 88    NJ Rimmington     16   103        1       9   6.437500
## 67      JW Hastings     15    88        0       8   5.866667
## 63      JP Faulkner     15   146        0       7   9.733333
## 27        CJ Gannon     17   147        1       7   8.647059
## 93          NM Lyon      8    51        0       7   6.375000
## 20      BCJ Cutting     27   226        0       7   8.370370
## 48          GB Hogg     22   167        0       7   7.590909
## 107       SM Boland     12    96        0       7   8.000000
## 15       B Laughlin     13    99        0       7   7.615385
## 87      MT Steketee     15   134        0       5   8.933333
## 121    Yasir Arafat      9    48        0       4   5.333333
## 96       PJ Cummins      8    83        0       4  10.375000
## 46      Fawad Ahmed     11    64        0       4   5.818182
## 76          MA Beer     12    63        0       4   5.250000
## 108     SNJ O'Keefe     15   104        0       4   6.933333
## 75   M Muralitharan      7    31        0       4   4.428571
## 10           AJ Tye     16   127        0       4   7.937500
## 52          J Botha     13    94        0       4   7.230769
## 56     JL Pattinson      7    71        0       4  10.142857
## 62   JP Behrendorff     16   119        0       4   7.437500
## 3           AC Agar     12    87        0       4   7.250000
## 24     BM Edmondson      4    40        0       4  10.000000
## 37        DJ Hussey      8    47        0       3   5.875000
## 49       GJ Maxwell      8    65        0       3   8.125000
## 84       MN Samuels      4    22        0       3   5.500000
## 81         MG Neser      5    54        0       3  10.800000
## 44     DT Christian      9   114        0       3  12.666667
## 50        GS Sandhu      7    51        0       3   7.285714
## ..              ...    ...   ...      ...     ...        ...
## 43        DP Nannes      8    58        0       1   7.250000
## 51         IA Moran      4    25        0       1   6.250000
## 55         JK Lalor     10    82        0       1   8.200000
## 54        JH Kallis      3    18        0       1   6.000000
## 73   LR Butterworth      4    25        0       1   6.250000
## 4      AC McDermott      2    28        0       1  14.000000
## 70         LA Doran      4    38        0       1   9.500000
## 69    KW Richardson      6    44        0       1   7.333333
## 119     WD Sheridan      2     6        0       0   3.000000
## 2       AB McDonald      1    15        0       0  15.000000
## 115      TD Andrews      3    23        0       0   7.666667
## 11          AK Heal      4    33        0       0   8.250000
## 7        AD Russell      4    40        0       0  10.000000
## 8          AJ Finch      2    15        0       0   7.500000
## 9         AJ Turner      3    28        0       0   9.333333
## 60        JM Mennie      1    20        0       0  20.000000
## 18        BA Stokes      1     9        0       0   9.000000
## 26         CH Gayle      1    16        0       0  16.000000
## 28         CJ Green      4    44        0       0  11.000000
## 95   PD Collingwood      2    20        0       0  10.000000
## 31       CJ Simmons      4    21        0       0   5.250000
## 59       JM Holland      3    34        0       0  11.333333
## 36         DJ Bravo      6    64        0       0  10.666667
## 38     DJ Pattinson      2    16        0       0   8.000000
## 41       DJ Worrall      8    90        0       0  11.250000
## 72      LN O'Connor      6    56        0       0   9.333333
## 71        LJ Wright      3    27        0       0   9.000000
## 68       KA Pollard      1     7        0       0   7.000000
## 58       JM Herrick      4    23        0       0   5.750000
## 92       NM Hauritz      5    42        0       0   8.400000
## 
## [122 rows x 6 columns]

3.8 Big Bash League -Plot wins vs losses against all teams(Class 3)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\yorkpyBBL\\BBLT20-allMatchesAllOpposition"
path=os.path.join(dir1,"Sydney Sixers-allMatchesAllOpposition.csv")
ss_matches = pd.read_csv(path)
yka.plotWinLossByTeamAllOpposition(ss_matches,'Sydney Sixers')

3.9 Big Bash League -Wins vs losses by toss decision (Class 3)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\yorkpyBBL\\BBLT20-allMatchesAllOpposition"
path=os.path.join(dir1,"Adelaide Strikers-allMatchesAllOpposition.csv")
as_matches = pd.read_csv(path)
yka.plotWinsByRunOrWicketsAllOpposition(as_matches,'Adelaide Strikers')

3.10 Big Bash League -Batsmen Analysis (Class 4)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\yorkpyBBL\\BBLT20-BattingBowlingDetails"
# CA Lynn
name="CA Lynn"
team='Brisbane Heat'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanRunsVsStrikeRate(df,name)

# UT Khawaja
name="UT Khawaja"
team='Sydney Thunder'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanRunsAgainstOpposition(df,name)

3.11Big Bash League – Bowler analysis (Class 4)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\yorkpyBBL\\BBLT20-BattingBowlingDetails"
# CJ McKay
name="CJ McKay"
team='Sydney Thunder'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerCumulativeAvgWickets(df,name)

# AU Rashid
name="AU Rashid"
team='Adelaide Strikers'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerCumulativeAvgEconRate(df,name)

4. Natwest T20 Blast

The following functions for added to handle Natwest T20 teams

  1. saveAllMatchesBetween2NWBTeams()
  2. saveAllMatchesAllOppositionNWBT20

The Natwest teams are
Derbyshire, Durham, Essex, Glamorgan, Gloucestershire, Hampshire, Kent,Lancashire, Leicestershire, Middlesex,Northamptonshire, Nottinghamshire, Somerset, Surrey, Sussex, Warwickshire, Worcestershire,Yorkshire

In order to perform analysis with yorkpy, the YAML data has to be converted to pandas dataframe and saves as CSV as shown

#import os
#import yorkpy.analytics as yka
#os.chdir('C:\\software\\cricket-package\\yorkpyNWB\\nwb')
#1. Convert YAML to dataframes and save as CSV
#yka.convertAllYaml2PandasDataframesT20(".", "..\\NWBT20-Matches")
#2. Save all matches between 2 NWBT20 teams
#dir1='C:\\software\\cricket-package\\yorkpyNWB\\NWBT20-Matches'
#yka.saveAllMatchesBetween2NWBTeams(dir1)
#3. Save all matches between a NWB T20 team and all other teams
#dir1='C:\\software\\cricket-package\\yorkpyNWB\\NWBT20-Matches'
#yka.saveAllMatchesAllOppositionNWBT20(dir1)
#4. Compute the batting details
dir1='C:\\software\\cricket-package\\yorkpyNWB\\NWBT20-Matches'
#yka.getTeamBattingDetails("Derbyshire",dir=dir1, save=True)
#yka.getTeamBattingDetails("Durham",dir=dir1,save=True)
#yka.getTeamBattingDetails("Essex",dir=dir1,save=True)
#..
#5. Compute bowling details
dir1='C:\\software\\cricket-package\\yorkpyNWB\\NWBT20-Matches'
#yka.getTeamBowlingDetails("Derbyshire",dir=dir1, save=True)
#yka.getTeamBowlingDetails("Durham",dir=dir1,save=True)
#yka.getTeamBowlingDetails("Essex",dir=dir1,save=True)
#...

Once the data is converted all yorkpy functions can be used. This has already been done and is available at github NWB

4.1 Natwest T20 Blast – Team score card (Class 1)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\\yorkpyNWB\\NWBT20-Matches"
path=os.path.join(dir1,".\\Durham-Yorkshire-2016-08-20.csv")
d_y=pd.read_csv(path)
scorecard,extras=yka.teamBattingScorecardMatch(d_y,"Durham")
print(scorecard)
##           batsman  runs  balls  4s  6s          SR
## 0     MD Stoneman    25     20   4   0  125.000000
## 1     KK Jennings    11     13   1   0   84.615385
## 2       BA Stokes    56     37   4   3  151.351351
## 3   MJ Richardson    29     23   4   1  126.086957
## 4     JTA Burnham    17     15   1   1  113.333333
## 5      RD Pringle    10      9   1   0  111.111111
## 6  PD Collingwood     2      3   0   0   66.666667
## 7        U Arshad     1      1   0   0  100.000000
print(extras)
##    total  wides  noballs  legbyes  byes  penalty  extras
## 0    305      2        0        5     0        0       7

4.2 Natwest T20 Blast -Team batsmen vs Bowlers (Class 1)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\\yorkpyNWB\\NWBT20-Matches"
path=os.path.join(dir1,".\\Derbyshire-Lancashire-2016-07-13.csv")
d_l=pd.read_csv(path)
yka.teamBatsmenVsBowlersMatch(d_l,'Lancashire','Derbyshire',plot=True)

4.3 Natwest T20 Blast -Team bowling scorecard match (Class 1)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\\yorkpyNWB\\NWBT20-Matches"
path=os.path.join(dir1,".\\Essex-Surrey-2016-05-20.csv")
e_s=pd.read_csv(path)
a=yka.teamBowlingScorecardMatch(e_s,'Essex')
print(a)
##           bowler  overs  runs  maidens  wicket   econrate
## 0  Azhar Mahmood      3    38        0       4  12.666667
## 1       GJ Batty      4    33        0       1   8.250000
## 2       JE Burke      1    18        0       0  18.000000
## 3     MW Pillans      3    28        0       0   9.333333
## 4      SM Curran      4    23        0       2   5.750000
## 5      TK Curran      4    21        0       3   5.250000

4.4 Natwest T20 Blast -Match Worm chart (Class 1)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\\yorkpyNWB\\NWBT20-Matches"
path=os.path.join(dir1,".\\Gloucestershire-Glamorgan-2016-06-10.csv")
ss_ms=pd.read_csv(path)
yka.matchWormChart(ss_ms,"Gloucestershire", "Glamorgan")

path=os.path.join(dir1,".\\Leicestershire-Northamptonshire-2016-05-20.csv")
hh_bh=pd.read_csv(path)
yka.matchWormChart(hh_bh,"Northamptonshire", "Leicestershire")

4.5 Natwest T20 Blast -Team Batting partnerships all matches 2 teams (Class 2)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\yorkpyNWB\\NWBT20-allMatchesBetween2Teams"
path=os.path.join(dir1,"Hampshire-Sussex-allMatches.csv")
h_s_matches = pd.read_csv(path)
yka.teamBatsmenPartnershipOppnAllMatchesChart(h_s_matches,"Hampshire","Sussex",plot=True, top=4, partnershipRuns=10)

4.6 Natwest T20 Blast -Team Bowling wicket kind all matches 2 teams (Class 2)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\yorkpyNWB\\NWBT20-allMatchesBetween2Teams"
path=os.path.join(dir1,"Kent-Somerset-allMatches.csv")
k_s_matches = pd.read_csv(path)
yka.teamBowlersVsBatsmenOppnAllMatches(k_s_matches,'Kent','Somerset',plot=True,
top=5,runsConceded=10)

4.7 Natwest T20 Blast -Team Bowling scorecard all teams (Class 3)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\yorkpyNWB\\NWBT20-allMatchesAllOpposition"
path=os.path.join(dir1,"Middlesex-allMatchesAllOpposition.csv")
m_matches = pd.read_csv(path)
scorecard=yka.teamBowlingScorecardAllOppnAllMatches(m_matches,"Middlesex")
print(scorecard)
##               bowler  overs  runs  maidens  wicket   econrate
## 1             AJ Tye      8    75        0       6   9.375000
## 5         BAC Howell      8    41        0       5   5.125000
## 26         GR Napier      7    65        0       5   9.285714
## 15        DI Stevens      4    31        0       4   7.750000
## 19       DW Lawrence      6    37        0       4   6.166667
## 32       JW Dernbach      4    33        0       3   8.250000
## 7          BTJ Wheal      4    43        0       3  10.750000
## 18         DR Briggs      4    24        0       3   6.000000
## 50     RK Kleinveldt      4    24        0       3   6.000000
## 46         R McLaren      7    59        0       3   8.428571
## 47         R Rampaul      3    21        0       3   7.000000
## 34         L Gregory      6    51        0       2   8.500000
## 33   KMDN Kulasekara      2    24        0       2  12.000000
## 40          MG Hogan      3    17        0       2   5.666667
## 43        MTC Waller      4    31        0       2   7.750000
## 49        RJ Gleeson      4    20        0       2   5.000000
## 48  RE van der Merwe      5    24        0       2   4.800000
## 51  RN ten Doeschate      4    32        0       2   8.000000
## 53        S Prasanna      4    20        0       2   5.000000
## 56           SW Tait      3    17        0       2   5.666667
## 57     Shahid Afridi      8    55        0       2   6.875000
## 59  T van der Gugten      3    13        1       2   4.333333
## 64          TS Mills      3    34        0       2  11.333333
## 65          WAT Beer      4    23        0       2   5.750000
## 31          JH Davey      4    28        0       2   7.000000
## 68         ZS Ansari      3    16        0       2   5.333333
## 25         GM Andrew      3    19        0       2   6.333333
## 23          GJ Batty      6    55        0       2   9.166667
## 16          DJ Bravo      3    27        0       2   9.000000
## 41          MR Quinn      6    65        0       1  10.833333
## ..               ...    ...   ...      ...     ...        ...
## 24     GL van Buuren      7    49        0       1   7.000000
## 37           MD Hunn      3    35        0       1  11.666667
## 36        LC Norwell      6    62        0       1  10.333333
## 29       JC Tredwell      4    35        0       1   8.750000
## 35         LA Dawson      6    53        0       1   8.833333
## 62           TL Best      4    51        0       0  12.750000
## 58         T Westley      2    12        0       0   6.000000
## 4         Azharullah      3    24        0       0   8.000000
## 60     TD Groenewald      1    21        0       0  21.000000
## 61         TK Curran      4    35        0       0   8.750000
## 38         MD Taylor      3    30        0       0  10.000000
## 30        JG Myburgh      1     5        0       0   5.000000
## 8          C Overton      2    18        0       0   9.000000
## 2        Ashar Zaidi      1     5        0       0   5.000000
## 66          WR Smith      2    25        0       0  12.500000
## 28         J Overton      2    24        0       0  12.000000
## 6          BJ Taylor      1     6        0       0   6.000000
## 22          GG White      4    31        0       0   7.750000
## 55          SP Crook      1     9        0       0   9.000000
## 39        ME Claydon      4    40        0       0  10.000000
## 52         RS Bopara      4    32        0       0   8.000000
## 10           CD Nash      2    19        0       0   9.500000
## 11         CH Morris      4    36        0       0   9.000000
## 12         DA Cosker      3    32        0       0  10.666667
## 13      DA Griffiths      4    39        0       0   9.750000
## 45          PD Trego      1    11        0       0  11.000000
## 44   PA van Meekeren      2    19        0       0   9.500000
## 42          MS Crane      2    25        0       0  12.500000
## 20        FK Cowdrey      1    19        0       0  19.000000
## 14        DD Masters      2    16        0       0   8.000000
## 
## [69 rows x 6 columns]

4.8 Natwest T20 Blast -Plot wins vs losses against all teams(Class 3)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\yorkpyNWB\\NWBT20-allMatchesAllOpposition"
path=os.path.join(dir1,"Warwickshire-allMatchesAllOpposition.csv")
w_matches = pd.read_csv(path)
yka.plotWinLossByTeamAllOpposition(w_matches,'Warwickshire')

4.9 Natwest T20 Blast -Batsmen Analysis (Class 4)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\yorkpyNWB\\NWBT20-BattingBowlingDetails"
# M Klinger
name="M Klinger"
team='Gloucestershire'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanRunsAgainstOpposition(df,name)

# CA Ingram
name="CA Ingram"
team='Glamorgan'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanCumulativeStrikeRate(df,name)

4.11 Natwest T20 Blast -Bowler analysis (Class 4)

import os
import pandas as pd
import yorkpy.analytics as yka
dir1="C:\\software\\cricket-package\\yorkpyNWB\\NWBT20-BattingBowlingDetails"
# BAC Howell
name="BAC Howell"
team='Gloucestershire'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerCumulativeAvgEconRate(df,name)

# GR Napier
name="GR Napier"
team='Essex'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerWicketsVenue(df,name)

Note: yorkpy will work for all T20 leagues which are in YAML format as specified in Cricsheet.

You can clone/fork the latest code for yorkpy from github yorkpy

The data for IPL, Intl. T20, BBL and Natwest T20 have already been converted into pandas dataframes and saved as CSVs. You can download the converted files from Github at [allYorkpyT20Data])(https://github.com/tvganesh/allYorkpyT20Data)

Conclusion This post shows the kind of detailed analysis that can be performed with yorkpy. In fact with all the converted data it should be possible to also train a Machine Learning model, which I will probably keep for another day. You could go ahead and use the data in other innovative ways. Do keep me posted if you do!!

Important note: Do check out my other posts using yorkpy at yorkpy-posts

Have fun with yorkpy!!

See also
1. Take 4+: Presentations on ‘Elements of Neural Networks and Deep Learning’ – Parts 1-8
2. My book ‘Practical Machine Learning in R and Python: Third edition’ on Amazon
3. Hand detection through Haartraining: A hands-on approach
4.My book ‘Deep Learning from first principles:Second Edition’ now on Amazon
5. Introducing QCSimulator: A 5-qubit quantum computing simulator in R
6. The 3rd paperback & kindle editions of my books on Cricket, now on Amazon

To see all posts click Index of posts

Pitching yorkpy … in the block hole – Part 4

A good programmer is someone who always looks both ways before crossing a one-way street.  Doug Linder

There are two ways to write error-free programs; only the third one works. Alan J. Perlis

In order to understand recursion, one must first understand recursion. Anonymous

This is the fourth and final part of my Python package yorkpy. In this part yorkpy, the python avatar of my R package yorkr see Introducing cricket package yorkr: Part 1- Beaten by sheer pace!, develops wings and is prepared for take-off. The yorkpy package uses data from Cricsheet

You can clone/download the code at Github yorkpy
This post has been published to RPubs at yorkpy-Part4
You can download this post as PDF at IPLT20-yorkpy-part4
You can download all the data used in this post and the previous post at yorkpyData

This post is a continuation of the earlier posts on yorkpy

1. Pitching yorkpy . short of good length to IPL – Part 1 In this part I included functions that convert the yaml data of IPL matches into Pandas dataframe which are then saved as CSV. This part can perform analysis of individual IPL matches. Note The converted data is available at yorkpyData
2. Pitching yorkpy.on the middle and outside off-stump to IPL – Part 2 This part included functions to create a large data frame for head-to-head confrontation between any 2IPL teams says CSK-MI, DD-KKR etc, which can be saved as CSV. Analysis is then performed on these team-2-team confrontations. Note The converted data is available at yorkpyData
3. Pitching yorkpy.swinging away from the leg stump to IPL – Part 3 The 3rd part includes the performance of any IPL team against all other IPL teams. The data can also be saved as CSV.Note The converted data is available at yorkpyData

Note: If you would like to do a similar analysis for a different set of batsman and bowlers, you can clone/download my skeleton yorkpy-template from Github (which is the R Markdown file I have used for the analysis below).

This 4th and final part includes analysis of batting and bowling performances of any IPL player. The batting and bowling details for all teams have already been converted and are available at IPLT20-Batting-BowlingDetails

This part includes the following new functions

Batsman functions

  1. batsmanRunsVsDeliveries
  2. batsmanFoursSixes
  3. batsmanDismissals
  4. batsmanRunsVsStrikeRate
  5. batsmanMovingAverage
  6. batsmanCumulativeAverageRuns
  7. batsmanCumulativeStrikeRate
  8. batsmanRunsAgainstOpposition
  9. batsmanRunsVenue

Bowler functions

  1. bowlerMeanEconomyRate
  2. bowlerMeanRunsConceded
  3. bowlerMovingAverage
  4. bowlerCumulativeAvgWickets
  5. bowlerCumulativeAvgEconRate
  6. bowlerWicketPlot
  7. bowlerWicketsAgainstOpposition
  8. bowlerWicketsVenue

A. Batsman functions

1. Get IPL Team Batting details

The function below gets the overall IPL team batting details based on the CSV files that were saved for IPL T20 matches. This is currently also available in Github at yorkpyData. The batting details of the IPL team in each match is created and a huge data frame is created by combining the batting details from each match. This can be saved as a csv file with name as for e.g. Delhi Daredevils-BattingDetails.csv.

dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
#csk_details = yka.getTeamBattingDetails("Chennai Super Kings",dir=dir1, save=True)
#dd_details = yka.getTeamBattingDetails("Delhi Daredevils",dir=dir1,save=True)
#kkr_details = yka.getTeamBattingDetails("Kolkata Knight Riders",dir=dir1,save=True)

2. Get IPL batsman details

This function is used to get the individual IPL T20 batting record for a the specified batsman of the team as in the functions below.

For the batsmen functions below I have chosen Rishabh Pant, Kane Williamson and Ambati Rayudu for the analysis as they top the batting lists. You can choose any IPL batsmen for the analysis

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
# 1. Rishabh Pant
name="RR Pant"
team='Delhi Daredevils'
rpant=yka.getBatsmanDetails(team,name,dir=dir1)

3 Batsman Runs vs Deliveries (in IPL matches)

This functions plots the runs vs deliveries faced for batsman

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
# 1. Rishabh Pant
name="RR Pant"
team='Delhi Daredevils'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanRunsVsDeliveries(df,name)

# 2. Kane Williamson
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
name="KS Williamson"
team='Sunrisers Hyderabad'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanRunsVsDeliveries(df,name)

#3. Ambati Rayudu
name="AT Rayudu"
team='Mumbai Indians'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanRunsVsDeliveries(df,name)

4. Batsman fours and sixes (in IPL matches)

This plots the fours, sixes and the total runs for a batsman

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
# 1. Rishabh Pant
name="RR Pant"
team='Delhi Daredevils'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanFoursSixes(df,name)


# 2. Kane Williamson
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
name="KS Williamson"
team='Sunrisers Hyderabad'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanFoursSixes(df,name)

#3. Ambati Rayudu
name="AT Rayudu"
team='Mumbai Indians'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanFoursSixes(df,name)

5. Batsman dismissals (in IPL matches)

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
# 1. Rishabh Pant
name="RR Pant"
team='Delhi Daredevils'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanDismissals(df,name)

# 2. Kane Williamson
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
name="KS Williamson"
team='Sunrisers Hyderabad'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanDismissals(df,name)

#3. Ambati Rayudu
name="AT Rayudu"
team='Mumbai Indians'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanDismissals(df,name)

6. Batsman Runs vs Strike Rate (in IPL matches)

The plots below give the Runs vs Strike rate for batsmen

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
# 1. Rishabh Pant
name="RR Pant"
team='Delhi Daredevils'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanRunsVsStrikeRate(df,name)

# 2. Kane Williamson
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
name="KS Williamson"
team='Sunrisers Hyderabad'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanRunsVsStrikeRate(df,name)

#3. Ambati Rayudu
name="AT Rayudu"
team='Mumbai Indians'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanRunsVsStrikeRate(df,name)

7. Batsman Moving average of runs (in IPL matches)

The plots below compute and plot the moving average of batsmen

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
# 1. Rishabh Pant
name="RR Pant"
team='Delhi Daredevils'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanMovingAverage(df,name)

# 2. Kane Williamson
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
name="KS Williamson"
team='Sunrisers Hyderabad'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanMovingAverage(df,name)

#3. Ambati Rayudu
name="AT Rayudu"
team='Mumbai Indians'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanMovingAverage(df,name)

8. Batsman Cumulative average of runs (in IPL matches)

The functions below plot the cumulative average of the batsmen

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
# 1. Rishabh Pant
name="RR Pant"
team='Delhi Daredevils'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanCumulativeAverageRuns(df,name)

# 2. Kane Williamson
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
name="KS Williamson"
team='Sunrisers Hyderabad'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanCumulativeAverageRuns(df,name)

#3. Ambati Rayudu
name="AT Rayudu"
team='Mumbai Indians'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanCumulativeAverageRuns(df,name)

9. Batsman Cumulative Strike Rate (in IPL matches)

The functions below plot the cumulative strike rate of the batsmen

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
# 1. Rishabh Pant
name="RR Pant"
team='Delhi Daredevils'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanCumulativeStrikeRate(df,name)

# 2. Kane Williamson
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
name="KS Williamson"
team='Sunrisers Hyderabad'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanCumulativeStrikeRate(df,name)

#3. Ambati Rayudu
name="AT Rayudu"
team='Mumbai Indians'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanCumulativeStrikeRate(df,name)

10. Batsman performance against opposition (in IPL matches)

The plots below show how the batsmen performed against other IPL teams

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
# 1. Rishabh Pant
name="RR Pant"
team='Delhi Daredevils'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanRunsAgainstOpposition(df,name)

# 2. Kane Williamson
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
name="KS Williamson"
team='Sunrisers Hyderabad'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanRunsAgainstOpposition(df,name)

#3. Ambati Rayudu
name="AT Rayudu"
team='Mumbai Indians'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanRunsAgainstOpposition(df,name)

11. Batsman performance at different venues (in IPL matches)

The plots below show how the batsmen performed at different venues

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
# 1. Rishabh Pant
name="RR Pant"
team='Delhi Daredevils'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanRunsVenue(df,name)

# 2. Kane Williamson
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
name="KS Williamson"
team='Sunrisers Hyderabad'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanRunsVenue(df,name)

#3. Ambati Rayudu
name="AT Rayudu"
team='Mumbai Indians'
df=yka.getBatsmanDetails(team,name,dir=dir1)
yka.batsmanRunsVenue(df,name)

B. Bowler functions

12. Get bowling details in IPL matches

The function below gets the overall team IPL T20 bowling details based on the RData file available in IPL T20 matches. This is currently also available in Github at yorkpyData. The IPL T20 bowling details of the IPL team in each match is created, and a huge data frame is created by stacking the individual dataframes. This can be saved as a CSV file for e.g. Chennai Super Kings-BowlingDetails.csv

dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
#kkr_bowling = yka.getTeamBowlingDetails("Kolkata Knight Riders",dir=dir1,save=True)
#csk_bowling = yka.getTeamBowlingDetails("Chennai Super Kings",dir=dir1,save=True)
#kxip_bowling = yka.getTeamBowlingDetails("Kings XI Punjab",dir=dir1,save=True)

13. Get bowling details of the individual IPL bowlers

This function is used to get the individual bowling record for a specified bowler of the country as in the functions below.

The plots below deal with bowler’s performance. For this analysis I have chosen Amit Mishra, Piyush Chawla and Bhuvaneshwar Kumar for the analysis. You can chose any other IPL bowler

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
# 1. Amit Mishra
name="A Mishra"
team='Delhi Daredevils'
#df=yka.getBowlerWicketDetails(team,name,dir=dir1)

14. Bowler Economy Rate (in IPL matches)

The plots below show the economy rate of the selected bowlers

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
# 1. Amit Mishra
name="A Mishra"
team='Delhi Daredevils'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerMeanEconomyRate(df,name)

# 2. Piyush Chawla
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
name="PP Chawla"
team='Kolkata Knight Riders'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerMeanEconomyRate(df,name)

#3. Bhuvneshwar Kumar
name="B Kumar"
team='Sunrisers Hyderabad'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerMeanEconomyRate(df,name)

15. Bowler Mean Runs conceded (in IPL matches)

The plots below show the mean runs conceded by the selected bowlers

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
# 1. Amit Mishra
name="A Mishra"
team='Delhi Daredevils'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerMeanRunsConceded(df,name)

# 2. Piyush Chawla
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
name="PP Chawla"
team='Kolkata Knight Riders'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerMeanRunsConceded(df,name)

#3. Bhuvneshwar Kumar
name="B Kumar"
team='Sunrisers Hyderabad'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerMeanRunsConceded(df,name)

16. Moving average of wickets for bowler (in IPL matches)

The moving average of the bowlers are plotted below

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
# 1. Amit Mishra
name="A Mishra"
team='Delhi Daredevils'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerMovingAverage(df,name)

# 2. Piyush Chawla
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
name="PP Chawla"
team='Kolkata Knight Riders'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerMovingAverage(df,name)

#3. Bhuvneshwar Kumar
name="B Kumar"
team='Sunrisers Hyderabad'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerMovingAverage(df,name)

17. Cumulative average wickets for bowler (in IPL matches)

The cumulative average wickets for each bowler is computed and plotted

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
# 1. Amit Mishra
name="A Mishra"
team='Delhi Daredevils'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerCumulativeAvgWickets(df,name)

# 2. Piyush Chawla
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
name="PP Chawla"
team='Kolkata Knight Riders'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerCumulativeAvgWickets(df,name)

#3. Bhuvneshwar Kumar
name="B Kumar"
team='Sunrisers Hyderabad'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerCumulativeAvgWickets(df,name)

18. Cumulative average economy rate for bowler (in IPL matches)

The plots below give the cumulative average economy rate for each bowler

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
# 1. Amit Mishra
name="A Mishra"
team='Delhi Daredevils'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerCumulativeAvgEconRate(df,name)

# 2. Piyush Chawla
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
name="PP Chawla"
team='Kolkata Knight Riders'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerCumulativeAvgEconRate(df,name)

#3. Bhuvneshwar Kumar
name="B Kumar"
team='Sunrisers Hyderabad'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerCumulativeAvgEconRate(df,name)

19. Bowler wicket plot (in IPL matches)

The plots below give the over vs wickets for bowlers

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
# 1. Amit Mishra
name="A Mishra"
team='Delhi Daredevils'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerWicketPlot(df,name)

# 2. Piyush Chawla
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
name="PP Chawla"
team='Kolkata Knight Riders'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerWicketPlot(df,name)

#3. Bhuvneshwar Kumar
name="B Kumar"
team='Sunrisers Hyderabad'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerWicketPlot(df,name)

20. Bowler wicket against opposition (in IPL matches)

The performance of the bowlers against different IPL teams is shown below

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
# 1. Amit Mishra
name="A Mishra"
team='Delhi Daredevils'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerWicketsAgainstOpposition(df,name)

# 2. Piyush Chawla
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
name="PP Chawla"
team='Kolkata Knight Riders'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerWicketsAgainstOpposition(df,name)

#3. Bhuvneshwar Kumar
name="B Kumar"
team='Sunrisers Hyderabad'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerWicketsAgainstOpposition(df,name)

21. Bowler wicket in different venues (in IPL matches)

The plots below show how the bowlers perform at different venues

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
# 1. Amit Mishra
name="A Mishra"
team='Delhi Daredevils'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerWicketsVenue(df,name)

# 2. Piyush Chawla
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data3"
name="PP Chawla"
team='Kolkata Knight Riders'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerWicketsVenue(df,name)

#3. Bhuvneshwar Kumar
name="B Kumar"
team='Sunrisers Hyderabad'
df=yka.getBowlerWicketDetails(team,name,dir=dir1)
yka.bowlerWicketsVenue(df,name)

Note:You can clone/download the code at Github yorkpy

Important note: Do check out my other posts using yorkpy at yorkpy-posts

Conclusion: This concludes the python package yorkpy. Go ahead and give yorkpy a spin!

Also see
1. Take 4+: Presentations on ‘Elements of Neural Networks and Deep Learning’ – Parts 1-8
2. My book ‘Practical Machine Learning in R and Python: Third edition’ on Amazon
3. Hand detection through Haartraining: A hands-on approach
4.My book ‘Deep Learning from first principles:Second Edition’ now on Amazon
5. Big Data-1: Move into the big league:Graduate from Python to Pyspark
6. Cricpy takes a swing at the ODIs

To see all posts click Index of posts

Take 4+: Presentations on ‘Elements of Neural Networks and Deep Learning’ – Parts 1-8

“Lights, camera and … action – Take 4+!”

This post includes  a rework of all presentation of ‘Elements of Neural Networks and Deep  Learning Parts 1-8 ‘ since my earlier presentations had some missing parts, omissions and some occasional errors. So I have re-recorded all the presentations.
This series of presentation will do a deep-dive  into Deep Learning networks starting from the fundamentals. The equations required for performing learning in a L-layer Deep Learning network  are derived in detail, starting from the basics. Further, the presentations also discuss multi-class classification, regularization techniques, and gradient descent optimization methods in deep networks methods. Finally the presentations also touch on how  Deep Learning Networks can be tuned.

The corresponding implementations are available in vectorized R, Python and Octave are available in my book ‘Deep Learning from first principles:Second edition- In vectorized Python, R and Octave

1. Elements of Neural Networks and Deep Learning – Part 1
This presentation introduces Neural Networks and Deep Learning. A look at history of Neural Networks, Perceptrons and why Deep Learning networks are required and concluding with a simple toy examples of a Neural Network and how they compute. This part also includes a small digression on the basics of Machine Learning and how the algorithm learns from a data set

2. Elements of Neural Networks and Deep Learning – Part 2
This presentation takes logistic regression as an example and creates an equivalent 2 layer Neural network. The presentation also takes a look at forward & backward propagation and how the cost is minimized using gradient descent


The implementation of the discussed 2 layer Neural Network in vectorized R, Python and Octave are available in my post ‘Deep Learning from first principles in Python, R and Octave – Part 1‘

3. Elements of Neural Networks and Deep Learning – Part 3
This 3rd part, discusses a primitive neural network with an input layer, output layer and a hidden layer. The neural network uses tanh activation in the hidden layer and a sigmoid activation in the output layer. The equations for forward and backward propagation are derived.


To see the implementations for the above discussed video see my post ‘Deep Learning from first principles in Python, R and Octave – Part 2

4. Elements of Neural Network and Deep Learning – Part 4
This presentation is a continuation of my 3rd presentation in which I derived the equations for a simple 3 layer Neural Network with 1 hidden layer. In this video presentation, I discuss step-by-step the derivations for a L-Layer, multi-unit Deep Learning Network, with any activation function g(z)


The implementations of L-Layer, multi-unit Deep Learning Network in vectorized R, Python and Octave are available in my post Deep Learning from first principles in Python, R and Octave – Part 3

5. Elements of Neural Network and Deep Learning – Part 5
This presentation discusses multi-class classification using the Softmax function. The detailed derivation for the Jacobian of the Softmax is discussed, and subsequently the derivative of cross-entropy loss is also discussed in detail. Finally the final set of equations for a Neural Network with multi-class classification is derived.


The corresponding implementations in vectorized R, Python and Octave are available in the following posts
a. Deep Learning from first principles in Python, R and Octave – Part 4
b. Deep Learning from first principles in Python, R and Octave – Part 5

6. Elements of Neural Networks and Deep Learning – Part 6
This part discusses initialization methods specifically like He and Xavier. The presentation also focuses on how to prevent over-fitting using regularization. Lastly the dropout method of regularization is also discussed


The corresponding implementations in vectorized R, Python and Octave of the above discussed methods are available in my post Deep Learning from first principles in Python, R and Octave – Part 6

7. Elements of Neural Networks and Deep Learning – Part 7
This presentation introduces exponentially weighted moving average and shows how this is used in different approaches to gradient descent optimization. The key techniques discussed are learning rate decay, momentum method, rmsprop and adam.

The equivalent implementations of the gradient descent optimization techniques in R, Python and Octave can be seen in my post Deep Learning from first principles in Python, R and Octave – Part 7

8. Elements of Neural Networks and Deep Learning – Part 8
This last part touches on the method to adopt while tuning hyper-parameters in Deep Learning networks

Checkout my book ‘Deep Learning from first principles: Second Edition – In vectorized Python, R and Octave’. My book starts with the implementation of a simple 2-layer Neural Network and works its way to a generic L-Layer Deep Learning Network, with all the bells and whistles. The derivations have been discussed in detail. The code has been extensively commented and included in its entirety in the Appendix sections. My book is available on Amazon as paperback ($18.99) and in kindle version($9.99/Rs449).

This concludes this series of presentations on “Elements of Neural Networks and Deep Learning’

Also
1. My book ‘Practical Machine Learning in R and Python: Third edition’ on Amazon
2. Introducing cricpy:A python package to analyze performances of cricketers
3. Natural language processing: What would Shakespeare say?
4. Big Data-2: Move into the big league:Graduate from R to SparkR
5. Presentation on Wireless Technologies – Part 1
6. Introducing cricketr! : An R package to analyze performances of cricketers

To see all posts click Index of posts

Pitching yorkpy…swinging away from the leg stump to IPL – Part 3

Clocks offer at best a convenient fiction They imply that time ticks steadily, predictably forward, when our experience shows that it often does the opposite: it stretches and compresses, skips a beat and doubles back.

                                 David Eagleman
                                 

Memory is the space in which a thing happens for a second time

                                 Paul Auster
      

Introduction

In this 3rd post, yorkpy, the python avatar of my R package yorkr develops more muscle. The first two posts of yorkpy were

1. Pitching yorkpy . short of good length to IPL – Part 1 This post dealt with function which perform analytics on an IPL match between any 2 IPL teams
2. Pitching yorkpy…on the middle and outside off-stump to IPL – Part 2 The second post dealt with analytics on all matches between any 2 IPL teams.

This third post deals with analyses and analytics of an IPL team in all matches against all other IPL teams. The data for yorkpy comes from Cricsheet. The data in Cricsheet are in the form of yaml files. These files have already been converted as dataframes and stored as CSV as seen in the earlier posts.You can download all the data used in this post and the previous post at yorkpyData

The signatures of yorkpy and yorkr are identical and will work in almost the same way. However there may be some unique functions in yorkr & yorkpy, based on what my thought process was on that day!

-You can clone/download the code at Github yorkpy
-This post has been published to RPubs at yorkpy-Part3
-Download this post as PDF at IPLT20-yorkpy-part3
-You can download all the data used in this post and the previous post at yorkpyData

Note: If you would like to do a similar analysis for a different set of batsman and bowlers, you can clone/download my skeleton yorkpy-template from Github (which is the R Markdown file I have used for the analysis below).

The IPL T20 functions in yorkpy are shown below

2. Get data for all T20 matches between an IPL team and all other IPL teams

We can get all IPL T20 matches between an IPL team  and all other teams using the function below. The dir parameter should point to the folder which has the IPL T20 csv files of the individual matches (see Pitching yorkpy…short of good length to IPL-Part 1). This function creates a data frame of all the IPL T20 matches between the IPL team and all other teams and and also saves the dataframe as CSV file if save=True. If save=False the dataframe is just returned and not saved.

import pandas as pd
import os
import yorkpy.analytics as yka
#dir1= "C:\\software\\cricket-package\\yorkpyPkg\\yorkpyData\\IPLConverted"
#getAllMatchesAllOpposition("Kolkata Knight Riders",dir=dir1,save=True)

3. Save data for all matches between an IPL team and all oppositions

This can be done locally using the function below. You could use this function to get combine all IPL matches of an IPL team against all other IPL teams

import pandas as pd
import os
import yorkpy.analytics as yka
#dir1= "C:\\software\\cricket-package\\yorkpyPkg\\yorkpyData\\IPLConverted"
#saveAllMatchesAllOppositionIPLT20(dir1)

Note: In the functions below, I have randomly chosen an IPL team for the analyses. You are free to choose any IPL team for your analysis

4.Team Batsmen partnership in Twenty20 (all matches against all IPL teams – summary)

The function below computes the highest partnerships for an IPL team against all other IPL teams for e.g. the batsmen with the highest partnership from Chennai Super Kings in all matches against all other IPL teams. Any other IPL team could have also been chosen.

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data2"
path=os.path.join(dir1,"Chennai Super Kings-allMatchesAllOpposition.csv") 
csk_matches = pd.read_csv(path)
m=yka.teamBatsmenPartnershiAllOppnAllMatches(csk_matches,'Chennai Super Kings',report="summary")
print(m)
##         batsman  totalPartnershipRuns
## 42     SK Raina                  3699
## 28     MS Dhoni                  2986
## 25   MEK Hussey                  1768
## 24      M Vijay                  1600
## 36  S Badrinath                  1441

5. Team Batsmen partnership in Twenty20 (all matches against all IPL teams -detailed)

The function below gives the detailed breakup of partnerships for Mumbai Indian against all other IPL teams

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data2"
path=os.path.join(dir1,"Mumbai Indians-allMatchesAllOpposition.csv")
mi_matches = pd.read_csv(path)
theTeam='Mumbai Indians'
m=yka.teamBatsmenPartnershiAllOppnAllMatches(mi_matches,theTeam,report="detailed", top=3)
print(m)
##        batsman  totalPartnershipRuns      non_striker  partnershipRuns
## 0    RG Sharma                3037.0        A Symonds            142.0
## 1    RG Sharma                3037.0      AC Blizzard              5.0
## 2    RG Sharma                3037.0         AJ Finch              2.0
## 3    RG Sharma                3037.0          AP Tare             32.0
## 4    RG Sharma                3037.0        AT Rayudu            566.0
## 5    RG Sharma                3037.0          BR Dunk              1.0
## 6    RG Sharma                3037.0      CJ Anderson            183.0
## 7    RG Sharma                3037.0        CM Gautam             22.0
## 8    RG Sharma                3037.0         DR Smith             50.0
## 9    RG Sharma                3037.0       GJ Maxwell              6.0
## 10   RG Sharma                3037.0         HH Gibbs            109.0
## 11   RG Sharma                3037.0        HH Pandya            105.0
## 12   RG Sharma                3037.0  Harbhajan Singh             86.0
## 13   RG Sharma                3037.0       JC Buttler            105.0
## 14   RG Sharma                3037.0     JEC Franklin             50.0
## 15   RG Sharma                3037.0       KA Pollard            633.0
## 16   RG Sharma                3037.0       KD Karthik            170.0
## 17   RG Sharma                3037.0        KH Pandya             34.0
## 18   RG Sharma                3037.0        KV Sharma             33.0
## 19   RG Sharma                3037.0      LMP Simmons            172.0
## 20   RG Sharma                3037.0       MEK Hussey             21.0
## 21   RG Sharma                3037.0       MJ Guptill             61.0
## 22   RG Sharma                3037.0   MJ McClenaghan              2.0
## 23   RG Sharma                3037.0           N Rana             25.0
## 24   RG Sharma                3037.0         PA Patel            103.0
## 25   RG Sharma                3037.0          RE Levi             25.0
## 26   RG Sharma                3037.0       SL Malinga              0.0
## 27   RG Sharma                3037.0     SR Tendulkar            208.0
## 28   RG Sharma                3037.0        SS Tiwary             27.0
## 29   RG Sharma                3037.0         TL Suman              7.0
## ..         ...                   ...              ...              ...
## 70  KA Pollard                2344.0      CJ Anderson             82.0
## 71  KA Pollard                2344.0        CM Gautam             16.0
## 72  KA Pollard                2344.0         DR Smith             10.0
## 73  KA Pollard                2344.0      DS Kulkarni             15.0
## 74  KA Pollard                2344.0        HH Pandya            158.0
## 75  KA Pollard                2344.0  Harbhajan Singh            158.0
## 76  KA Pollard                2344.0        J Suchith             26.0
## 77  KA Pollard                2344.0       JC Buttler             37.0
## 78  KA Pollard                2344.0     JEC Franklin             38.0
## 79  KA Pollard                2344.0        JP Duminy             63.0
## 80  KA Pollard                2344.0       KD Karthik             40.0
## 81  KA Pollard                2344.0        KH Pandya            111.0
## 82  KA Pollard                2344.0        KV Sharma             13.0
## 83  KA Pollard                2344.0      LMP Simmons             77.0
## 84  KA Pollard                2344.0       MEK Hussey             10.0
## 85  KA Pollard                2344.0       MG Johnson              1.0
## 86  KA Pollard                2344.0           N Rana             60.0
## 87  KA Pollard                2344.0         PA Patel             18.0
## 88  KA Pollard                2344.0          PP Ojha             12.0
## 89  KA Pollard                2344.0         R Dhawan             25.0
## 90  KA Pollard                2344.0        R McLaren             20.0
## 91  KA Pollard                2344.0        R Sathish             27.0
## 92  KA Pollard                2344.0        RG Sharma            587.0
## 93  KA Pollard                2344.0      RJ Peterson              0.0
## 94  KA Pollard                2344.0         S Dhawan             20.0
## 95  KA Pollard                2344.0       SL Malinga             14.0
## 96  KA Pollard                2344.0     SR Tendulkar             69.0
## 97  KA Pollard                2344.0        SS Tiwary             42.0
## 98  KA Pollard                2344.0         TL Suman              2.0
## 99  KA Pollard                2344.0           Z Khan              1.0
## 
## [100 rows x 4 columns]

6. Team Batsmen partnership in Twenty20 – Chart (all matches against all IPL teams)

The function below plots the partnerships of an IPL team against all other IPL teams

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data2"
path=os.path.join(dir1,"Delhi Daredevils-allMatchesAllOpposition.csv")
dd_matches = pd.read_csv(path)
yka.teamBatsmenPartnershipAllOppnAllMatchesChart(dd_matches,'Delhi Daredevils', plot=True, top=4, partnershipRuns=100)

7.Team Batsmen partnership in Twenty20 – Dataframe (all matches against all IPL teams)

This function does not plot the data but returns the dataframe to the user to plot or manipulate.

Note: Many of the plots include an additional parameters for e.g. plot which is either True or False. The default value is plot=True. When plot=True the plot will be displayed. When plot=False the data frame will be returned to the user. The user can use this to create an interactive charts. The parameter top= specifies the number of top batsmen that need to be included in the chart, and partnershipRuns gives the minimum cutoff runs in partnwerships to be considered

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data2"
path=os.path.join(dir1,"Kochi Tuskers Kerala-allMatchesAllOpposition.csv")
ktk_matches = pd.read_csv(path)
m=yka.teamBatsmenPartnershipAllOppnAllMatchesChart(ktk_matches,'Kochi Tuskers Kerala', plot=False, top=3, partnershipRuns=100)
print(m)
##              batsman       non_striker  partnershipRuns
## 0        BB McCullum          BJ Hodge             17.0
## 1        BB McCullum  DPMD Jayawardene            160.0
## 2        BB McCullum         M Klinger             67.0
## 3        BB McCullum          PA Patel             40.0
## 4        BB McCullum         RA Jadeja             19.0
## 5        BB McCullum        VVS Laxman             41.0
## 6        BB McCullum  Y Gnaneswara Rao             13.0
## 7   DPMD Jayawardene       BB McCullum            152.0
## 8   DPMD Jayawardene          BJ Hodge             41.0
## 9   DPMD Jayawardene         KM Jadhav              4.0
## 10  DPMD Jayawardene         M Klinger             28.0
## 11  DPMD Jayawardene           OA Shah              9.0
## 12  DPMD Jayawardene          PA Patel             25.0
## 13  DPMD Jayawardene         RA Jadeja             18.0
## 14  DPMD Jayawardene          RV Gomez             10.0
## 15  DPMD Jayawardene        VVS Laxman             12.0
## 16          BJ Hodge       BB McCullum             18.0
## 17          BJ Hodge  DPMD Jayawardene             47.0
## 18          BJ Hodge         KM Jadhav              2.0
## 19          BJ Hodge           OA Shah             19.0
## 20          BJ Hodge          PA Patel             79.0
## 21          BJ Hodge         RA Jadeja             99.0
## 22          BJ Hodge          RV Gomez             21.0

8. Team batsmen versus bowler in Twenty20-Chart (all matches against all IPL teams)

The plots below provide information on how each of the top batsmen of the IPL team fared against the opposition bowlers of all other IPL teams.

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data2"
path=os.path.join(dir1,"Royal Challengers Bangalore-allMatchesAllOpposition.csv")
rcb_matches = pd.read_csv(path)
yka.teamBatsmenVsBowlersAllOppnAllMatches(rcb_matches,"Royal Challengers Bangalore",plot=True,top=3,runsScored=60)

9 Team batsmen versus bowler in Twenty20-Dataframe (all matches against all IPL teams)

This function provides the batting performance of an IPL team against all other IPL teams

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data2"
path=os.path.join(dir1,"Kings XI Punjab-allMatchesAllOpposition.csv")
kxip_matches = pd.read_csv(path)
m=yka.teamBatsmenVsBowlersAllOppnAllMatches(kxip_matches,'Kings XI Punjab',plot=False,top=2,runsScored=50)
print(m)
##        batsman            bowler  runsScored
## 0     SE Marsh        A Chandila        20.0
## 1     SE Marsh       A Choudhary         1.0
## 2     SE Marsh          A Kumble        37.0
## 3     SE Marsh          A Mishra         0.0
## 4     SE Marsh          A Mithun         9.0
## 5     SE Marsh           A Nehra        33.0
## 6     SE Marsh           A Singh         2.0
## 7     SE Marsh         A Symonds         5.0
## 8     SE Marsh         AA Chavan        19.0
## 9     SE Marsh   AA Jhunjhunwala        15.0
## 10    SE Marsh        AB Agarkar        27.0
## 11    SE Marsh          AB Dinda        31.0
## 12    SE Marsh       AB McDonald         9.0
## 13    SE Marsh         AC Thomas         1.0
## 14    SE Marsh        AD Mathews         7.0
## 15    SE Marsh        AD Russell         8.0
## 16    SE Marsh            AJ Tye         0.0
## 17    SE Marsh        AL Menaria         6.0
## 18    SE Marsh          AM Salvi         8.0
## 19    SE Marsh          AN Ahmed        16.0
## 20    SE Marsh           AS Raut         7.0
## 21    SE Marsh      Ankit Sharma         2.0
## 22    SE Marsh        Ankit Soni        11.0
## 23    SE Marsh           B Kumar        10.0
## 24    SE Marsh             B Lee         1.0
## 25    SE Marsh        BAW Mendis        11.0
## 26    SE Marsh           BB Sran         3.0
## 27    SE Marsh          BJ Hodge        18.0
## 28    SE Marsh      Basil Thampi        17.0
## 29    SE Marsh   C de Grandhomme         8.0
## ..         ...               ...         ...
## 235  DA Miller          R Sharma         7.0
## 236  DA Miller         R Tewatia         3.0
## 237  DA Miller     R Vinay Kumar        30.0
## 238  DA Miller         RA Jadeja        84.0
## 239  DA Miller         RD Chahar         3.0
## 240  DA Miller  RE van der Merwe         5.0
## 241  DA Miller  RN ten Doeschate         1.0
## 242  DA Miller          RP Singh        35.0
## 243  DA Miller       Rashid Khan         0.0
## 244  DA Miller         S Aravind         7.0
## 245  DA Miller            S Kaul        23.0
## 246  DA Miller         S Kaushik         8.0
## 247  DA Miller           S Ladda         6.0
## 248  DA Miller          S Nadeem        11.0
## 249  DA Miller          SK Raina         2.0
## 250  DA Miller        SL Malinga         9.0
## 251  DA Miller   SMSM Senanayake         6.0
## 252  DA Miller         SP Narine        10.0
## 253  DA Miller         SR Watson        16.0
## 254  DA Miller         STR Binny        14.0
## 255  DA Miller   Shakib Al Hasan         3.0
## 256  DA Miller          TA Boult        20.0
## 257  DA Miller        TG Southee        11.0
## 258  DA Miller          UT Yadav        51.0
## 259  DA Miller          VR Aaron        19.0
## 260  DA Miller          VS Malik         3.0
## 261  DA Miller         YK Pathan         0.0
## 262  DA Miller         YS Chahal        35.0
## 263  DA Miller      Yuvraj Singh        11.0
## 264  DA Miller            Z Khan         2.0
## 
## [265 rows x 3 columns]

10. Team batting scorecard(all matches against all IPL teams)

This function provides the overall scorecard for an IPL team in all matches against all other IPL teams. The batting scorecard shows the top batsmen for Kolkata Knight Riders below

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data2"
path=os.path.join(dir1,"Kolkata Knight Riders-allMatchesAllOpposition.csv")
kkr_matches = pd.read_csv(path)
scorecard=yka.teamBattingScorecardAllOppnAllMatches(kkr_matches,'Kolkata Knight Riders')
print(scorecard)
##              batsman    runs  balls   4s  6s          SR
## 19         G Gambhir  3035.0   2533  352  46  119.818397
## 17         YK Pathan  1893.0   1421  150  86  133.216045
## 22        RV Uthappa  1806.0   1311  200  54  137.757437
## 16         JH Kallis  1295.0   1237  128  23  104.688763
## 23         MK Pandey  1270.0   1048  103  38  121.183206
## 0         SC Ganguly  1031.0    977  105  36  105.527124
## 12         MK Tiwary  1002.0    921   86  23  108.794788
## 1        BB McCullum   882.0    754   92  32  116.976127
## 25          SA Yadav   608.0    474   54  21  128.270042
## 15          MS Bisla   543.0    518   60  16  104.826255
## 26        AD Russell   516.0    308   45  34  167.532468
## 4          DJ Hussey   511.0    417   31  28  122.541966
## 24   Shakib Al Hasan   498.0    399   44  15  124.812030
## 10          BJ Hodge   476.0    430   47  10  110.697674
## 11          CH Gayle   463.0    350   45  26  132.285714
## 18        EJG Morgan   444.0    373   45  16  119.034853
## 54           CA Lynn   378.0    250   30  23  151.200000
## 6          LR Shukla   374.0    320   31  15  116.875000
## 29  RN ten Doeschate   326.0    238   26  15  136.974790
## 21            DB Das   304.0    267   23  16  113.857678
## 3            WP Saha   298.0    213   24  12  139.906103
## 28         SP Narine   271.0    193   36  12  140.414508
## 13        AD Mathews   249.0    211   20   8  118.009479
## 33       Salman Butt   193.0    172   30   2  112.209302
## 41        MN van Wyk   167.0    135   19   1  123.703704
## 7         AB Agarkar   160.0    137   12   5  116.788321
## 20          R Bhatia   159.0    134   15   3  118.656716
## 51   C de Grandhomme   126.0     92   10   6  136.956522
## 39         CA Pujara   122.0    119   14   3  102.521008
## 40           OA Shah   115.0     96    7   5  119.791667
## ..               ...     ...    ...  ...  ..         ...
## 50         JO Holder    22.0     20    2   1  110.000000
## 65     Kuldeep Yadav    20.0     22    2   0   90.909091
## 71         BJ Haddin    18.0     11    2   1  163.636364
## 70   NM Coulter-Nile    14.0     13    0   2  107.692308
## 47          L Balaji    13.0     12    1   0  108.333333
## 55   SMSM Senanayake    10.0     17    0   0   58.823529
## 53          M Morkel     9.0      8    0   0  112.500000
## 62          AN Ghosh     7.0      8    1   0   87.500000
## 32           GB Hogg     7.0      6    0   0  116.666667
## 56        MV Boucher     6.0      6    0   0  100.000000
## 77     Azhar Mahmood     6.0      8    1   0   75.000000
## 78          DM Bravo     6.0      5    1   0  120.000000
## 68         SS Shaikh     6.0      7    1   0   85.714286
## 66          TA Boult     5.0      8    0   0   62.500000
## 76    Mohammed Shami     5.0     10    0   0   50.000000
## 80           P Dogra     5.0      8    0   0   62.500000
## 69     R Vinay Kumar     4.0      7    0   0   57.142857
## 75        AS Rajpoot     4.0      7    1   0   57.142857
## 43     Mandeep Singh     4.0     11    1   0   36.363636
## 37          AB Dinda     4.0      8    0   0   50.000000
## 79        PJ Sangwan     4.0      2    1   0  200.000000
## 73         R McLaren     3.0      6    0   0   50.000000
## 67         SB Bangar     2.0      9    0   0   22.222222
## 57       RS Gavaskar     2.0      8    0   0   25.000000
## 72     Shoaib Akhtar     2.0      8    0   0   25.000000
## 38  Mashrafe Mortaza     2.0      2    0   0  100.000000
## 63        BAW Mendis     1.0      2    0   0   50.000000
## 58           SE Bond     1.0      2    0   0   50.000000
## 44     CK Langeveldt     0.0      1    0   0    0.000000
## 30        PJ Cummins     0.0      2    0   0    0.000000
## 
## [81 rows x 6 columns]

10a. Team batting scorecard(all matches against all IPL teams)

The output below shows the Chennai Super Kings against all other IPL teams

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data2"
path=os.path.join(dir1,"Chennai Super Kings-allMatchesAllOpposition.csv")
csk_matches = pd.read_csv(path)
scorecard=yka.teamBattingScorecardAllOppnAllMatches(csk_matches,'Chennai Super Kings')
print(scorecard)
##             batsman  runs  balls   4s   6s          SR
## 3          SK Raina  3699   2735  322  150  135.246801
## 5          MS Dhoni  2986   2199  218  126  135.788995
## 17       MEK Hussey  1768   1461  181   45  121.013005
## 11          M Vijay  1600   1289  141   66  124.127230
## 4       S Badrinath  1441   1245  154   28  115.742972
## 9         ML Hayden  1107    838  121   44  132.100239
## 18     F du Plessis  1081    867   92   29  124.682814
## 25         DR Smith   965    766  102   50  125.979112
## 26      BB McCullum   841    634   83   42  132.649842
## 6         JA Morkel   827    591   51   48  139.932318
## 20         DJ Bravo   706    543   54   30  130.018416
## 19        RA Jadeja   670    533   46   23  125.703565
## 0          PA Patel   516    529   67    7   97.542533
## 2        SP Fleming   196    171   27    3  114.619883
## 13         R Ashwin   190    208   19    1   91.346154
## 21         S Vidyut   145    115   21    3  126.086957
## 31          WP Saha   144    138    8    8  104.347826
## 1        S Anirudha   133    116    9    7  114.655172
## 33        DJ Hussey   116     96    8    6  120.833333
## 38           P Negi   116     77   10    5  150.649351
## 10         JDP Oram   106    107    6    5   99.065421
## 29        GJ Bailey    63     67    9    0   94.029851
## 22       A Flintoff    62     57    5    2  108.771930
## 8           MS Gony    50     39    2    5  128.205128
## 7   Joginder Sharma    36     30    1    2  120.000000
## 27         M Manhas    35     26    3    1  134.615385
## 28        MM Sharma    29     26    1    2  111.538462
## 23        SB Jakati    27     28    3    0   96.428571
## 12          JM Kemp    26     25    1    1  104.000000
## 14         L Balaji    22     35    1    1   62.857143
## 24     DE Bollinger    21     23    1    1   91.304348
## 41    CK Kapugedera    16     24    0    0   66.666667
## 37        CH Morris    14     17    0    0   82.352941
## 30       T Thushara    12     19    0    0   63.157895
## 42          M Ntini    11     19    2    0   57.894737
## 15   M Muralitharan     9     13    1    0   69.230769
## 32  KMDN Kulasekara     5      3    1    0  166.666667
## 34        SB Styris     5      2    1    0  250.000000
## 35       B Laughlin     4      9    0    0   44.444444
## 16          S Tyagi     3      4    0    0   75.000000
## 45  KB Arun Karthik     3      5    0    0   60.000000
## 36       AS Rajpoot     2      6    0    0   33.333333
## 43          RG More     2      2    0    0  100.000000
## 44         S Randiv     2      4    0    0   50.000000
## 39          A Nehra     1      7    0    0   14.285714
## 40         A Mukund     0      1    0    0    0.000000

11.Team Bowling scorecard (all matches against all IPL teams)

The output below gives the bowling performance of an IPL team against all other IPL teams

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data2"
path=os.path.join(dir1,"Sunrisers Hyderabad-allMatchesAllOpposition.csv")
srh_matches = pd.read_csv(path)
scorecard=yka.teamBowlingScorecardAllOppnAllMatches(srh_matches,'Sunrisers Hyderabad')
## C:\Users\Ganesh\ANACON~1\lib\site-packages\yorkpy\analytics.py:564: SettingWithCopyWarning: 
## A value is trying to be set on a copy of a slice from a DataFrame.
## Try using .loc[row_indexer,col_indexer] = value instead
## 
## See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
##   df1['over']=df1.delivery.astype(int)
## C:\Users\Ganesh\ANACON~1\lib\site-packages\yorkpy\analytics.py:567: SettingWithCopyWarning: 
## A value is trying to be set on a copy of a slice from a DataFrame.
## Try using .loc[row_indexer,col_indexer] = value instead
## 
## See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
##   df1['runsConceded']=df1['runs'] + df1['wides'] + df1['noballs']
print(scorecard)
##               bowler  overs  runs  maidens  wicket   econrate
## 60       JP Faulkner     28   192        0      15   6.857143
## 83         MM Sharma     37   334        0      13   9.027027
## 119       SL Malinga     31   215        0      13   6.935484
## 123        SR Watson     30   281        0      13   9.366667
## 90   NM Coulter-Nile     24   166        0      12   6.916667
## 31          DJ Bravo     26   184        0      12   7.076923
## 135         UT Yadav     37   297        0      12   8.027027
## 125   Sandeep Sharma     32   280        0      11   8.750000
## 75          M Morkel     25   195        0       9   7.800000
## 81    MJ McClenaghan     24   175        0       9   7.291667
## 5           AB Dinda     23   165        0       9   7.173913
## 55        JD Unadkat     20   167        0       8   8.350000
## 36       DS Kulkarni     28   200        0       8   7.142857
## 25         CH Morris     24   190        0       7   7.916667
## 101         R Bhatia     18   128        0       7   7.111111
## 70     Kuldeep Yadav     16   129        0       7   8.062500
## 11          AR Patel     27   208        0       7   7.703704
## 122        SP Narine     43   282        0       7   6.558140
## 141        YS Chahal     26   224        0       6   8.615385
## 44   Harbhajan Singh     39   264        0       6   6.769231
## 96         PP Chawla     21   140        0       6   6.666667
## 4            A Zampa      4    19        0       6   4.750000
## 126  Shakib Al Hasan     14    99        1       6   7.071429
## 80        MG Johnson     20   155        0       6   7.750000
## 59         JP Duminy     10    80        0       5   8.000000
## 58         JO Holder     15   113        0       5   7.533333
## 92           P Kumar     23   173        0       5   7.521739
## 100         R Ashwin     28   142        0       5   5.071429
## 2           A Mishra     18   144        0       4   8.000000
## 106    R Vinay Kumar     19   154        0       4   8.105263
## ..               ...    ...   ...      ...     ...        ...
## 6     AD Mascarenhas      4    25        0       0   6.250000
## 13        Ankit Soni      2    31        0       0  15.500000
## 132          TM Head      1    11        0       0  11.000000
## 10          AN Ahmed      6    63        0       0  10.500000
## 131       TM Dilshan      1    10        0       0  10.000000
## 134     Tejas Baroka      3    33        0       0  11.000000
## 73          M Ashwin      1     6        0       0   6.000000
## 109        RG Sharma      1     5        0       0   5.000000
## 22      Basil Thampi      2    21        0       0  10.500000
## 23           C Munro      1     8        0       0   8.000000
## 68         KV Sharma      2    19        0       0   9.500000
## 77           M Vijay      4    24        0       0   6.000000
## 66         KJ Abbott      3    34        0       0  11.333333
## 65         KH Pandya      2    17        0       0   8.500000
## 82          MM Patel      3    22        0       0   7.333333
## 62          K Rabada      4    59        0       0  14.750000
## 85        MP Stoinis      3    28        0       0   9.333333
## 54         JA Morkel      3    35        0       0  11.666667
## 46          I Sharma      8    64        0       0   8.000000
## 94        PJ Cummins      4    37        0       0   9.250000
## 95        PJ Sangwan      8    82        0       0  10.250000
## 103        R Sathish      1     9        0       0   9.000000
## 38          DW Steyn      2    17        0       0   8.500000
## 108          RG More      2    28        0       0  14.000000
## 34         DJG Sammy      2    18        0       0   9.000000
## 33     DJ Muthuswami      2    20        0       0  10.000000
## 32          DJ Hooda      5    45        0       0   9.000000
## 24          CH Gayle      3    24        0       0   8.000000
## 116        SA Abbott      2    21        0       0  10.500000
## 72         LR Shukla      2    28        0       0  14.000000
## 
## [144 rows x 6 columns]

11a.Team Bowling scorecard (all matches against all IPL teams)

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data2"
path=os.path.join(dir1,"Rajasthan Royals-allMatchesAllOpposition.csv")
rr_matches = pd.read_csv(path)
scorecard=yka.teamBowlingScorecardAllOppnAllMatches(rr_matches,'Rajasthan Royals')
print(scorecard)
##                bowler  overs  runs  maidens  wicket   econrate
## 2            A Mishra     63   426        0      29   6.761905
## 66          JA Morkel     38   301        0      16   7.921053
## 129     R Vinay Kumar     48   406        0      15   8.458333
## 135          RP Singh     41   255        0      14   6.219512
## 95        MF Maharoof     23   139        0      14   6.043478
## 118         PP Chawla     45   353        0      14   7.844444
## 130         RA Jadeja     32   227        0      14   7.093750
## 50           DW Steyn     43   232        0      13   5.395349
## 56    Harbhajan Singh     45   341        0      12   7.577778
## 1            A Kumble     21   108        1      12   5.142857
## 159        SL Malinga     49   363        0      12   7.408163
## 60          IK Pathan     37   279        0      11   7.540541
## 82         KA Pollard     21   201        0      11   9.571429
## 119           PP Ojha     46   426        0      11   9.260870
## 121          R Ashwin     29   222        0      11   7.655172
## 22            B Kumar     31   233        0      11   7.516129
## 3             A Nehra     32   214        0      11   6.687500
## 41           DJ Bravo     30   292        0      10   9.733333
## 110           P Kumar     48   329        1      10   6.854167
## 58           I Sharma     37   284        0       9   7.675676
## 168   Shakib Al Hasan     25   153        0       9   6.120000
## 87           L Balaji     33   277        0       9   8.393939
## 122          R Bhatia     19   121        0       8   6.368421
## 48        DS Kulkarni     21   148        0       8   7.047619
## 101         MM Sharma     20   142        0       8   7.100000
## 174          UT Yadav     25   203        0       8   8.120000
## 15           AR Patel     16   110        0       7   6.875000
## 133         RJ Harris     16   132        0       7   8.250000
## 72          JH Kallis     37   254        0       7   6.864865
## 192            Z Khan     33   213        0       7   6.454545
## ..                ...    ...   ...      ...     ...        ...
## 170      Shoaib Ahmed      2    19        0       0   9.500000
## 54          GS Sandhu      4    49        0       0  12.250000
## 139          RV Gomez      1     9        0       0   9.000000
## 163         SPD Smith      0     5        0       0        inf
## 115       PC Valthaty      3    35        0       0  11.666667
## 34        CJ Anderson      4    26        0       0   6.500000
## 81         K Upadhyay      3    29        0       0   9.666667
## 79             K Goel      1    11        0       0  11.000000
## 28          BJ Rohrer      1    12        0       0  12.000000
## 78    Joginder Sharma      2    23        0       0  11.500000
## 99          MK Tiwary      2    28        0       0  14.000000
## 26       BE Hendricks      4    57        0       0  14.250000
## 102          MR Marsh      1    10        0       0  10.000000
## 106       NL McCullum      3    22        0       0   7.333333
## 113        P Prasanth      1    18        0       0  18.000000
## 114           P Suyal      4    45        0       0  11.250000
## 46      DP Vijaykumar      1    10        0       0  10.000000
## 154         SB Styris      2    14        0       0   7.000000
## 71       JEC Franklin      3    32        0       0  10.666667
## 70          JE Taylor      3    22        0       0   7.333333
## 18       Ankit Sharma      4    33        0       0   8.250000
## 134  RN ten Doeschate      2    14        0       0   7.000000
## 16       Abdur Razzak      2    29        0       0  14.500000
## 65           J Theron      6    48        0       0   8.000000
## 146          S Narwal      2    17        0       0   8.500000
## 63            J Botha      1    19        0       0  19.000000
## 149           S Tyagi      8    65        0       0   8.125000
## 151         SB Bangar      2    20        0       0  10.000000
## 13           AM Nayar      2     7        0       0   3.500000
## 0      A Ashish Reddy      3    22        0       0   7.333333
## 
## [193 rows x 6 columns]

12. Team Bowling wicket kind -Chart (all matches against all IPL teams)

The functions compute and display the kind of wickets taken(bowled, caught, lbw etc) by an IPL team in all matches against all other IPL teams

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data2"
path=os.path.join(dir1,"Gujarat Lions-allMatchesAllOpposition.csv")
gl_matches = pd.read_csv(path)
yka.teamBowlingWicketKindAllOppnAllMatches(gl_matches,'Gujarat Lions',plot=True,top=5,wickets=2)

13. Team Bowling wicket kind -Dataframe (all matches against all IPL teams)

This gives the type of wickets taken for an IPL team against all other IPL teams.

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data2"
path=os.path.join(dir1,"Rising Pune Supergiants-allMatchesAllOpposition.csv")
rps_matches = pd.read_csv(path)
m=yka.teamBowlingWicketKindAllOppnAllMatches(rps_matches,'Rising Pune Supergiants',plot=False,top=4,wickets=10)
print(m)
##           bowler               kind  wickets
## 0        A Nehra             caught        4
## 1        A Nehra            run out        2
## 2      MM Sharma             caught        3
## 3      MM Sharma  caught and bowled        1
## 4      MM Sharma            run out        1
## 5      SR Watson             bowled        1
## 6      SR Watson             caught        4
## 7  KW Richardson             caught        3
## 8  KW Richardson       retired hurt        1

14 Team Bowler vs Batman -Plot (all matches against all IPL teams)

The function below gives the performance of bowlers against batsmen ,in all matches against another IPL team.

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data2"
path=os.path.join(dir1,"Rising Pune Supergiants-allMatchesAllOpposition.csv")
rps_matches = pd.read_csv(path)
yka.teamBowlersVsBatsmenAllOppnAllMatches(rps_matches,'Rising Pune Supergiants',plot=True,top=5,runsConceded=10)

15 Team Bowler vs Batman – Dataframe (all matches against all IPL teams)

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data2"
path=os.path.join(dir1,"Deccan Chargers-allMatchesAllOpposition.csv")
dc_matches = pd.read_csv(path)
m=yka.teamBowlersVsBatsmenAllOppnAllMatches(dc_matches,'Deccan Chargers',plot=False,top=2,runsConceded=30)
print(m)
##        bowler          batsman  runsConceded
## 0     P Kumar   A Ashish Reddy           6.0
## 1     P Kumar        A Symonds          15.0
## 2     P Kumar      AA Bilakhia          12.0
## 3     P Kumar  AA Jhunjhunwala           1.0
## 4     P Kumar     AC Gilchrist          20.0
## 5     P Kumar    Anirudh Singh          11.0
## 6     P Kumar         B Chipli           1.0
## 7     P Kumar         CL White          11.0
## 8     P Kumar     DB Ravi Teja          15.0
## 9     P Kumar        DJ Harris           2.0
## 10    P Kumar         DR Smith           5.0
## 11    P Kumar       FH Edwards           3.0
## 12    P Kumar         HH Gibbs          46.0
## 13    P Kumar         J Theron           0.0
## 14    P Kumar        JP Duminy           4.0
## 15    P Kumar    KC Sangakkara          15.0
## 16    P Kumar        MD Mishra           4.0
## 17    P Kumar         PA Patel           9.0
## 18    P Kumar        RG Sharma          36.0
## 19    P Kumar        RJ Harris           3.0
## 20    P Kumar         S Dhawan          37.0
## 21    P Kumar          S Sohal           6.0
## 22    P Kumar        SB Styris           6.0
## 23    P Kumar    Shahid Afridi           0.0
## 24    P Kumar         TL Suman          22.0
## 25    P Kumar       VVS Laxman           5.0
## 26    P Kumar  Y Venugopal Rao           1.0
## 27  PP Chawla   A Ashish Reddy           2.0
## 28  PP Chawla        A Symonds          35.0
## 29  PP Chawla  AA Jhunjhunwala           6.0
## 30  PP Chawla     AC Gilchrist           4.0
## 31  PP Chawla         B Chipli           8.0
## 32  PP Chawla         CL White          16.0
## 33  PP Chawla     DB Ravi Teja          30.0
## 34  PP Chawla        DJ Harris           9.0
## 35  PP Chawla        DNT Zoysa           1.0
## 36  PP Chawla         HH Gibbs          30.0
## 37  PP Chawla        JP Duminy          10.0
## 38  PP Chawla    KC Sangakkara           1.0
## 39  PP Chawla         MR Marsh           1.0
## 40  PP Chawla         PA Patel           4.0
## 41  PP Chawla         PA Reddy           8.0
## 42  PP Chawla        RG Sharma          50.0
## 43  PP Chawla         S Dhawan          33.0
## 44  PP Chawla        SB Bangar           1.0
## 45  PP Chawla         TL Suman          17.0
## 46  PP Chawla       VVS Laxman           7.0
## 47  PP Chawla  Y Venugopal Rao           3.0

16 Team Wins and Losses – Summary (all matches against all IPL teams)

The function below computes and plots the number of wins and losses between an IPL team and all other IPL teams in all matches. The summary just gives the wins, losses and ties

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data2"
path=os.path.join(dir1,"Chennai Super Kings-allMatchesAllOpposition.csv")
csk_matches = pd.read_csv(path)
team1='Chennai Super Kings'
yka.plotWinLossByTeamAllOpposition(csk_matches,team1,plot="summary")

16a Team Wins and Losses – Detailed (all matches against all IPL teams)

The function below computes and plot the number of wins and losses between an IPL team and all other IPL teams in all matches. This gives a breakup of which team won against this team.

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data2"
path=os.path.join(dir1,"Chennai Super Kings-allMatchesAllOpposition.csv")
csk_matches = pd.read_csv(path)
team1='Chennai Super Kings'
yka.plotWinLossByTeamAllOpposition(csk_matches,team1,plot="detailed")

16b Team Wins and Losses – Summary (all matches against all IPL teams)

This plot gives the wins vs losses of MI against all other IPL teams

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data2"
path=os.path.join(dir1,"Mumbai Indians-allMatchesAllOpposition.csv")
mi_matches = pd.read_csv(path)
team1='Mumbai Indians'
yka.plotWinLossByTeamAllOpposition(mi_matches,team1,plot="summary")

16c Team Wins and Losses – Detailed (all matches against all IPL teams)

The function below computes and plot the number of wins and losses between an IPL team and all other IPL teams in all matches. This gives the breakup of MI wins, losses and ties

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data2"
path=os.path.join(dir1,"Mumbai Indians-allMatchesAllOpposition.csv")
mi_matches = pd.read_csv(path)
team1='Mumbai Indians'
yka.plotWinLossByTeamAllOpposition(mi_matches,team1,plot="detailed")

17 Team Wins by win type (all matches against all IPL teams)

This function shows how the win happened whether by runs or by wickets in all matches played against all other IPL teams

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data2"
path=os.path.join(dir1,"Royal Challengers Bangalore-allMatchesAllOpposition.csv")
rcb_matches = pd.read_csv(path)
yka.plotWinsByRunOrWicketsAllOpposition(rcb_matches,'Royal Challengers Bangalore')

18 Team Wins by toss decision (summary) (all matches against all IPL teams)

This show how Royal Challengers Bangalore fared when it chose to field on winning the toss

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data2"
path=os.path.join(dir1,"Royal Challengers Bangalore-allMatchesAllOpposition.csv")
rcb_matches = pd.read_csv(path)
yka.plotWinsbyTossDecisionAllOpposition(rcb_matches,'Royal Challengers Bangalore',tossDecision='field',plot='summary')

18a. Team Wins by toss decision (detailed) (all matches against all IPL teams)

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data2"
path=os.path.join(dir1,"Kings XI Punjab-allMatchesAllOpposition.csv")
kxip_matches = pd.read_csv(path)
yka.plotWinsbyTossDecisionAllOpposition(kxip_matches,'Kings XI Punjab',tossDecision='field',plot='detailed')

19 Team Wins by toss decision (summary) (all matches against all IPL teams)

This plot shows how Mumbai Indians fared when it chose to bat on winning the toss against all other IPL teams.

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data2"
path=os.path.join(dir1,"Delhi Daredevils-allMatchesAllOpposition.csv")
mi_rcb_matches = pd.read_csv(path)
yka.plotWinsbyTossDecisionAllOpposition(mi_rcb_matches,'Mumbai Indians',tossDecision='bat',plot='summary')

20 Team Wins by toss decision (detailed)(all matches against all IPL teams)

This plot shows how Kings X1 Punjab fared when it chose to bat on winning the toss

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data2"
path=os.path.join(dir1,"Kings XI Punjab-allMatchesAllOpposition.csv")
kxip_matches = pd.read_csv(path)
yka.plotWinsbyTossDecisionAllOpposition(kxip_matches,'Kings XI Punjab',tossDecision='bat',plot='detailed')

Feel free to clone/download the code from Github yorkpy

Conclusion

This post included analysis of an IPL team against all other IPL teams. You can download the data for this and the earlier posts from [yorkpyData](https://github.com/tvganesh/yorkpyData

The code can be cloned/downloaded from Github

Important note: Do check out my other posts using yorkpy at yorkpy-posts

To be continued. Watch this space!

Also see
1. My book ‘Practical Machine Learning in R and Python: Third edition’ on Amazon
2. My book ‘Deep Learning from first principles:Second Edition’ now on Amazon
3. Designing a Social Web Portal
4. Computer Vision: Ramblings on derivatives, histograms and contours
5. Introducing cricket package yorkr: Part 3-Foxed by flight!
6. The making of Total Control Android game

To see all posts click Index of posts

Pitching yorkpy…on the middle and outside off-stump to IPL – Part 2

When you come to a fork in the road, take it.
You’ve got to be very careful if you don’t know where you are going, because you might not get there

      Yogi Berra

Try taking his (Rahul Dravid’s) wicket in the first 15 minutes. If you can’t then only try to take the remaining wickets

      Steve Waugh
      

Introduction

This post is a follow-up to my previous post, Pitching yorkpy…short of good length to IPL-Part 1, in which I analyzed individual IPL matches. In this 2nd post I analyze the data in all matches between any 2 IPL teams, say CSK-RCB, MI-KKR or DD-RPS and so on. As I have already mentioned yorky is the python clone of my R packkage yorkr and this post is almost a mirror image of my post with yorkr namely yorkr crashes the IPL party! – Part 2. The signatures of yorkpy and yorkr are identical and will work in amost the same way. yorkpy, like yorkr, uses data from Cricsheet

You can clone/download the code at Github yorkpy
This post has been published to RPubs at yorkpy-Part2
You can download this post as PDF at IPLT20-yorkpy-part2
You can download all the data used in this post and the previous post at yorkpyData

Note: If you would like to do a similar analysis for a different set of batsman and bowlers, you can clone/download my skeleton yorkpy-template from Github (which is the R Markdown file I have used for the analysis below).

2. Get data for all T20 matches between 2 teams

We can get all IPL T20 matches between any 2 teams using the function below. The dir parameter should point to the folder which has the IPL T20 csv files of the individual matches (see Pitching yorkpy…short of good length to IPL-Part 1). This function creates a data frame of all the IPL T20 matches and and also saves the dataframe as CSV file if save=True. If save=False the dataframe is just returned and not saved.

import pandas as pd
import os
import yorkpy.analytics as yka
#dir1= "C:\\software\\cricket-package\\yorkpyPkg\\yorkpyData\\IPLConverted"
#yka.getAllMatchesBetweenTeams("Kolkata Knight Riders","Delhi Daredevils",dir=dir1,save=True)

3. Save data for all matches between all combination of 2 teams

This can be done locally using the function below. You could use this function to combine all IPL Twenty20 matches between any 2 IPL teams into a single dataframe and save it in the current folder. All the dataframes for all combinations have already been done and are available as CSV files in Github at yorkpyData

import pandas as pd
import os
import yorkpy.analytics as yka
#dir1= "C:\\software\\cricket-package\\yorkpyPkg\\yorkpyData\\IPLConverted"
#yka.saveAllMatchesBetween2IPLTeams(dir1)

Note: In the functions below, I have randomly chosen any 2 IPL teams and analyze how the teams have performed against each other in different areas. You are free to choose any 2 combination of IPL teams for your analysis

4.Team Batsmen partnership in Twenty20 (all matches with opposing IPL team – summary)

The function below computes the highest partnerships between the 2 IPL teams Chennai Superkings and Delhi Daredevils. Any other 2 IPL team could have also been chosen. The summary gives the top 3 batsmen for Delhi Daredevils namely Sehwag, Gambhir and Dinesh Karthik when the report=‘summary’

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data1"
path=os.path.join(dir1,"Chennai Super Kings-Delhi Daredevils-allMatches.csv")
csk_dd_matches = pd.read_csv(path)
m=yka.teamBatsmenPartnershiOppnAllMatches(csk_dd_matches,'Delhi Daredevils',report="summary")
print(m)
##            batsman  totalPartnershipRuns
## 49        V Sehwag                   233
## 12       G Gambhir                   200
## 21      KD Karthik                   180
## 10       DA Warner                   134
## 4   AB de Villiers                   133

5. Team Batsmen partnership in Twenty20 (all matches with opposing IPL team -detailed)

The function below gives the detailed breakup of partnerships between Deccan Chargers and Mumbai Indians for Deccan Chargers.

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data1"
path=os.path.join(dir1,"Deccan Chargers-Mumbai Indians-allMatches.csv")
dc_mi_matches = pd.read_csv(path)
theTeam='Deccan Chargers'
m=yka.teamBatsmenPartnershiOppnAllMatches(dc_mi_matches,theTeam,report="detailed", top=4)
print(m)
##          batsman  totalPartnershipRuns      non_striker  partnershipRuns
## 0   AC Gilchrist                   201        A Symonds                0
## 1   AC Gilchrist                   201         HH Gibbs               53
## 2   AC Gilchrist                   201        MD Mishra                0
## 3   AC Gilchrist                   201        RG Sharma               20
## 4   AC Gilchrist                   201    Shahid Afridi                6
## 5   AC Gilchrist                   201         TL Suman                7
## 6   AC Gilchrist                   201       VVS Laxman              115
## 7       S Dhawan                   122         A Mishra                9
## 8       S Dhawan                   122         B Chipli                1
## 9       S Dhawan                   122         CL White                2
## 10      S Dhawan                   122     DT Christian               52
## 11      S Dhawan                   122         IR Jaggi                2
## 12      S Dhawan                   122        JP Duminy                9
## 13      S Dhawan                   122    KC Sangakkara               16
## 14      S Dhawan                   122         PA Patel               22
## 15      S Dhawan                   122          S Sohal                9
## 16     RG Sharma                   103        A Symonds               11
## 17     RG Sharma                   103     AC Gilchrist               18
## 18     RG Sharma                   103         DR Smith                6
## 19     RG Sharma                   103         HH Gibbs                3
## 20     RG Sharma                   103   Jaskaran Singh               15
## 21     RG Sharma                   103        KAJ Roach                4
## 22     RG Sharma                   103        LPC Silva                0
## 23     RG Sharma                   103         TL Suman               14
## 24     RG Sharma                   103  Y Venugopal Rao               32
## 25      HH Gibbs                   102     AC Gilchrist               40
## 26      HH Gibbs                   102         DR Smith               24
## 27      HH Gibbs                   102        MD Mishra               27
## 28      HH Gibbs                   102        RG Sharma                8
## 29      HH Gibbs                   102       VVS Laxman                1
## 30      HH Gibbs                   102  Y Venugopal Rao                2

6. Team Batsmen partnership in Twenty20 – Chart (all matches with opposing IPL team)

The function below plots the partnerships in all matches between 2 IPL teams and plots as chart

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data1"
path=os.path.join(dir1,"Gujarat Lions-Kings XI Punjab-allMatches.csv")
gl_kxip_matches = pd.read_csv(path)
yka.teamBatsmenPartnershipOppnAllMatchesChart(gl_kxip_matches,'Kings XI Punjab','Gujarat Lions', plot=True, top=4, partnershipRuns=20)

7.Team Batsmen partnership in Twenty20 – Dataframe (all matches with opposing IPL team)

This function does not plot the data but returns the dataframe to the user to plot or manipulate.

Note: Many of the plots include an additional parameters for e.g. plot which is either True or False. The default value is plot=True. When plot=True the plot will be displayed. When plot=False the data frame will be returned to the user. The user can use this to create an interactive charts. The parameter top= specifies the number of top batsmen that need to be included in the chart, and partnershipRuns gives the minimum cutoff runs in partnerships to be considered

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data1"
path=os.path.join(dir1,"Kolkata Knight Riders-Rising Pune Supergiants-allMatches.csv")
kkr_rps_matches = pd.read_csv(path)
m=yka.teamBatsmenPartnershipOppnAllMatchesChart(kkr_rps_matches,'Rising Pune Supergiants','Kolkata Knight Riders', plot=False, top=5, partnershipRuns=20)
print(m)
##         batsman   non_striker  partnershipRuns
## 0     AM Rahane  F du Plessis               20
## 1     AM Rahane     JA Morkel               16
## 2     AM Rahane   NLTC Perera                6
## 3     AM Rahane     SPD Smith               25
## 4     AM Rahane    UT Khawaja                2
## 5     GJ Bailey     IK Pathan                4
## 6     GJ Bailey     SS Tiwary               28
## 7     GJ Bailey    UT Khawaja                1
## 8      MS Dhoni     IK Pathan                5
## 9      MS Dhoni     JA Morkel                1
## 10     MS Dhoni   NLTC Perera                2
## 11     MS Dhoni      R Ashwin                1
## 12     MS Dhoni      R Bhatia               22
## 13    SPD Smith     AM Rahane               31
## 14  NLTC Perera     AM Rahane               12
## 15  NLTC Perera      MS Dhoni               13

8. Team batsmen versus bowler in Twenty20-Chart (all matches with opposing IPL team)

The plots below provide information on how each of the top batsmen of the IPL teams fared against the opposition bowlers

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data1"
path=os.path.join(dir1,"Rajasthan Royals-Royal Challengers Bangalore-allMatches.csv")
rr_rcb_matches = pd.read_csv(path)
yka.teamBatsmenVsBowlersOppnAllMatches(rr_rcb_matches,'Rajasthan Royals',"Royal Challengers Bangalore",plot=True,top=3,runsScored=20)

9 Team batsmen versus bowler in Twenty20-Dataframe (all matches with opposing IPL team)

This function provides the bowling performance, the number of overs bowled, maidens, runs conceded. wickets taken and economy rate for the IPL match

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data1"
path=os.path.join(dir1,"Mumbai Indians-Delhi Daredevils-allMatches.csv")
mi_dd_matches = pd.read_csv(path)
m=yka.teamBatsmenVsBowlersOppnAllMatches(mi_dd_matches,'Delhi Daredevils',"Mumbai Indians",plot=False,top=2,runsScored=50)
print(m)
##       batsman           bowler  runsScored
## 0    V Sehwag          A Nehra         6.0
## 1    V Sehwag       AG Murtaza         6.0
## 2    V Sehwag         AM Nayar        14.0
## 3    V Sehwag         CJ McKay        10.0
## 4    V Sehwag     CRD Fernando         9.0
## 5    V Sehwag         DJ Bravo         9.0
## 6    V Sehwag      DJ Thornely         0.0
## 7    V Sehwag         DR Smith        13.0
## 8    V Sehwag      DS Kulkarni        20.0
## 9    V Sehwag  Harbhajan Singh        54.0
## 10   V Sehwag        JJ Bumrah        19.0
## 11   V Sehwag       KA Pollard        37.0
## 12   V Sehwag         MM Patel        27.0
## 13   V Sehwag          PP Ojha         7.0
## 14   V Sehwag         R Shukla         9.0
## 15   V Sehwag      RJ Peterson         7.0
## 16   V Sehwag         RP Singh        28.0
## 17   V Sehwag       SL Malinga        32.0
## 18   V Sehwag       SM Pollock        25.0
## 19   V Sehwag    ST Jayasuriya        29.0
## 20   V Sehwag           Z Khan        14.0
## 21  JP Duminy      CJ Anderson         3.0
## 22  JP Duminy        HH Pandya         7.0
## 23  JP Duminy  Harbhajan Singh        29.0
## 24  JP Duminy        J Suchith         5.0
## 25  JP Duminy        JJ Bumrah        70.0
## 26  JP Duminy       KA Pollard        29.0
## 27  JP Duminy        KH Pandya         8.0
## 28  JP Duminy       M de Lange         6.0
## 29  JP Duminy   MJ McClenaghan        14.0
## 30  JP Duminy           N Rana         1.0
## 31  JP Duminy          PP Ojha        16.0
## 32  JP Duminy    R Vinay Kumar        18.0
## 33  JP Duminy        RG Sharma         3.0
## 34  JP Duminy          S Gopal         8.0
## 35  JP Duminy       SL Malinga         8.0
## 36  JP Duminy       TG Southee         3.0

10. Team batting scorecard(all matches with opposing IPL team)

This function provides the overall scorecard for an IPL team in all matches against another IPL team. In the snippet below the batting scorecard of RCB is show against CSK. Kohli, Gayle and De villiers lead the pack.

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data1"
path=os.path.join(dir1,"Royal Challengers Bangalore-Chennai Super Kings-allMatches.csv")
rcb_csk_matches = pd.read_csv(path)
scorecard=yka.teamBattingScorecardOppnAllMatches(rcb_csk_matches,'Royal Challengers Bangalore',"Chennai Super Kings")
print(scorecard)
##              batsman  runs  balls  4s  6s          SR
## 5            V Kohli   706    570  51  30  123.859649
## 20          CH Gayle   270    228  12  23  118.421053
## 19    AB de Villiers   241    157  26   9  153.503185
## 6           R Dravid   133    117  18   0  113.675214
## 3          JH Kallis   123    113  21   0  108.849558
## 22        MA Agarwal   120    104  15   4  115.384615
## 2        LRPL Taylor   117    102   5   6  114.705882
## 11        RV Uthappa   115     77   7   8  149.350649
## 21         SS Tiwary    86     88   4   3   97.727273
## 17         MK Pandey    73     72  10   0  101.388889
## 32        KD Karthik    61     58   9   0  105.172414
## 34           D Wiese    51     43   4   2  118.604651
## 33           SN Khan    50     36   5   1  138.888889
## 1           W Jaffer    50     36   5   2  138.888889
## 7            P Kumar    39     25   2   2  156.000000
## 28      Yuvraj Singh    38     33   2   1  115.151515
## 4         MV Boucher    37     33   4   1  112.121212
## 23     LA Pomersbach    31     21   2   2  147.619048
## 8             Z Khan    29     27   3   0  107.407407
## 12      KP Pietersen    23     15   2   1  153.333333
## 38          CL White    21     13   2   1  161.538462
## 26       YV Takawale    19     17   4   0  111.764706
## 31          MS Bisla    17     14   3   0  121.428571
## 14     R Vinay Kumar    17     10   1   1  170.000000
## 25        RR Rossouw    15     13   1   1  115.384615
## 40        AUK Pathan    14      6   2   1  233.333333
## 42   JJ van der Wath    14     11   1   1  127.272727
## 27            VH Zol    13     12   0   1  108.333333
## 30          MA Starc    13     16   1   0   81.250000
## 24      MC Henriques    12      4   3   0  300.000000
## 44          A Mithun    11      8   2   0  137.500000
## 50          PA Patel    10     14   2   0   71.428571
## 36        SP Goswami    10     19   1   0   52.631579
## 0           B Chipli     8     12   1   0   66.666667
## 9            B Akhil     8     12   1   0   66.666667
## 29            S Rana     6      8   0   0   75.000000
## 16  RE van der Merwe     5     12   0   0   41.666667
## 49   KB Arun Karthik     5      5   0   0  100.000000
## 54     Mandeep Singh     4      7   0   0   57.142857
## 37     Misbah-ul-Haq     4      6   0   0   66.666667
## 52      NJ Maddinson     4      7   1   0   57.142857
## 51          AN Ahmed     4      1   1   0  400.000000
## 15          A Kumble     3      6   0   0   50.000000
## 43        DL Vettori     3      4   0   0   75.000000
## 47      DT Christian     2      2   0   0  100.000000
## 45   J Syed Mohammad     2      3   0   0   66.666667
## 35          HV Patel     2      5   0   0   40.000000
## 41         CA Pujara     2      6   0   0   33.333333
## 10          DW Steyn     1      5   0   0   20.000000
## 18        EJG Morgan     1      4   0   0   25.000000
## 46        RR Bhatkal     0      2   0   0    0.000000
## 48         R Rampaul     0      6   0   0    0.000000
## 13         R Bishnoi     0      1   0   0    0.000000
## 39        TM Dilshan     0      1   0   0    0.000000
## 53     Iqbal Abdulla     0      3   0   0    0.000000
## 55         S Aravind     0      1   0   0    0.000000

11.Team Bowling scorecard (all matches with opposing IPL team)

The output below gives the performance of Rajasthan Royals bowlers against Kolkata Knight Riders in all matches between the 2 IPL teams.

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data1"
path=os.path.join(dir1,"Kolkata Knight Riders-Rajasthan Royals-allMatches.csv")
rcb_csk_matches = pd.read_csv(path)
scorecard=yka.teamBowlingScorecardOppnAllMatches(rcb_csk_matches,'Rajasthan Royals',"Kolkata Knight Riders")
print(scorecard)
##               bowler  overs  runs  maidens  wicket   econrate
## 31   Shakib Al Hasan     25   153        0       9   6.120000
## 12          I Sharma     15   118        0       6   7.866667
## 33          Umar Gul      8    61        0       6   7.625000
## 29         SP Narine     24   155        0       6   6.458333
## 1           AB Dinda     20   126        0       6   6.300000
## 23     R Vinay Kumar      8    72        0       5   9.000000
## 22          R Bhatia     15   104        0       5   6.933333
## 0         AB Agarkar     12   105        0       4   8.750000
## 17         LR Shukla     12    87        0       4   7.250000
## 6              B Lee     15    90        0       4   6.000000
## 3         AD Russell      7    59        0       4   8.428571
## 34         YK Pathan      8    61        0       4   7.625000
## 14        JD Unadkat      4    26        0       3   6.500000
## 15         JH Kallis     20   149        0       3   7.450000
## 16          L Balaji     11    73        0       3   6.636364
## 27           SE Bond      8    52        1       3   6.500000
## 10     CK Langeveldt      4    15        0       3   3.750000
## 13     Iqbal Abdulla     10    70        0       3   7.000000
## 28   SMSM Senanayake      4    26        0       2   6.500000
## 7         BAW Mendis      4    19        0       2   4.750000
## 18          M Kartik      8    56        0       2   7.000000
## 4      Anureet Singh      4    35        0       2   8.750000
## 32          UT Yadav      7    67        0       2   9.571429
## 30         SS Sarkar      3    15        0       1   5.000000
## 26        SC Ganguly      6    61        0       1  10.166667
## 5      Azhar Mahmood      3    41        0       1  13.666667
## 19          M Morkel      8    78        0       1   9.750000
## 11         DJ Hussey      2    26        0       0  13.000000
## 2         AD Mathews      3    33        0       0  11.000000
## 8           BJ Hodge      2    34        0       0  17.000000
## 25          S Narwal      2    17        0       0   8.500000
## 24  RN ten Doeschate      2    14        0       0   7.000000
## 21         PP Chawla      4    39        0       0   9.750000
## 20    Mohammed Shami      3    26        0       0   8.666667
## 9           CH Gayle      4    20        0       0   5.000000

12. Team Bowling wicket kind -Chart (all matches with opposing IPL team)

The functions compute and display the kind of wickets taken(bowled, caught, lbw etc) by an IPL team in all matches against another IPL team

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data1"
path=os.path.join(dir1,"Chennai Super Kings-Rajasthan Royals-allMatches.csv")
csk_rr_matches = pd.read_csv(path)
yka.teamBowlingWicketKindOppositionAllMatches(csk_rr_matches,'Chennai Super Kings','Rajasthan Royals',plot=True,top=5,wickets=1)

13. Team Bowling wicket kind -Dataframe (all matches with opposing IPL team)

This gives the type of wickets taken

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data1"
path=os.path.join(dir1,"Delhi Daredevils-Pune Warriors-allMatches.csv")
dd_pw_matches = pd.read_csv(path)
m=yka.teamBowlingWicketKindOppositionAllMatches(dd_pw_matches,'Pune Warriors','Delhi Daredevils',plot=False,top=4,wickets=1)
print(m)
##       bowler    kind  wickets
## 0  IK Pathan  bowled        1
## 1  IK Pathan  caught        3
## 2   M Morkel  bowled        1
## 3   M Morkel  caught        3
## 4   S Nadeem  bowled        1
## 5   S Nadeem  caught        2
## 6   UT Yadav  caught        3

14 Team Bowler vs Batman -Plot (all matches with opposing IPL team)

The function below gives the performance of bowlers in all matches against another IPL team.

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data1"
path=os.path.join(dir1,"Sunrisers Hyderabad-Kolkata Knight Riders-allMatches.csv")
srh_kkr_matches = pd.read_csv(path)
yka.teamBowlersVsBatsmenOppnAllMatches(srh_kkr_matches,'Sunrisers Hyderabad','Kolkata Knight Riders',plot=True,top=5,runsConceded=10)

15 Team Bowler vs Batman – Dataframe (all matches with opposing IPL team)

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data1"
path=os.path.join(dir1,"Royal Challengers Bangalore-Kings XI Punjab-allMatches.csv")
srh_kkr_matches = pd.read_csv(path)
m=yka.teamBowlersVsBatsmenOppnAllMatches(srh_kkr_matches,'Royal Challengers Bangalore','Kings XI Punjab',plot=False,top=1,runsConceded=30)
print(m)
##        bowler           batsman  runsConceded
## 0   PP Chawla          A Kumble             1
## 1   PP Chawla          A Mithun             1
## 2   PP Chawla       AB McDonald             3
## 3   PP Chawla    AB de Villiers            29
## 4   PP Chawla         CA Pujara            13
## 5   PP Chawla          CH Gayle            62
## 6   PP Chawla     CK Langeveldt             1
## 7   PP Chawla          CL White             3
## 8   PP Chawla        DL Vettori             1
## 9   PP Chawla          DT Patil             4
## 10  PP Chawla         JH Kallis            17
## 11  PP Chawla   JJ van der Wath             1
## 12  PP Chawla   KB Arun Karthik             4
## 13  PP Chawla      KP Pietersen            14
## 14  PP Chawla       LRPL Taylor             6
## 15  PP Chawla            M Kaif             2
## 16  PP Chawla         MK Pandey            10
## 17  PP Chawla        MV Boucher             9
## 18  PP Chawla     Misbah-ul-Haq             0
## 19  PP Chawla           P Kumar             0
## 20  PP Chawla          R Dravid            28
## 21  PP Chawla  RE van der Merwe             7
## 22  PP Chawla        RV Uthappa            19
## 23  PP Chawla         SS Tiwary             6
## 24  PP Chawla           V Kohli            56
## 25  PP Chawla            Z Khan             0

16 Team Wins and Losses (all matches with opposing IPL team)

The function below computes and plot the number of wins and losses in a head-on confrontation between 2 IPL teams

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data1"
path=os.path.join(dir1,"Chennai Super Kings-Delhi Daredevils-allMatches.csv")
csk_dd_matches = pd.read_csv(path)
yka.plotWinLossBetweenTeams(csk_dd_matches,'Chennai Super Kings','Delhi Daredevils')

17 Team Wins by win type (all matches with opposing IPL team)

This function shows how the win happened whether by runs or by wickets

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data1"
path=os.path.join(dir1,"Chennai Super Kings-Delhi Daredevils-allMatches.csv")
csk_dd_matches = pd.read_csv(path)
yka.plotWinsByRunOrWickets(csk_dd_matches,'Chennai Super Kings')

18 Team Wins by toss decision-field (all matches with opposing IPL team)

This show how Rajasthan Royals fared when it chose to field on winning the toss

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data1"
path=os.path.join(dir1,"Rajasthan Royals-Kings XI Punjab-allMatches.csv")
rr_kxip_matches = pd.read_csv(path)
yka.plotWinsbyTossDecision(rr_kxip_matches,'Rajasthan Royals',tossDecision='field')

18 Team Wins by toss decision-bat (all matches with opposing IPL team)

This plot shows how Mumbai Indians fared when it chose to bat on winning the toss

import pandas as pd
import os
import yorkpy.analytics as yka
dir1= "C:\\software\\cricket-package\\yorkpyIPLData\\data1"
path=os.path.join(dir1,"Mumbai Indians-Royal Challengers Bangalore-allMatches.csv")
mi_rcb_matches = pd.read_csv(path)
yka.plotWinsbyTossDecision(mi_rcb_matches,'Mumbai Indians',tossDecision='bat')

Feel free to clone/download the code from Github yorkpy

Important note: Do check out my other posts using yorkpy at yorkpy-posts

My presentations on ‘Elements of Neural Networks & Deep Learning’ -Parts 6,7,8

This is the final set of presentations in my series ‘Elements of Neural Networks and Deep Learning’. This set follows the earlier 2 sets of presentations namely
1. My presentations on ‘Elements of Neural Networks & Deep Learning’ -Part1,2,3
2. My presentations on ‘Elements of Neural Networks & Deep Learning’ -Parts 4,5

In this final set of presentations I discuss initialization methods, regularization techniques including dropout. Next I also discuss gradient descent optimization methods like momentum, rmsprop, adam etc. Lastly, I briefly also touch on hyper-parameter tuning approaches. The corresponding implementations are available in vectorized R, Python and Octave are available in my book ‘Deep Learning from first principles:Second edition- In vectorized Python, R and Octave

1. Elements of Neural Networks and Deep Learning – Part 6
This part discusses initialization methods specifically like He and Xavier. The presentation also focuses on how to prevent over-fitting using regularization. Lastly the dropout method of regularization is also discusses


The corresponding implementations in vectorized R, Python and Octave of the above discussed methods are available in my post Deep Learning from first principles in Python, R and Octave – Part 6

2. Elements of Neural Networks and Deep Learning – Part 7
This presentation introduces exponentially weighted moving average and shows how this is used in different approaches to gradient descent optimization. The key techniques discussed are learning rate decay, momentum method, rmsprop and adam.


The equivalent implementations of the gradient descent optimization techniques in R, Python and Octave can be seen in my post Deep Learning from first principles in Python, R and Octave – Part 7

3. Elements of Neural Networks and Deep Learning – Part 8
This last part touches upon hyper-parameter tuning in Deep Learning networks


This concludes this series of presentations on “Elements of Neural Networks and Deep Learning’

Important note: Do check out my later version of these videos at Take 4+: Presentations on ‘Elements of Neural Networks and Deep Learning’ – Parts 1-8 . These have more content and also include some corrections. Check it out!

Checkout my book ‘Deep Learning from first principles: Second Edition – In vectorized Python, R and Octave’. My book starts with the implementation of a simple 2-layer Neural Network and works its way to a generic L-Layer Deep Learning Network, with all the bells and whistles. The derivations have been discussed in detail. The code has been extensively commented and included in its entirety in the Appendix sections. My book is available on Amazon as paperback ($18.99) and and in kindle version($9.99/Rs449).

See also
1. My book ‘Practical Machine Learning in R and Python: Third edition’ on Amazon
2. Big Data-1: Move into the big league:Graduate from Python to Pyspark
3. My travels through the realms of Data Science, Machine Learning, Deep Learning and (AI)
4. Revisiting crimes against women in India
5. Introducing cricket package yorkr: Part 1- Beaten by sheer pace!
6. Deblurring with OpenCV: Weiner filter reloaded
7. Taking a closer look at Quantum gates and their operations

To see all posts click Index of posts