GooglyPlusPlus2021 with IPL 2021, as-it-happens!

A brand new season of IPL 2021 is on its way, and I intend to keep my Shiny app GooglyPlusPlus updated with all the analysis “as-it-happens”. I had written a post earlier Big Data 7: yorkr waltzes with Apache NiFi, where I used my R package yorkr in a NiFi pipeline to automate downloading, converting and generating appropriate data for GooglyPlusPlus. However, using Apache NiFi for daily updates would be an overkill. So, I have created a ‘big bash script’ (with shell,R,python scripts) and scheduled daily with CRON, which will get daily updates from Cricsheet, convert the yaml files, generate the necessary data files for GooglyPlusPlus in an automated way, using my R package yorkr, and integrate with the Shiny app

Now, you should be able to do detailed analysis of batsmen, bowlers, IPL matches, IPL teams and also do the ranking of the batsmen and bowlers as new data is added on a daily basis. Also remember that GooglyPlusPlus2021 can do similar analysis for all T20 formats (Intl. T20 (men,women), BBL, NTB, PSL, CPL, WBB etc.)

Check out GooglyPlusPlus2021-IPL 2021

I will be keeping the app updated as data becomes available after the match. Do check it out. Here are some random analysis of the the completed 29 matches (see included matches in table below)

You can download/clone the code for GooglyPlusPlus2021 from Github at gpp2021-1

You can perform analysis of the completed matches in the ‘IPL Match’ tab as shown below

A) Mumbai Indians-Royal Challengers Bangalore-2021-04-09

a) Match scorecard – Mumbai Indians

Note: The scorecards are computed in real time.

b) Batting Partnerships – Royal Challengers Bangalore

c) Bowling Wicket Kind – Royal Challengers Bangalore

B) Chennai Super Kings vs Delhi Capitals – 2021-04-10

d) Batting Partnerships (table) – Delhi Capitals

e) Match Worm Graph

C) Kolkata Knight Riders vs Sunrisers Hyderabad – 2021-04-11

f) Batsmen vs Bowlers

D) Final ranks of IPL 2021 batsmen

E) Final ranks of IPL 2021 bowlers

Incidentally GooglyPlusPlus2021 has also been updated with all matches PSL 2021. Here is a sample

PSL 2021

F) Quetta Gladiators vs Lahore Qalanders – 22-02-2021

G) Ranks of batsmen PSL 2021

H) Ranks bowlers PSL 2021

Important note :

Note: 1) The GooglyPlusPlus2021 Shiny app also includes functions for performing the following analysis namely

  • detailed analysis of batsmen
  • detailed analysis of bowlers
  • match analysis
  • Head-to-head between 2 IPL teams
  • Analysis of IPL team against all other teams
  • Ranking of batsmen based on number of years and matches played
  • Ranking of bowlers based on number of years and matches played

So do check the other tabs of this app

2) GooglyPlusPlus2021 can do similar analysis for other T20 games like Intl. T20 (men,women), BBL, NTB, PSL and so on.

Give GooglyPlusPlus2021 a spin!!

Download/clone the code for GooglyPlusPlus2021 from Github at gpp2021-1

GooglyPlusPlus2021 has been updated with all the completed 29 matches

Mumbai Indians-Royal Challengers Bangalore-2021-04-09Chennai Super Kings-Delhi Capitals-2021-04-10
Kolkata Knight Riders-Sunrisers Hyderabad-2021-04-11Punjab Kings-Rajasthan Royals-2021-04-12
Mumbai Indians-Kolkata Knight Riders-2021-04-13Royal Challengers Bangalore-Sunrisers Hyderabad-2021-04-14
Delhi Capitals-Rajasthan Royals-2021-04-15Punjab Kings-Chennai Super Kings-2021-04-16
Mumbai Indians-Sunrisers Hyderabad-2021-04-17Royal Challengers Bangalore-Kolkata Knight Riders-2021-04-18
Punjab Kings-Delhi Capitals-2021-04-18Chennai Super Kings-Rajasthan Royals-2021-04-19
Mumbai Indians-Delhi Capitals-2021-04-20Punjab Kings-Sunrisers Hyderabad-2021-04-21
Chennai Super Kings-Kolkata Knight Riders-2021-04-21Rajasthan Royals-Royal Challengers Bangalore-2021-04-22
Mumbai Indians-Punjab Kings-2021-04-23Kolkata Knight Riders-Rajasthan Royals-2021-04-24
Chennai Super Kings-Royal Challengers Bangalore-2021-04-25Delhi Capitals-Sunrisers Hyderabad-2021-04-25
Punjab Kings-Kolkata Knight Riders-2021-04-26Royal Challengers Bangalore-Delhi Capitals-2021-04-27
Sunrisers Hyderabad-Chennai Super Kings-2021-04-28Rajasthan Royals-Mumbai Indians-2021-04-29
Kolkata Knight Riders-Delhi Capitals-2021-04-29Punjab Kings-Royal Challengers Bangalore-2021-04-30.RData
Chennai Super Kings-Mumbai Indians-2021-05-01Rajasthan Royals-Sunrisers Hyderabad-2021-05-02
Punjab Kings-Delhi Capitals-2021-05-02

Watch this space!

Also see

  1. Introducing GooglyPlusPlus!!!
  2. Introducing cricket package yorkr: Part 1- Beaten by sheer pace!
  3. Cricketr adds team analytics to its repertoire!!!
  4. Deep Learning from first principles in Python, R and Octave – Part 3
  5. Introducing QCSimulator: A 5-qubit quantum computing simulator in R
  6. Practical Machine Learning with R and Python – Part 5

To see all posts click Index of posts

GooglyPlusPlus2021 bubbles up top T20 players in all formats!

“Would you tell me, please, which way I ought to go from here?”
“That depends a good deal on where you want to get to.”
“I don’t much care where –”
“Then it doesn’t matter which way you go.”
                              Lewis Carroll, Alice in Wonderland

This latest version of GooglyPlusPlus2021 has the following updates

– brand new strategy and algorithm for ranking T20 players in any format (IPL, BBL, NTB, PSL etc.)

– integrates the Caribbean Premier League T20 into the app

– includes the latest BBL matches in 2020-2021

– includes all the latest Natwest T20 matches 2020

– has a new and better user interface

Interestingly the new Ranking algorithm has come just before the IPL auction. Check out who’s who in IPL T20 by taking GooglyPlusPlus2021 for ride!!!

Try out GooglyPlusPlus2021!!

You can clone/fork the code from Github at GooglyPlusPlus2021

1) Ranking Algorithm

In my last post GooglyPlusPlus2021 interactively ranks T20 batsmen and bowlers!!!  I had shown how by changing the number of matches played we see that KL Rahul and Rishabh Pant move above Virat Kohli. That set me thinking. So, I redesigned the ranking so that we start to identify newer players earlier.  This is how the new ranking works.

a) Ranking T20 batsmen :

I have the following controls

a) Since year : This tells the year range  to consider for the  batsmen. This slider moves from right to left

b) Matches played : Number of matches played by the batsman in the year range. This moves left to right

c) Mode : The ranking algorithm sorts on and ranks on 2 columns namely Runs and Strike rate. The mode tells whether to consider Runs over Strike rate of Strike rate over runs. 

By default, the control for “Since year” will be set to the ‘beginning of time’ which for IPL data is 2008. If you move the ‘Since year’ since year slider to the left, the ‘Matches played’ slider will move to the right and the corresponding maximum value of matches played will be set appropriately.

b) Ranking T20 Bowlers:

This has the following controls

a) Since year : This tells the year range to consider for  bowler in IPL. Moves right to left.

b) Matches played : Number of matches played by the bowler in the year range. Move left to right.

c) Mode 1: The bowlers are sorted and ranked on 2 columns namely Wickets and Economy rate. The mode tells whether to consider Wickets over Economy rate or Economy rate over wickets when ranking the bowlers. 

By default, the “Since year” will be set to the year when the T20 data is available. If you move the ‘Since year’ since year slider to the left, the ‘Matches played’ slider will move to the right and the corresponding maximum value of matches played will be set appropriately.

2) Strategy for ranking:

Here is the rationale and philosophy behind these controls

The ranking in my earlier post GooglyPlusPlus2021 interactively ranks T20 batsmen and bowlers!!!  is based on sorting batsmen and bowlers from the start of IPL tournament. Hence we will find players who have played a lot of matches. So in the bowler ranking you will SL Malinga who no longer plays IPL.

Hence I decided that the user should be allowed to choose the start year to consider for the ranking. We can move the “Since year’ slider from right to left to choose the data for the year range we are interested in e.g. (2014-2020, or 2017-2020). Changing the ‘Since year’ slider will obviously change the maximum matches played by any player. However the user can move this slider right to left and decide the number of matches to be considered for a batsman or bowler.  Finally, the ‘Mode’. will allow the user to choose whether the list if batsmen should be ranked first by average runs and then average strike rate or vice versa. In the case of bowlers, the choice is whether to first sort by number of wickets and then economy rate or vice versa. 

The consideration for ‘Mode’ and “Mode1′ is that while runs and wickets are important for batsman or bowler, it is clear the Strike rate and Economy rate become critical in ‘death overs‘. Batsmen who can accelerate the scoring rate when needed and bowlers who can put the brakes on during death overs is of paramount importance in T20 cricket.

Let me walk through a  few scenarios in IPL T20. The same functionality is also available in alll other T20 formats (Intl. T20 (men, women), BBL, NTB, PSL, WBB, CPL etc.)

3) Ranking IPL batsmen

Note: Those players who are underlined in red are the shooting stars**

a) Scenario 1

These are  the consistent and reliable players

Since year = 2013, Matches played = 95 and Mode = ‘Runs over Strike Rate’



b) Scenario 2

Since year = 2015, Matches played = 67 and Mode = ‘Runs over Strike Rate’

c) Scenario 3

We can turn the above over its head and choose Mode = “Strike rate over Runs’

Since year = 2015, Matches played = 67 and Mode = ‘Strike Rate over Runs’

d) Scenario 4: 

Since year = 2018, Matches played = 35 and Mode = ‘Runs over Strike Rate’

4) Ranking IPL bowlers

e) Scenario 5:

Since year = 2016, Matches played = 55 and Mode1 = ‘ Wickets over Economy Rate’

f) Scenario 6

Since year = 2018, Matches played = 33 and Mode1 = ‘ Economy Rate over wickets’. Note the economy rate is sorted in ascending order

Note:  Incidentally the ranking of IPL and other T20 players in my earlier posts is a special case, when you consider all matches from the beginning of time (or since the data is available, rather than choosing a range from later years).

Note 1: Personally, if I had to form a team, I would choose

– at least 2-3 batsmen who are reliable and have good average runs and a good strike rate

– 2 batsmen who can be used to accelerate strike rate during critical junctures or at death overs

– 2-3 bowlers who have a great record of wicket taking with good economy rate (2 + 1 pace/spin)

– 2 bowlers who have good economy rate over wickets

– 2 all rounders with good batting and bowling average

– 1 wicketkeeper batsmen

The key point is how many matches would you consider as a minimum for reliability or strike rate. More is good but not always better as you could miss out on rising stars* who may be risky but good picks and possibly cost less, since the team composition will also depend on the available budget for each team. You could also check other T20 formats for good T20 players. Creating a T20 team in IPL is an optimization problem where the objective is to maximize the runs, strike rate for batsmen,  or maximizing the wickets taken, while minimizing the economy rate for bowlers with the constraint of the overall budget of the team.

Note 2: The ranking algorithm has been included for all T20 formats in GooglyPlusPlus2020. See below

5) International T20 Batsmen (men) ranking

Since year = 2012, Matches played = 50 and Mode = ‘Runs over Strike Rate’


6) International T20 Bowlers (men) ranking

Since year = 2013, Matches played = 33 and Mode1 = ‘Wickets over Economy Rate ‘

Here are some top class T20 bowlers


7) International T20 Batsmen (women) ranking

Since year = 2015, Matches played = 29 and Mode = ‘Runs over Strike Rate’. Smriti Mandhana makes it to the top 3 in Runs over Strike Rate!



and 2nd when Strike rate over runs is considered!


8) Integrating Carribean Premier League T20

In this version I have also integrated Carribean Premier League (CPL). It took me about 3 -4  hours of focused work to setup the data and the associated code. Like every other T20 format,  CPL league has access to 100+ functions of yorkr. So you can do analysis of CPL batsmen, bowlers, CPL matches, CPL head-to-head confrontation and CPL team against all other teams besides the ranking function. You can also generate batting and bowling scorecard for matches, for a team against all other teams and the overall scorecard in all matches against all other teams. Here is a random sample

a) C Munro – Cumulative Average Runs


b) RR Emrit – Bowler’s wickets against opposition


c) Head-to-head (Barbados Tridents vs Antigua Hawkbill all matches)


d) Rank CPL batsmen 

Since year= 2016, Matches played = 41 Mode = Strike rate over Runs

Nicholas Pooran tops the list

9) BBL 2020-21

GooglyPlusPlus2021 now has the latest Big Bash League matches of 2020-21, in fact “hot off the pitch“. So you should be able to do all the analysis on current BBL data namely batsmen and bowler analysis, match analysis, team analysis vs another team or against all other teams, and finally ranking of batsmen and bowlers. Here is a sample with yesterday’s match

a) Match scorecard -Perth Scorchers- Sydney Sixers 06 Feb 2021 (Final)


b) Predict runs of batsman – CA Lynn


10) Natwest T20 Blast 2020

I noticed recently the Cricsheet  has more data. Now NTB data includes all matches till 2020. This data has been incorporated into NTB and you should be able to use all the yorkr functions to analyze batsmen, bowlers, teams, team-vs-team and team vs all other teams, besides the ranking functions. Here are a couple below

a) Head-to-head Derbyshire vs Essex all matches


b) Team Batsmen vs Bowlers – All matches all opposition Leicestershire (MJ Cosgrove)


Do check out the various functions of GooglyPlusPlus2021. Take a look at the ranks of the T20 batsmen and bowlers. Hope you have a good time!

Take GooglyPlusPlus2021 for a test ride!!

Feel free to clone/fork the code from Github at GooglyPlusPlus2021

Also see

  1. Cricketr adds team analytics to its repertoire!!!
  2. Introducing cricket package yorkr: Part 2-Trapped leg before wicket
  3. Introducing cricpy:A python package to analyze performances of cricketers
  4. Big Data-5: kNiFi-ing through cricket data with yorkpy
  5. Using Linear Programming (LP) for optimizing bowling change or batting lineup in T20 cricket
  6. See all 72+ posts on cricket in R, Python and Octave
  7. Deep Learning from first principles in Python, R and Octave – Part 5
  8. Practical Machine Learning with R and Python – Part 3
  9. Understanding Neural Style Transfer with Tensorflow and Keras

To see all post click Index of posts



GooglyPlusPlus2021 interactively ranks T20 batsmen and bowlers!!!

Every time I think that I have my R packages or Shiny apps all wrapped up, I find another idea trots up and knocks at my door. Since I intend to keep GooglyPlusPlus current with the latest data, I decided to include the ranking functions in my Shiny app GooglyPlusPlus.

Fortunately, since GooglyPlusPlus is based on my R package ‘yorkr‘ (see Introducing cricket package yorkr: Beaten by sheer pace!), I could make the necessary changes to the ranking functions in the package, so that it could be incorporated into my latest Shiny app GooglyPlusPlus2021!! To know how to use GooglyPlusPlus see my post Introducing GooglyPlusPlus

Note: GooglyPlusPlus can analyze batsmen, bowlers, matches and teams.

Take GooglyPlusPlus2021 for a test drive!!!

You can clone/fork GooglyPlusPlus2021 from Github

Here are a few scenarios from GooglyPlusPlus2021

A) Ranking batsmen

Ranking IPL batsmen (minMatches = 80) – The following table shows the ranking of IPL players who have played 80 matches or more

B) Identifying batsmen of potential and promise

Ranking IPL batsmen (minMatches =70) –  If we reduce the minimum number of matches played to 70, then we see it pushes up KL Rahul above Kohli.

Ranking IPL batsmen (minMatches =60) – When the slider is moved to 60, we see that Rishabh Pant has a better mean average and mean strike rate and is also ranked above Kohli. We can identify promising players this way. However, it is also likely that some players may be just a bright flash in the pan

C) Ranking T20 bowlers (men)

D) Ranking NTB Batsmen

GooglyPlusPlus2021 can rank all T20 formats (IPL, BBL, Intl. T20 (men), Intl. T20 (women), NTB, PSL and WBB. Do give it a try!

Also remember that GooglyPlusPlus2021 includes close to 100+ functions which enable it to perform analysis of batsmen, bowlers, T20 matches, head-to-head confrontation of T20 teams and overall performance of T20 teams . To know more about GooglyPlusPlus2021 see Introducing GooglyPlusPlus

You can download the code for this app from Github at GooglyPlusPlus2021

Do give GooglyPlusPlus2021 a spin!!

I do have some other ideas also which I will be incorporating  into GooglyPlusPlus2021.

Watch this space!!

Also see
1. Deep Learning from first principles in Python, R and Octave – Part 7
2. A method to crowd source pothole marking on (Indian) roads
3. Big Data 7: yorkr waltzes with Apache NiFi
4. Understanding Neural Style Transfer with Tensorflow and Keras
5. Revisiting World Bank data analysis with WDI and gVisMotionChart
6. Natural language processing: What would Shakespeare say?
7. Introducing QCSimulator: A 5-qubit quantum computing simulator in R
8. Introducing cricpy:A python package to analyze performances of cricketers
9. Simulating an Edge Shape in Android

To see all posts click Index of posts

GooglyPlusPlus 2020!!

I have updated my GooglyPlusPlus Shiny app with data from latest IPL 2020. GooglyPlusPlus  2020 is also based on my R package yorkr.  To know more about yorkr (see Revitalizing R package yorkr.) Now you should be able to analyze IPL matches, teams, players upto IPL 2020. Note: My latest GooglyPlusPlus 2020 can analyze all formats of T20 matches. yorkr uses data from Cricsheet

There are 5 tabs in each of the T20 formats

i) Analyze T20 batsmen ii) Analyze T20 bowlers. iii) Analyze T20 match iv) Analyze T20 team

vs another T20 team v) Analyze overall performance of T20 against all other teams

I plan to update GooglyPlusPlus  at least twice a year  to keep it abreast of all the latest data of all T20 formats

In GooglyPlusPlus 2020 you can check out IPL data upto 2020, besides other T20 formats like BBL, PSL, NTB, WBBL, Intl. T20 etc.

Try out GooglyPlusPlus 2020 Shiny app!!

You can clone/fork the code from Github GooglyPlusPlus2020

Important note: My earlier app GooglyPlusPlus handled all T20 formats including ODI (men and women). Due to an issue with Shiny, I could not include ODI matches in GooglyPlusPlus 2020

Here are some snapshots from GooglyPlusPlus 2020

A. Batting – Runs vs Deliveries (Shreyas Iyer)



B. Batting – Cumulative Batting Average (Shubman Gill)


C. Bowling – Mean Economy Rate (T. Natarajan)


D. Bowling -Bowler’s wickets against opposition (N A Saini)

E. Match scorecard – CSK vs DC 2020-10-17

The scorecards batting and bowling are computed on the fly for all T20 matches


F. Match – Batsmen vs Bowlers (DD vs KKR 2015-04-20)


G. Head-to-head: MI vs  KXIP all matches – Batting scorecard

H. Overall team performance- Team Bowler Wicket kind: Rajasthan Royals

Clone/fork the code from Github GooglyPlusPlus2020

Do take GooglyPlusPlus 2020 for a drive! While I have highlighted only IPL T20, because I have updated with the latest data, GooglyPlusPlus 2020 can also handle other T20 formats like BBL, Natwest, PSL, Intl. T20 (men &women) and WBB


Hope you have fun!

Also see

1.Big Data 7: yorkr waltzes with Apache NiFi

2. Deep Learning from first principles in Python, R and Octave – Part 6

3. Deconstructing Convolutional Neural Networks with Tensorflow and Keras

4. Sea shells on the seashore

5. Practical Machine Learning with R and Python – Part 3

6. Benford’s law meets IPL, Intl. T20 and ODI cricket

To see all posts click Index of posts

Introducing GooglyPlusPlus!!!

“We can lift ourselves out of ignorance, we can find ourselves as creatures of excellence and intelligence and skill.”
“Heaven is not a place, and it is not a time. Heaven is being perfect.”
“Your whole body, from wingtip to wingtip, is nothing more than your thought itself, in a form you can see. Break the chains of your thought, and you break the chains of your body, too.”

From Jonathan Livingstone Seagull, by Richard Bach


The metamorphosis is complete, from eggs to the butterfly! My R package yorkr, went on to become Googly,  and then to GooglyPlus and  now finally GooglyPlusPlus. My latest R Shiny app now provides interactive visualisation of almost all data in Cricsheet. GooglyPlusPlus visualizes the following matches

1. ODI (men)
2. ODI (women)
3. Intl. T20 (men)
4. Intl T20 (women)
5. IPL (Indian Premier League)
6. BBL (Big Bash League)
7. NTB (Natwest T20)
8. PSL (Pakistan Super League)
9. WBBL – Women’s BBL

GooglyPlusPlus is entirely based on my R package yorkr. To know more about yorkr see ‘Revitalizing R package yorkr‘ and the roughly 25+ posts on yorkr in Index of posts

This Shiny app was quite involved, and it took a lot of work to keep things organised and separate for the different forms of cricket. Anyway it is done and I am happy with the outcome.

Before you use the app, I would suggest that you take a look at the video “How to use GooglyPlusPlus?“. In this video, I show the different features of GooglyPlusPlus and how to navigate through them.

Check out GooglyPlusPlus Shiny at GooglyPlusPlus

You can clone/fork and play around with the code of GooglyPlusPlus here at Github

A. Highlights of GooglyPlusPlus.

The R Shiny app GooglyPlusPlus has the following main pages for the 9 different cricket formats. See below


Important note: Below I will be including some random output from the GooglyPlusPlus app for different match formats, however there is a lot more features in GooglyPlusPlus

1.  Indian Premier League (IPL)

a. IPL batsman – Batsman Runs vs Deliveries


b. IPL Match – Match  batting scorecard


c. Head-to-head between 2 IPL Teams – Team Batsmen Batting Partnership All Matches




d. Overall Performance – Team Bowling Scorecard Overall




2. International T20 Men

a. Batsman Function- Runs vs Strike rate




b. Bowler Function – Mean Economy Rate



3. International T20 (Women)

a.Batsman Functions – Batsman Cumulative Average Runs



b. Intl T20 Women’s match – Match worm Graph






4. Big Bash League (BBL)

a.Head-to-Head: Team batsmen batting partnerships


b.  Overall Performance – Team batsmen vs bowlers



5. Natwest T20 (NTB)

a. Head-to-head : Team bowlers vs batsmen




b. Batsman Runs vs Deliveries



6. Pakistan Super League (PSL)

a. Overall Performance – Batsmen Partnership


b. Bowling Scorecard


7. Women’s Big Bash League (WBBL)

a. Bowler wicket against opposition



8. One Day International (ODI) Men

a. Batsman Runs Against Opposition


b. Team Batsmen against bowlers



9. One Day International (ODI) women)

a. Match Batting Scorecard

b. Batsman Cumulative Strike Rate





There you have it. I have randomly shown  2 functions for each cricket format. There are many functions in each tab for the for the different match formats – namely IPL, BBL, Intl T20 (men,women), PSL etc.  Go ahead and give GooglyPlusPlus a spin!

To try out GooglyPlusPlus click GooglyPlusPlus. Don’t forget to check out the video How to use GooglyPlusPlus?

You can clone/fork the code from Github at GooglyPlusPlus

Hope you have fun with GooglyPlusPlus!!

You may also like

1. Big Data 6: The T20 Dance of Apache NiFi and yorkpy
2. Deep Learning from first principles in Python, R and Octave – Part 7
3. De-blurring revisited with Wiener filter using OpenCV
4. Exploring Quantum Gate operations with QCSimulator
5. Latency, throughput implications for the Cloud
6. Programming Zen and now – Some essential tips-2
7. The Anomaly
8. Practical Machine Learning with R and Python – Part 3
9. Introducing cricpy:A python package to analyze performances of cricketers
10. The making of Total Control Android game

To see all posts click Index of posts

Big Data 7: yorkr waltzes with Apache NiFi

In this post, I construct an end-to-end Apache NiFi pipeline with my R package yorkr. This post is a mirror of my earlier post Big Data-5: kNiFing through cricket data with yorkpy based on my Python package yorkpy. The  Apache NiFi Data Pipeilne  flows all the way from the source, where the data is obtained, all the way  to target analytics output. Apache NiFi was created to automate the flow of data between systems.  NiFi dataflows enable the automated and managed flow of information between systems. This post automates the flow of data from Cricsheet, from where the zip file it is downloaded, unpacked, processed, transformed and finally T20 players are ranked.

This post uses the functions of my R package yorkr to rank IPL players. This is a example flow, of a typical Big Data pipeline where the data is ingested from many diverse source systems, transformed and then finally insights are generated. While I execute this NiFi example with my R package yorkr, in a typical Big Data pipeline where the data is huge, of the order of 100s of GB, we would be using the Hadoop ecosystem with Hive, HDFS Spark and so on. Since the data is taken from Cricsheet, which are few Megabytes, this approach would suffice. However if we hypothetically assume that there are several batches of cricket data that are being uploaded to the source, of different cricket matches happening all over the world, and the historical data exceeds several GBs, then we could use a similar Apache NiFi pattern to process the data and generate insights. If the data is was large and distributed across the Hadoop cluster , then we would need to use SparkR or SparklyR to process the data.

This is shown below pictorially

While this post displays the ranks of IPL batsmen, it is possible to create a cool dashboard using UI/UX technologies like AngularJS/ReactJS.  Take a look at my post Big Data 6: The T20 Dance of Apache NiFi and yorkpy where I create a simple dashboard of multiple analytics

My R package yorkr can handle both men’s and women’s ODI, and all formats of T20 in Cricsheet namely Intl. T20 (men’s, women’s), IPL, BBL, Natwest T20, PSL, Women’s BBL etc. To know more details about yorkr see Revitalizing R package yorkr

The code can be forked from Github at yorkrWithApacheNiFi

You can take a look at the live demo of the NiFi pipeline at yorkr waltzes with Apache NiFi


Basic Flow

1. Overall flow

The overall NiFi flow contains 2 Process Groups a) DownloadAnd Unpack. b) Convert and Rank IPL batsmen. While it appears that the Process Groups are disconnected, they are not. The first process group downloads the T20 zip file, unpacks the. zip file and saves the YAML files in a specific folder. The second process group monitors this folder and starts processing as soon the YAML files are available. It processes the YAML converting it into dataframes before storing it as CSV file. The next  processor then does the actual ranking of the batsmen before writing the output into IPLrank.txt

1.1 DownloadAndUnpack Process Group

This process group is shown below


1.1.1 GetT20Data

The GetT20Data Processor downloads the zip file given the URL

The ${T20data} variable points to the specific T20 format that needs to be downloaded. I have set this to This could be set any other data set. In fact we could have parallel data flows for different T20/ Sports data sets and generate

1.1.2 SaveUnpackedData

This processor stores the YAML files in a predetermined folder, so that the data can be picked up  by the 2nd Process Group for processing


1.2 ProcessAndRankT20Players Process Group

This is the second process group which converts the YAML files to pandas dataframes before storing them as. CSV files. The RankIPLPlayers will then read all the CSV files, stack them and then proceed to rank the IPL players. The Process Group is shown below


1.2.1 ListFile and FetchFile Processors

The left 2 Processors ListFile and FetchFile get all the YAML files from the folder and pass it to the next processor

1.2.2 convertYaml2DataFrame Processor

The convertYaml2DataFrame Processor uses the ExecuteStreamCommand which call Rscript. The Rscript invoked the yorkr function convertYaml2DataframeT20() as shown below


I also use a 16 concurrent tasks to convert 16 different flowfiles at once


convertYaml2RDataframeT20(args[1], args[2], args[3])

1.2.3 MergeContent Processor

This processor’s only job is to trigger the rankIPLPlayers when all the FlowFiles have merged into 1 file.

1.2.4 RankT20Players

This processor is an ExecuteStreamCommand Processor that executes a Rscript which invokes a yorrkr function rankIPLT20Batsmen()



1.2.5 OutputRankofT20Player Processor

This processor writes the generated rank to an output file.


1.3 Final Ranking of IPL T20 players

The Nodejs based web server picks up this file and displays on the web page the final ranks (the code is based on a good youtube for reading from file)

[1] "Chennai Super Kings"
[1] "Deccan Chargers"
[1] "Delhi Daredevils"
[1] "Kings XI Punjab"
[1] "Kochi Tuskers Kerala"
[1] "Kolkata Knight Riders"
[1] "Mumbai Indians"
[1] "Pune Warriors"
[1] "Rajasthan Royals"
[1] "Royal Challengers Bangalore"
[1] "Sunrisers Hyderabad"
[1] "Gujarat Lions"
[1] "Rising Pune Supergiants"
[1] "Chennai Super Kings-BattingDetails.RData"
[1] "Deccan Chargers-BattingDetails.RData"
[1] "Delhi Daredevils-BattingDetails.RData"
[1] "Kings XI Punjab-BattingDetails.RData"
[1] "Kochi Tuskers Kerala-BattingDetails.RData"
[1] "Kolkata Knight Riders-BattingDetails.RData"
[1] "Mumbai Indians-BattingDetails.RData"
[1] "Pune Warriors-BattingDetails.RData"
[1] "Rajasthan Royals-BattingDetails.RData"
[1] "Royal Challengers Bangalore-BattingDetails.RData"
[1] "Sunrisers Hyderabad-BattingDetails.RData"
[1] "Gujarat Lions-BattingDetails.RData"
[1] "Rising Pune Supergiants-BattingDetails.RData"
# A tibble: 429 x 4
   batsman     matches meanRuns meanSR
   <chr>         <int>    <dbl>  <dbl>
 1 DA Warner       130     37.9   128.
 2 LMP Simmons      29     37.2   106.
 3 CH Gayle        125     36.2   134.
 4 HM Amla          16     36.1   108.
 5 ML Hayden        30     35.9   129.
 6 SE Marsh         67     35.9   120.
 7 RR Pant          39     35.3   135.
 8 MEK Hussey       59     33.8   105.
 9 KL Rahul         59     33.5   128.
10 MN van Wyk        5     33.4   112.
# … with 419 more rows



This post demonstrated an end-to-end pipeline with Apache NiFi and R package yorkr. You can this pipeline and generated different analytics using the various functions of yorkr and display them on a dashboard.

Hope you enjoyed with post!


See also
1. The mechanics of Convolutional Neural Networks in Tensorflow and Keras
2. Deep Learning from first principles in Python, R and Octave – Part 7
3. Fun simulation of a Chain in Android
4. Natural language processing: What would Shakespeare say?
5. TWS-4: Gossip protocol: Epidemics and rumors to the rescue
6. Cricketr learns new tricks : Performs fine-grained analysis of players
7. Introducing QCSimulator: A 5-qubit quantum computing simulator in R
8. Practical Machine Learning with R and Python – Part 5
9. Cricpy adds team analytics to its arsenal!!

To see posts click Index of posts

It’s a wrap! yorkr wraps up BBL, NTB, PSL and WBB!!!

“Do not take life too seriously. You will never get out of it alive.” – Elbert Hubbard

“How many people here have telekenetic powers? Raise my hand.” – Emo Philips

Have you ever noticed that anybody driving slower than you is an idiot, and anyone going faster than you is a maniac?” – George Carlin


It’s a wrap!!! In my previous posts,Revitalizing yorkr, I showed how you can use yorkr functions for Intl. ODI, Intl. T20 and IPL. My next post yorkr rocks women’s ODI and women’s Intl T20 yorkr handled women’s ODI and Intl. T20. In this post, yorkr wraps the remaining T20 formats namely

  1. Big Bash League (BBL)
  2. Natwest Super T20 (NTB)
  3. Pakistan Super League (PSL)
  4. Women’s Big Bash League (WBB)

The data for all the above T20 formats are taken from Cricsheet.

-All the data has been converted and is available in Github at yorkrData2020 organized as below. You can use any of the 90+ yorkr functions on the converted data.

Screenshot 2020-05-16 at 12.32.07 PM

-This post has been published at RPubs at yorkrWrapUpT20formats

-You can download a PDF version of this file at yorkrWrapsUpT20Formats

  • For ODI Matches men’s and women’ use
  1. ODI-Part1, 2. ODI-Part2,3. ODI-Part3, 4.ODI-Part 4
  • For any of the T20s formats you can use the following posts
  1. T20-Part1, 2. T20-Part2, 3. T20-Part3, 4. T20-Part4

or you can use these templates Intl. T20, or similar to IPL T20

I am going to randomly pick 2 yorkr functions for each of the T20 formats BBL, NTB, PSL and WBB to demonstrate yorkr below, however you can use any of the 90+ yorkr functions

install.packages("../../../yorkrgit/yorkr_0.0.9.tar.gz",repos = NULL, type="source")

Note: In the following T20 formats I have randomly picked 2 of the 90+ yorkr functions

A. Big Bash League (BBL)

A1.Batting Scorecard

load("../../../yorkrData2020/bbl/bblMatches/Adelaide Strikers-Brisbane Heat-2017-12-31.RData")
as_bh <- overs
teamBattingScorecardMatch(as_bh,'Adelaide Strikers')
## Total= 139
## # A tibble: 9 x 5
##   batsman      ballsPlayed fours sixes  runs
##   <chr>              <int> <dbl> <dbl> <dbl>
## 1 AT Carey               6     0     0     2
## 2 CA Ingram             21     2     0    23
## 3 J Weatherald          14     2     1    20
## 4 JS Lehmann            17     3     0    22
## 5 JW Wells              13     1     0    12
## 6 MG Neser              25     3     2    40
## 7 PM Siddle              1     0     0     1
## 8 Rashid Khan            2     0     1     6
## 9 TM Head               17     0     0    13

A2.Batting Partnership

load("../../../yorkrData2020/bbl/bblMatches2Teams/Melbourne Renegades-Sydney Sixers-allMatches.RData")
mr_ss_matches <- matches
m <-teamBatsmenPartnershiOppnAllMatches(mr_ss_matches,'Sydney Sixers',report="summary")
## # A tibble: 28 x 2
##    batsman      totalRuns
##    <chr>            <dbl>
##  1 MC Henriques       277
##  2 JR Philippe        186
##  3 NJ Maddinson       183
##  4 MJ Lumb            165
##  5 DP Hughes          158
##  6 JC Silk            141
##  7 SPD Smith          116
##  8 JM Vince            97
##  9 TK Curran           68
## 10 J Botha             33
## # … with 18 more rows

B. Natwest Super League

B1.Team Match Partnership

db_nt <-overs

B2.Batsmen vs Bowlers

load("../../../yorkrData2020/ntb/ntbMatches2Teams/Birmingham Bears-Leicestershire-allMatches.RData")
bb_le_matches <- matches
teamBatsmenVsBowlersOppnAllMatches(bb_le_matches,"Birmingham Bears","Leicestershire",top=3)

C. Pakistan Super League (PSL)

C1.Individual performance of Babar Azam


babar <- getBatsmanDetails(team="Karachi Kings",name="Babar Azam",dir="../../../yorkrData2020/psl/pslBattingBowlingDetails/")
## [1] "../../../yorkrData2020/psl/pslBattingBowlingDetails//Karachi Kings-BattingDetails.RData"
## [1] 40 15
p1 <-batsmanRunsVsStrikeRate(babar,"Babar Azam")
p2 <-batsmanMovingAverage(babar,"Babar Azam")
p3 <- batsmanCumulativeAverageRuns(babar,"Babar Azam")
grid.arrange(p1,p2,p3, ncol=2)

C2.Bowling performance against all oppositions

load("../../../yorkrData2020/psl/pslMatches2Teams/Lahore Qalandars-Multan Sultans-allMatches.RData")
lq_ms_matches <- matches
teamBowlingPerfOppnAllMatches(lq_ms_matches,"Lahore Qalanders","Multan Sultans")
## # A tibble: 40 x 5
##    bowler              overs maidens  runs wickets
##    <chr>               <int>   <int> <dbl>   <dbl>
##  1 Shaheen Shah Afridi    11       1   134      11
##  2 Junaid Khan             5       0   154       8
##  3 Imran Tahir             5       0    74       6
##  4 Mohammad Ilyas          5       0    93       4
##  5 Haris Rauf              7       0   154       3
##  6 D Wiese                 7       0    92       3
##  7 Mohammad Irfan          5       0    91       3
##  8 S Lamichhane            5       0    74       3
##  9 SP Narine               8       0    48       3
## 10 MM Ali                  3       0    30       3
## # … with 30 more rows

D. Women Big Bash League

D1.Bowling scorecard

load("../../../yorkrData2020/wbb/wbbMatches/Hobart Hurricanes-Brisbane Heat-2018-12-30.RData")
hh_bh_match <- overs
teamBowlingScorecardMatch(hh_bh_match,'Brisbane Heat')
## # A tibble: 6 x 5
##   bowler      overs maidens  runs wickets
##   <chr>       <int>   <int> <dbl>   <dbl>
## 1 DM Kimmince     3       0    31       2
## 2 GM Harris       4       0    23       3
## 3 H Birkett       1       0     7       0
## 4 JL Barsby       3       0    21       0
## 5 JL Jonassen     4       0    33       0
## 6 SJ Johnson      4       0    17       0

D2.Team batsmen partnerships

load("../../../yorkrData2020/wbb/wbbAllMatchesAllTeams/allMatchesAllOpposition-Perth Scorchers.RData")
ps_matches <- matches
teamBatsmenPartnershipAllOppnAllMatchesPlot(ps_matches,"Perth Scorchers",main="Perth Scorchers")

As mentioned above, I have randomly picked 2 yorkr functions for each of the T20 formats. You can use any of the 90+ functions for analysis of matches, teams, batsmen and bowlers.

1a. Ranking Big Bash League (BBL) batsman

## # A tibble: 62 x 4
##    batsman      matches meanRuns meanSR
##    <chr>          <int>    <dbl>  <dbl>
##  1 DJM Short         44     41.6   126.
##  2 SE Marsh          48     39.1   120.
##  3 AJ Finch          60     36.0   130.
##  4 AT Carey          36     35.9   129.
##  5 KP Pietersen      31     33.5   118.
##  6 UT Khawaja        40     31.5   112.
##  7 BJ Hodge          38     31.5   127.
##  8 CA Lynn           72     31.3   128.
##  9 MP Stoinis        53     30.7   112.
## 10 TM Head           45     30     131.
## # … with 52 more rows

1b. Ranking Big Bash League (BBL) bowlers

## # A tibble: 53 x 4
##    bowler         matches totalWickets meanER
##    <chr>            <int>        <dbl>  <dbl>
##  1 SA Abbott           60           90   8.42
##  2 AJ Tye              45           69   7.32
##  3 B Laughlin          48           66   7.96
##  4 BCJ Cutting         71           63   8.87
##  5 BJ Dwarshuis        54           62   7.87
##  6 MG Neser            54           57   8.36
##  7 Rashid Khan         40           55   6.32
##  8 JP Behrendorff      41           53   6.55
##  9 SNJ O'Keefe         53           52   6.76
## 10 A Zampa             42           51   7.34
## # … with 43 more rows

2a. Ranking Natwest T20 League (NTB) batsman


## # A tibble: 42 x 4
##    batsman          matches meanRuns meanSR
##    <chr>              <int>    <dbl>  <dbl>
##  1 SR Hain               24     34.6   107.
##  2 M Klinger             26     34.1   118.
##  3 MH Wessels            26     33.9   122.
##  4 DJ Bell-Drummond      21     33.1   112.
##  5 DJ Malan              26     33     129.
##  6 T Kohler-Cadmore      23     33.0   118.
##  7 A Lyth                22     31.4   150.
##  8 JJ Cobb               26     30.7   110.
##  9 CA Ingram             25     30.5   153.
## 10 IA Cockbain           26     29.8   121.
## # … with 32 more rows

2b. Ranking Natwest T20 League (NTB) bowlers


## # A tibble: 23 x 4
##    bowler          matches totalWickets meanER
##    <chr>             <int>        <dbl>  <dbl>
##  1 HF Gurney            23           45   8.63
##  2 AJ Tye               26           40   7.81
##  3 TS Roland-Jones      26           37   8.10
##  4 BAC Howell           20           35   6.89
##  5 TT Bresnan           21           31   8.82
##  6 MJJ Critchley        25           31   7.33
##  7 LA Dawson            24           30   6.80
##  8 TK Curran            23           28   8.19
##  9 NA Sowter            25           28   8.09
## 10 MTC Waller           25           27   7.59
## # … with 13 more rows

3a. Ranking Pakistan Super League (PSL) batsman


## # A tibble: 47 x 4
##    batsman      matches meanRuns meanSR
##    <chr>          <int>    <dbl>  <dbl>
##  1 Babar Azam        40     33.7   102.
##  2 L Ronchi          31     32.9   143.
##  3 DR Smith          24     30.8   111.
##  4 JJ Roy            15     30.6   123.
##  5 Kamran Akmal      46     30.1   112.
##  6 SR Watson         40     29.2   126.
##  7 Shoaib Malik      35     28.1   113.
##  8 Fakhar Zaman      38     27.6   119.
##  9 Imam-ul-Haq       15     27.4   115.
## 10 RR Rossouw        36     27.0   130.
## # … with 37 more rows

3b. Ranking Pakistan Super League (PSL) bowlers


## # A tibble: 25 x 4
##    bowler              matches totalWickets meanER
##    <chr>                 <int>        <dbl>  <dbl>
##  1 Wahab Riaz               44           70   6.94
##  2 Hasan Ali                41           61   7.43
##  3 Faheem Ashraf            30           50   7.84
##  4 Mohammad Amir            38           48   7.16
##  5 Usman Shinwari           26           43   8.64
##  6 Mohammad Sami            29           40   7.60
##  7 Shadab Khan              40           38   7.57
##  8 Shaheen Shah Afridi      24           34   7.88
##  9 Rumman Raees             24           33   7.77
## 10 Mohammad Hasnain         16           28   8.65
## # … with 15 more rows

4a. Ranking Women’s Big Bash League (WBB) batsman

## # A tibble: 36 x 4
##    batsman    matches meanRuns meanSR
##    <chr>        <int>    <dbl>  <dbl>
##  1 BL Mooney       27     46.7  129. 
##  2 SFM Devine      22     43.5  111. 
##  3 EA Perry        16     41.1   97.1
##  4 MM Lanning      19     38     98.2
##  5 JE Cameron      22     32.9  127. 
##  6 DN Wyatt        24     32    112. 
##  7 AE Jones        17     28.9  107. 
##  8 AJ Healy        19     28.4  122. 
##  9 M du Preez      19     27    101. 
## 10 L Lee           18     26.9   98.9
## # … with 26 more rows

4b. Ranking Women’s Big Bash League (WBB) bowlers

## # A tibble: 31 x 4
##    bowler      matches totalWickets meanER
##    <chr>         <int>        <dbl>  <dbl>
##  1 M Strano         23           37   7.25
##  2 DM Kimmince      24           36   7.46
##  3 SJ Coyte         22           29   7.59
##  4 JL Jonassen      24           28   6.81
##  5 SJ Johnson       24           27   6.61
##  6 ML Schutt        22           26   6.03
##  7 SFM Devine       22           24   7.58
##  8 M Brown          23           23   7.33
##  9 M Kapp           19           23   5.05
## 10 H Graham         19           22   7.68
## # … with 21 more rows


yorkr can handle ODI and T20 matches in the format as represented in Cricsheet. In my posts, I have shown how yorkr can be used for Intl. ODI and Intl. T20 for both men and women. yorkr can also handle all T20 formats like IPL T20, BBL, Natwest T20, PSL and women’s BBL. Go ahead take yorkr for a ride and check out your favorite teams and players.

Hope you have fun!!!

You may also like

  1. Getting started with Tensorflow, Keras in Python and R
  2. Computer Vision: Ramblings on derivatives, histograms and contours
  3. Cricpy adds team analytics to its arsenal!!
  4. Sixer – R package cricketr’s new Shiny avatar
  5. Big Data-2: Move into the big league:Graduate from R to SparkR
  6. Practical Machine Learning with R and Python – Part 5
  7. Deep Learning from first principles in Python, R and Octave – Part 7
  8. Exploring Quantum Gate operations with QCSimulator
  9. GooglyPlus: yorkr analyzes IPL players, teams, matches with plots and tables

To see all posts click Index of Posts

Revitalizing R package yorkr

There is nothing so useless as doing efficiently that which should not be done at all. Peter Drucker

The most important thing in communication is to hear what isn’t being said. Peter Drucker

“Work expands to fill the time available for its completion.” Corollary: “Expenditure rises to meet income.” Parkinson’s law


“Operation successful!!!the Programmer Surgeon in me, thought to himself. What should have been a routine surgery, turned out to be a major operation in the end, which involved several grueling hours. The surgeon looked at the large chunks of programming logic in the operation tray, which had been surgically removed, as they had outlived their utility and had partly become dysfunctional. The surgeon glanced at the new, concise code logic which had replaced the earlier somewhat convoluted logic, with a smile of satisfaction,

To, those who tuned in late, I am referring to my R package yorkr which I had created in many years ago, in early 2016. The package had worked well for quite some time on data from Cricsheet. Cricsheet went into a hiatus in late 2017-2018, and came alive back in 2019. Unfortunately, a key function in the package, started to malfunction. The diagnosis was that the format of the YAML files had changed, in newer files, which resulted in the problem. I had got mails from users mentioning that yorkr was not converting the new YAML files. This was on my to do list for a long time, and a week or two back, I decided to “bite the bullet” and fix the issue. I hoped the fix would be trivial but it was anything but. Finally, I took the hard decision of re-designing the core of the yorkr package, which involved converting YAML files to RData (dataframes). Also, since it has been a while since I did R code, having done more of Python stuff in recent times, I had to jog my memory with my earlier 2 posts Essential R and R vs Python

I spent many hours, tweaking and fixing the new logic so that it worked on the older and new files. Finally, I am happy to say that the new code is much more compact and probably less error prone.

I also had to ensure that the converted files performed exactly on all the other yorkr functions. I ran all the my yorkr functions in my yorkr posts on ODI, Intl. T20 and IPL and made sure the results were identical. (Phew!!)

The changes will be available in CRAN in yorkr_0.0.8

Do take a look at my yorkr posts. All the functions work correctly. Do use help, as I have changed a few functions. I will have my posts reflect the correct usage, but some function or other may slip the cracks.

  1. One Day Internationals ODI-Part1ODI-Part2ODI-Part3ODI-Part4
  2. International T20s – T20-Part1,T20-Part2,T20-Part3,T20-Part4
  3. Indian Premier League IPL-Part1IPL-Part2,IPL-Part3IPL-Part4

While making the changes, I also touched up some functions and made them more user friendly (added additional arguments etc). But by and large, yorkr is still yorkr and is intact.It just sports some spanking, new YAML conversion logic.


  1. The code is available in Github yorkr
  2. This RMarkdown has been published at RPubs Revitalizing yorkr
  3. I have already converted the YAML files for ODI, Intl T20 and IPL. You can access and download the converted data from Github at yorkrData2020
install.packages("yorkr_0.0.8.tar.gz",repos = NULL, type="source")

Below I rank batsmen and bowlers in ODIs, T20 and IPL based on the data from Cricsheet.

1a. Rank ODI Batsmen



## # A tibble: 151 x 4
##    batsman        matches meanRuns meanSR
##    <chr>            <int>    <dbl>  <dbl>
##  1 Babar Azam          52     50.2   87.2
##  2 SD Hope             51     48.7   71.0
##  3 V Kohli            207     48.4   79.4
##  4 HM Amla            159     46.6   82.4
##  5 DA Warner          114     46.1   88.0
##  6 AB de Villiers     190     45.5   94.5
##  7 JE Root            108     44.9   82.5
##  8 SR Tendulkar        96     43.9   77.1
##  9 IJL Trott           63     43.1   68.9
## 10 Q de Kock          106     42.0   82.7
## # … with 141 more rows

1b. Rank ODI Bowlers


## # A tibble: 265 x 4
##    bowler           matches totalWickets meanER
##    <chr>              <int>        <dbl>  <dbl>
##  1 SL Malinga           191          308   5.25
##  2 MG Johnson           142          238   4.73
##  3 Shakib Al Hasan      157          214   4.72
##  4 Shahid Afridi        166          213   4.69
##  5 JM Anderson          143          207   4.96
##  6 KMDN Kulasekara      161          190   4.94
##  7 SCJ Broad            115          189   5.31
##  8 DW Steyn             114          188   4.96
##  9 Mashrafe Mortaza     139          180   4.97
## 10 Saeed Ajmal          106          180   4.17
## # … with 255 more rows

2a. Rank T20 Batsmen


## # A tibble: 43 x 4
##    batsman          matches meanRuns meanSR
##    <chr>              <int>    <dbl>  <dbl>
##  1 V Kohli               61     39.0   132.
##  2 Mohammad Shahzad      52     31.8   123.
##  3 CH Gayle              50     31.1   124.
##  4 BB McCullum           69     30.7   126.
##  5 PR Stirling           66     29.6   116.
##  6 MJ Guptill            70     29.6   125.
##  7 DA Warner             75     29.1   128.
##  8 AD Hales              50     28.1   120.
##  9 TM Dilshan            78     26.7   105.
## 10 RG Sharma             72     26.4   120.
## # … with 33 more rows

2b. Rank T20 Bowlers



## # A tibble: 153 x 4
##    bowler          matches totalWickets meanER
##    <chr>             <int>        <dbl>  <dbl>
##  1 SL Malinga           78          115   7.39
##  2 Shahid Afridi        89           98   6.80
##  3 Saeed Ajmal          62           92   6.30
##  4 Umar Gul             56           87   7.40
##  5 KMDN Kulasekara      56           72   7.25
##  6 TG Southee           55           69   8.68
##  7 DJ Bravo             60           69   8.41
##  8 DW Steyn             47           69   7.00
##  9 Shakib Al Hasan      57           69   6.82
## 10 SCJ Broad            55           68   7.83
## # … with 143 more rows

3a. Rank IPL Batsmen


## # A tibble: 69 x 4
##    batsman        matches meanRuns meanSR
##    <chr>            <int>    <dbl>  <dbl>
##  1 DA Warner          130     37.9   128.
##  2 CH Gayle           125     36.2   134.
##  3 SE Marsh            67     35.9   120.
##  4 MEK Hussey          59     33.8   105.
##  5 KL Rahul            59     33.5   128.
##  6 V Kohli            175     31.6   119.
##  7 AM Rahane          116     30.7   108.
##  8 AB de Villiers     141     30.3   135.
##  9 F du Plessis        65     29.4   117.
## 10 S Dhawan           140     29.0   114.
## # … with 59 more rows

3a. Rank IPL Bowlers


## # A tibble: 143 x 4
##    bowler          matches totalWickets meanER
##    <chr>             <int>        <dbl>  <dbl>
##  1 SL Malinga          120          184   6.99
##  2 SP Narine           108          137   6.71
##  3 Harbhajan Singh     131          134   7.11
##  4 DJ Bravo             85          118   8.18
##  5 B Kumar              86          116   7.43
##  6 YS Chahal            82          102   7.85
##  7 R Ashwin             92           98   6.81
##  8 JJ Bumrah            76           91   7.47
##  9 PP Chawla            85           87   8.02
## 10 RA Jadeja            89           85   7.93
## # … with 133 more rows


Go ahead and give yorkr a spin once yorkr_0.0.8 is available in CRAN. I hope you have fun. Do get back to me if you have any issues.

I’ll be back. Watch this space!!

You may also like

  1. The mechanics of Convolutional Neural Networks in Tensorflow and Keras
  2. Big Data-5: kNiFi-ing through cricket data with yorkpy
  3. Using Linear Programming (LP) for optimizing bowling change or batting lineup in T20 cricket
  4. Re-introducing cricketr! : An R package to analyze performances of cricketers
  5. Deep Learning from first principles in Python, R and Octave – Part 6
  6. A primer on Qubits, Quantum gates and Quantum Operations
  7. Practical Machine Learning with R and Python – Part 3
  8. Pitching yorkpy … short of good length to IPL – Part 1

To see all posts click Index of posts

Big Data 6: The T20 Dance of Apache NiFi and yorkpy

“I don’t count my sit-ups. I only start counting once it starts hurting. ”

Muhammad Ali

“Hard work beats talent when talent doesn’t work hard.”

Tim Notke

In my previous post Big Data 5: kNiFI-ing through cricket data with Apache NiFi and yorkpy, I created a Big Data Pipeline that takes raw data in YAML format from a Cricsheet to processing and ranking IPL T20 players. In that post I had mentioned that we could create a similar pipeline to create a real time dashboard of IPL Analytics. I could have have done this but I needed to know how to create a Web UI. After digging and poking around, I have been able to create a simple Web UI running off Apache Web server. This UI uses basic JQuery and CSS to display a real time IPL T20 dashboard. As in my previous post, this is an end-2-end Big Data pipeline which can handle large data sets at scheduled times, process them and generate real time dashboards.

We could imagine an inter-galactic T20 championship league where T20 data comes in every hour or sooner and we need to perform analytics to see if us earthlings are any better than people with pointy heads  or little green men. The NiFi pipeline could be used as-is, however the yorkpy package would have to be rewritten in Pyspark. That is in another eon, though.

My package yorkpy has around ~45+ functions which fall in the following main categories

1. Pitching yorkpy . short of good length to IPL – Part 1 :Class 1: This includes functions that convert the yaml data of IPL matches into Pandas dataframe which are then saved as CSV. This part can perform analysis of individual IPL matches.
2. Pitching yorkpy.on the middle and outside off-stump to IPL – Part 2 :Class 2:This part includes functions to create a large data frame for head-to-head confrontation between any 2IPL teams says CSK-MI, DD-KKR etc, which can be saved as CSV. Analysis is then performed on these team-2-team confrontations.
3. Pitching yorkpy.swinging away from the leg stump to IPL – Part 3 Class 3:The 3rd part includes the performance of any IPL team against all other IPL teams. The data can also be saved as CSV.
4. Pitching yorkpy … in the block hole – Part 4 :Class 4: This part performs analysis of individual IPL batsmen and bowlers


Watch the live demo of the end-2-end NiFi pipeline at ‘The T20 Dance

You can download the NiFi template and associated code from Github at  T20 Dance

The Apache NiFi Pipeline is shown below

1. T20 Dance – Overall NiFi Pipeline


There are 5 process groups

2. ListAndConvertYaml2DataFrames

This post starts with having the YAML files downloaded and unpacked from Cricsheet.  The individual YAML files are converted into Pandas dataframes and saved as CSV. A concurrency of 12 is used to increase performance and process YAML files in parallel. The processor MergeContent creates a merged content to signal the completion of conversion and triggers the other Process Groups through a funnel.


3. Analyse individual IPL T20 matches

This Process Group ‘Analyse T20 matches’  used the yorkpy’s Class 1 functions which can perform analysis of individual IPL T20 matches. The matchWorm() and matchScorecard() functions are used, through any other function could have been used. The Process Group is shown below


4. Analyse performance of an IPL team in all matches against another IPL team

This Process Group ‘Analyse performance of IPL team in all matched against another IPL team‘ does analysis in all matches between any 2 IPL teams (Class 2) as shown below

5. Analyse performance of IPL team in all matches against all other IPL teams

This uses Class 3 functions. Individual data sets for each IPL team versus all other IPL teams is created before Class 3 yorkpy functions are invoked. This is included below

6. Analyse performances of IPL batsmen and bowlers

This Process Group uses Class 4 yorkpy functions. The match CSV files are processed to get batting and bowling details before calling the individual functions as shown below


7. IPL T20 Dashboard

The IPL T20 Dashboard is shown



This NiFI pipeline was done for IPL T20 however, it could be done for any T20 format like Intl T20, BBL, Natwest etc which are posted in Cricsheet. Also, only a subset of the yorkpy functions were used. There is a much wider variety of functions available.

Hope the T20 dance got your foot a-tapping!


You may also like
1. A primer on Qubits, Quantum gates and Quantum Operations
2.Computer Vision: Ramblings on derivatives, histograms and contours
3.Deep Learning from first principles in Python, R and Octave – Part 6
4.A Bluemix recipe with MongoDB and Node.js
5.Practical Machine Learning with R and Python – Part 4
6.Simulating the domino effect in Android using Box2D and AndEngine

To see all posts click Index of posts

Cricketr learns new tricks : Performs fine-grained analysis of players

“He felt that his whole life was some kind of dream and he sometimes wondered whose it was and whether they were enjoying it.”

“The ships hung in the sky in much the same way that bricks don’t.”

“We demand rigidly defined areas of doubt and uncertainty!”

“For a moment, nothing happened. Then, after a second or so, nothing continued to happen.”

“The Answer to the Great Question… Of Life, the Universe and Everything… Is… Forty-two,’ said Deep Thought, with infinite majesty and calm.”

                 The Hitchhiker's Guide to the Galaxy - Douglas Adams


In this post, I introduce 2 new functions in my R package ‘cricketr’ (cricketr v0.22) see Re-introducing cricketr! : An R package to analyze performances of cricketers which enable granular analysis of batsmen and bowlers. They are

  1. Step 1: getPlayerDataHA – This function is a wrapper around getPlayerData(), getPlayerDataOD() and getPlayerDataTT(), and adds an extra column ‘homeOrAway’ which says whether the match was played at home/away/neutral venues. A CSV file is created with this new column.
  2. Setp 2: getPlayerDataOppnHA – This function allows you to slice & dice the data for batsmen and bowlers against specific oppositions, at home/away/neutral venues and between certain periods. This reduced subset of data can be used to perform analyses. A CSV file is created as an output based on the parameters of opposition, home or away and the interval of time

Note All the existing cricketr functions can be used on this smaller fine-grained data set for a closer analysis of players

Note 1: You have to call the above functions only once. You can reuse the CSV files in other functions

Important note: Don’t go too fine-grained by choosing just one opposition, in one of home/away/neutral and for too short a period. Too small a dataset may defeat the purpose of the analysis!

This post has been published in Rpubs and can be accessed at Cricketr learns new tricks

You can download a PDF version of this post at Cricketr learns new tricks

If you are passionate about cricket, and love analyzing cricket performances, then check out my racy book on cricket ‘Cricket analytics with cricketr and cricpy – Analytics harmony with R & Python’! This book discusses and shows how to use my R package ‘cricketr’ and my Python package ‘cricpy’ to analyze batsmen and bowlers in all formats of the game (Test, ODI and T20). The paperback is available on Amazon at $21.99 and  the kindle version at $9.99/Rs 449/-. A must read for any cricket lover! Check it out!!


1. Analyzing Tendulkar at 3 different stages of his career

The following functions analyze Sachin Tendulkar during 3 different periods of his illustrious career. a) 1st Jan 2001-1st Jan 2002 b) 1st Jan 2005-1st Jan 2006 c) 1st Jan 2012-1st Jan 2013

# Get the homeOrAway dataset for Tendulkar in matches
#Note: I have commented the lines to getPlayerDataHA() as I already have 
# CSV file

# Get Tendulkar's data for 2001-02

# Get Tendulkar's data for 2005-06


# Get Tendulkar's data for 20012-13
#                        startDate="2012-01-01",endDate="2013-01-01")


1a Mean strike rate of Tendulkar in 2001,2005,2012

Note: Any of the cricketr R functions can be used on the fine-grained subset of data as below. The mean strike rate of Tendulkar is of the order of 60+ in 2001 which decreases to 50 and later to around 45

# Compute and plot mean strike rate of Tendulkar in the 3 periods
batsmanMeanStrikeRate ("./tendulkarTest2001.csv","Tendulkar-2001")

batsmanMeanStrikeRate ("./tendulkarTest2005.csv","Tendulkar-2005")

batsmanMeanStrikeRate ("./tendulkarTest2012.csv","Tendulkar-2012")

1b. Plot the performance of Tendulkar at venues during 2001,2005,2012

On an average Tendulkar score 60+ in 2001 and is really blazing. This performance decreases in 2005 and later in 2012




1c. Plot the performance of Tendulkar against different oppositions during 2001,2005,2012

Sachin uniformly scores 50+ against formidable oppositions in 2001. In 2005 this decreases to 40 in 2005 and in 2012 it is around 25



1d. Plot the relative cumulative average and relative strike rate of Tendulkar in 2001,2005,2012

The plot below compares Tendulkar’s cumulative strike rate and cumulative average during 3 different stages of his career

  1. The cumulative average runs of Tendulkar is in the high 60+ in 2001, which drops to ~50 in 2005 and later plummets to the low 25s in 2012
  2. The strike rate in 2001 for Tendulkar is amazing 60+


2. Analyzing Virat Kohli’s performance against England in England in 2014 and 2018

The analysis below looks at Kohli’s performance against England in ‘away’ venues (England) in 2014 and 2018

# Get the homeOrAway data for Kohli in Test matches

# Get the subset if data of Kohli's performance against England in England in 2014

# Get the subset if data of Kohli's performance against England in England in 2018

2a. Kohli’s performance at England grounds in 2014 & 2018

Kohli had a miserable outing to England in 2014 with a string of low scores. In 2018 Kohli pulls himself out of the morass



2a. Kohli’s cumulative average runs in 2014 & 2018

Kohli’s cumulative average runs in 2014 is in the low 15s, while in 2018 it is 70+. Kohli stamps his class back again and undoes the bad memories of 2014

batsmanCumulativeAverageRuns("kohliTestEng2014.csv", "Kohli-Eng-2014")

batsmanCumulativeAverageRuns("kohliTestEng2018.csv", "Kohli-Eng-2018")

3a. Compare the performances of Ganguly, Dravid and VVS Laxman against opposition in ‘away’ matches in Tests

The analyses below compares the performances of Sourav Ganguly, Rahul Dravid and VVS Laxman against Australia, South Africa, and England in ‘away’ venues between 01 Jan 2002 to 01 Jan 2008

#Get the HA data for Ganguly, Dravid and Laxman

# Slice the data 
                       ,opposition=c("Australia", "England", "South Africa"),

                       ,opposition=c("Australia", "England", "South Africa"),

                       ,opposition=c("Australia", "England", "South Africa"),

3b Plot the relative cumulative average runs and relative cumative strike rate

Plot the relative cumulative average runs and relative cumative strike rate of Ganguly, Dravid and Laxman

-Dravid towers over Laxman and Ganguly with respect to cumulative average runs. – Ganguly has a superior strike rate followed by Laxman and then Dravid



4. Compare the ODI performances of Rohit Sharma, Joe Root and Kane Williamson against opposition

Compare the performances of Rohit Sharma, Joe Root and Kane williamson in away & neutral venues against Australia, West Indies and Soouth Africa

  • Joe Root piles us the runs in about 15 matches. Rohit has played far more ODIs than the other two and averages a steady 35+
# Get the ODI HA data for Rohit, Root and Williamson

# Subset the data for specific opposition in away and neutral venues
                       ,opposition=c("Australia", "West Indies", "South Africa"),

                       ,opposition=c("Australia", "West Indies", "South Africa"),

                       ,opposition=c("Australia", "West Indies", "South Africa"),

4a. Compare cumulative strike rates and cumulative average runs of Rohit, Root and Williamson

The relative cumulative strike rate of all 3 are comparable

names=list("Rohit-ODI-AusWISA","Joe Root-ODI-AusWISA","Williamson-ODI-AusWISA")


5. Plot the performance of Dhoni in T20s against specific opposition at all venues

Plot the performances of Dhoni against Australia, West Indies, South Africa and England

# Get the HA T20 data for Dhoni

#Subset the data
                       ,opposition=c("Australia", "West Indies", "South Africa","England"),

5a. Plot Dhoni’s performances in T20

Note You can use any of cricketr’s functions against the fine grained data





6. Compute and performances of Anil Kumble, Muralitharan and Warne in ‘away’ test matches

Compute the performances of Kumble, Warne and Maralitharan against New Zealand, West Indies, South Africa and England in pitches that are not ‘home’ pithes

# Get the bowling data for Kumble, Warne and Muralitharan in Test matches

# Subset the data
                       ,opposition=c("New Zealand", "West Indies", "South Africa","England"),

                       ,opposition=c("New Zealand", "West Indies", "South Africa","England"),

                       ,opposition=c("New Zealand", "West Indies", "South Africa","England"),

6a. Plot the average wickets of Kumble, Warne and Murali




6b. Plot the average wickets in different grounds of Kumble, Warne and Murali




6c. Plot the cumulative average wickets and cumulative economy rate of Kumble, Warne and Murali

  • Murali has the best economy rate followed by Kumble and then Warne
  • Again Murali has the best cumulative average wickets followed by Warne and then Kumble


7. Compute and plot the performances of Bumrah in 2016, 2017 and 2018 in ODIs

# Get the HA data for Bumrah in ODI in bowling
## [1] "Working..."
# Slice the data for periods 2016, 2017 and 2018



7a. Compute the performances of Bumrah in 2016, 2017 and 2018

  • Very clearly Bumrah is getting better at his art. His economy rate in 2018 is the best!!!
  • Bumrah has had a very prolific year in 2017. However all the years he seems to be quite effective


8. Compute and plot the performances of Shakib, Bumrah and Jadeja in T20 matches for bowling

# Get the HA bowling data for Shakib, Bumrah and Jadeja
## [1] "Working..."
## [1] "Working..."
## [1] "Working..."
# Slice the data for performances against Sri Lanka, Australia, South Africa and England
                       ,opposition=c("Sri Lanka","Australia", "South Africa","England"),
                       ,opposition=c("Sri Lanka","Australia", "South Africa","England"),

                       ,opposition=c("Sri Lanka","Australia", "South Africa","England"),

8a. Compare the relative performances of Shakib, Bumrah and Jadeja

  • Jadeja and Bumrah have comparable economy rates. Shakib is more expensive
  • Shakib pips Bumrah in number of cumulative wickets, though Bumrah is close behind



By getting the homeOrAway data for players using the profileNo, you can slice and dice the data based on your choice of opposition, whether you want matches that were played at home/away/neutral venues. Finally by specifying the period for which the data has to be subsetted you can create fine grained analysis.

Hope you have a great time with cricketr!!!

Also see

1. My book ‘Deep Learning from first principles:Second Edition’ now on Amazon
2. Cricpy takes a swing at the ODIs
3. My book ‘Practical Machine Learning in R and Python: Third edition’ on Amazon
4. Googly: An interactive app for analyzing IPL players, matches and teams using R package yorkr
5. Big Data-2: Move into the big league:Graduate from R to SparkR
6. Rock N’ Roll with Bluemix, Cloudant & NodeExpress
7. A method to crowd source pothole marking on (Indian) roads
8. De-blurring revisited with Wiener filter using OpenCV

To see all posts click Index of posts