GooglyPlusPlus2021: Restarting IPL 2021 as-it-happens!!!

The IPL 2021 extravaganza has restarted again, now in Dubai, and it was time for me to crank up good ol’ GooglyPlusPlus2021. As in my earlier post, GooglyPlus2021 with IPL 2021 as it happens, during the initial set of IPL 2021 games,, a command script will execute automatically every day, download the latest data files, unzip, sort, process and put them in appropriate directories so that GooglyPlusPlus can work its magic on the data, with my R package yorkr. You can do analysis of IPL 2021 matches, batsmen, bowlers, historical performance analysis of head-to-head clashes and performances of teams.

Note: Since the earlier instalment of IPL 2021, there are 2 key changes that have taken place in GooglyPlusPlus.

Now,

a) All charts are interactive. You can hover over charts, click, double-click to get more details. To see more details on how to use the interactive charts, see my post GooglyPlusPlus2021 is now fully interactive!

b) You can now analyse historical performances, compute team batting and bowling scorecards for specified periods. To know details see GooglyPlusPlus2021 adds new bells and whistles!

You can try out my app GooglyPlusPlus2021 by clicking GooglyPlusPlus2021

The code for my R package yorkr is available at Github at yorkr

You can clone/fork GooglyPlusPlus2021 from github at gpp2021-6

IPL 2021 is already underway.

Some key analysis and highlights of the 2 recently concluded IPL matches

  • CSK vs MI
  • KKR vs RCB

a) CSK vs MI (19 Sep 2021) – Batting Partnerships (CSK)

b) CSK vs MI (19 Sep 2021) – Bowling scorecard (MI)

c) CSK vs MI (19 Sep 2021) – Match worm chart

Even though MI had a much better start and were cruising along to a victory, they lost the plot around the 18.1 th over as seen below (hover on the chart)

d

d) KKR vs RCB ( 20 Sep 2021) – Bowling wicket match

This chart gives the wickets taken by the bowler and the total runs conceded

e) KKR vs RCB ( 20 Sep 2021) – Match worm chart

This was a no contest. RCB batting was pathetic and KKR blasted their way to victory as seen in this worm chart

Note: You can also do historical analysis of teams with GooglyPlusPlus2021

For the match to occur today PBKS vs RR (21 Sep 2021) we can perform head-to-head historical analysis. Here Kings XI Punjab has been chosen instead of Punjab Kings as that was its name.

f) Head-to-head (PBKS vs RR) today’s match 21 Sep 2021

For the Rajasthan Royals Sanjy Samson and Jos Buttler have the best performance from 2018 -2021 as seen below

For Punjab Kings KL Rahul and Chris Gayle are the leading scorers for the period 2018-2021

g) Current ranking of batsmen IPL 2021

h) Current ranking of bowlers IPL 2021

Also you analyse individual batsman and bowlers

i) Batsman analysis

To see Rituraj Gaikwad performance checkout the batsman tab

j) Bowler analysis

Performance of Varun Chakaravarty

Remember to check out GooglyPlusPlus2021 for your daily analysis of matches, teams, batsmen and bowlers. Your ride will be waiting for you!!!

You can clone/fork GooglyPlusPlus2021 from github at gpp2021-6

GooglyPlusPlus2021 has been updated with all completed 31 matches

 

Mumbai Indians-Royal Challengers Bangalore-2021-04-09Chennai Super Kings-Delhi Capitals-2021-04-10
Kolkata Knight Riders-Sunrisers Hyderabad-2021-04-11Punjab Kings-Rajasthan Royals-2021-04-12
Mumbai Indians-Kolkata Knight Riders-2021-04-13Royal Challengers Bangalore-Sunrisers Hyderabad-2021-04-14
Delhi Capitals-Rajasthan Royals-2021-04-15Punjab Kings-Chennai Super Kings-2021-04-16
Mumbai Indians-Sunrisers Hyderabad-2021-04-17Royal Challengers Bangalore-Kolkata Knight Riders-2021-04-18
Punjab Kings-Delhi Capitals-2021-04-18Chennai Super Kings-Rajasthan Royals-2021-04-19
Mumbai Indians-Delhi Capitals-2021-04-20Punjab Kings-Sunrisers Hyderabad-2021-04-21
Chennai Super Kings-Kolkata Knight Riders-2021-04-21Rajasthan Royals-Royal Challengers Bangalore-2021-04-22
Mumbai Indians-Punjab Kings-2021-04-23Kolkata Knight Riders-Rajasthan Royals-2021-04-24
Chennai Super Kings-Royal Challengers Bangalore-2021-04-25Delhi Capitals-Sunrisers Hyderabad-2021-04-25
Punjab Kings-Kolkata Knight Riders-2021-04-26Royal Challengers Bangalore-Delhi Capitals-2021-04-27
Sunrisers Hyderabad-Chennai Super Kings-2021-04-28Rajasthan Royals-Mumbai Indians-2021-04-29
Kolkata Knight Riders-Delhi Capitals-2021-04-29Punjab Kings-Royal Challengers Bangalore-2021-04-30.RData
Chennai Super Kings-Mumbai Indians-2021-05-01Rajasthan Royals-Sunrisers Hyderabad-2021-05-02
Punjab Kings-Delhi Capitals-2021-05-02Chennai Super Kings-Mumbai Indians-2021-09-19
Royal Challengers Bangalore-Kolkata Knight Riders-2021-09-20Rajasthan Royals-Punjab Kings-2021-09-21
Sunrisers Hyderabad-Delhi Capitals-2021-09-22.RDataMumbai Indians-Kolkata Knight Riders-2021-09-23.RData

Also see

  1. Deep Learning from first principles in Python, R and Octave – Part 5
  2. Introducing QCSimulator: A 5-qubit quantum computing simulator in R
  3. Computer Vision: Ramblings on derivatives, histograms and contours
  4. Designing a Social Web Portal
  5. Understanding Neural Style Transfer with Tensorflow and Keras
  6. Big Data 6: The T20 Dance of Apache NiFi and yorkpy
  7. Practical Machine Learning with R and Python – Part 6
  8. Introducing cricpy:A python package to analyze performances of cricketers
  9. A closer look at “Robot Horse on a Trot” in Android
  10. Cricketr adds team analytics to its repertoire!!!

To see all posts click Index of posts

GooglyPlusPlus2021 adds new bells and whistles!!

This latest update of GooglyPlusPlus2021 includes new controls which allow for granular analysis of teams and matches. This version includes a new ‘Date Range’ widget which will allow you to choose a specific interval between which you would like to analyze data. The Date Range widget has been added to 2 tabs namely

a) Head-to-Head

b) Overall Performance

Important note:

This change is applicable to all T20 formats and ODI formats that GooglyPlusPlus2021 handles. This means you can do fine-grained analysis of the following formats

a. IPL b. Intl. T20 (men) c. Intl. T20 (women)

d. BBL e. NTB f. PSL

g. WBB h. CPL i. SSM

j. ODI (men) k. ODI (women)

Important note 1: Also note that all charts in GooglyPlusPlus2021 are interactive. You ca hover over the charts to get details of the data below. You can also selectively filter in bar charts using double-click and click. To know more about how to use GooglyPlusPlus2021 interactively, please see my post GooglyPlusPlus2021 is now fully interactive!!

You can clone/download the code for GooglyPlusPlus2021 from Github at GooglyPlusPlus2021

Try out GooglyPlusPlus2021 here GooglyPlusPlus2021

Here are some random examples from the latest version of GooglyPlusPlus2021

a) Team Batting Scorecard – MI vs CSK (all matches 2008-2013) – Tendulkar era

Tendulkar is the top scorer, followed by Rohit Sharma and Jayasuriya for Mumbai Indians

b) Team Batting Partnerships (MI -CSK) – Tendulkar’s partnerships

Partnerships for Tendulkar with his MI team mates

c) Team Bowler Wicket Kinds (Opposition countries vs India in all matches in T20)

d) Win vs Loss India vs Australia T20 Women (2010 – 2015)

Australia won all 3 matches against India

e) Win vs Loss India vs Australia T20 Women (2015 – 2020)

Between 2016-2020 the tally is 3-2 for Australia vs India

f) Wins vs Losses – MI vs all other teams 2013 – 2018

g) Team Batting Partnerships Head-to-head Australia vs England ODI (Women)

Partnerships of Australia women EA Perry and AJ Blackwell for Australia

Go ahead give GooglyPlusPlus2021 a try!

Hope you have fun!

Also see

  1. Exploring Quantum Gate operations with QCSimulator
  2. De-blurring revisited with Wiener filter using OpenCV
  3. Deep Learning from first principles in Python, R and Octave – Part 3
  4. Big Data-4: Webserver log analysis with RDDs, Pyspark, SparkR and SparklyR
  5. Cricpy adds team analytics to its arsenal!!
  6. Practical Machine Learning with R and Python – Part 5

To see all posts see Index of posts

GooglyPlusPlus2021 is now fully interactive!!!

GooglyPlusPlus2021 is now fully interactive. Please read the below post carefully to see the different ways you can interact with the data in the plots.

There are 2 main updates in this latest version of GooglyPlusPlus2021

a) GooglyPlusPlus gets all ‘touchy, feely‘ with the data and now you can interact with the plot/chart to get more details of the underlying data. There are many ways you can slice’n dice the data in the charts. The examples below illustrate a few of this. You can interact with plots by hover’ing, ‘click’ing and ‘double-click’ing curves, plots, barplots to get details of the data.

b) GooglyPlusPlus also includes the ‘Super Smash T20’ league from New Zealand. You can analyze batsmen, bowlers, matches, teams and rank Super Smash (SSM) also

Note: GooglyPlusPlus2021 can handle a total of 11 formats including T20 and ODI. They are

i) IPL ii) Intl. T20(men) ii) Intl. T20(women) iv) BBL

v) NTB vi) PSL vii) WBB. viii) CPL

ix) SSM x) ODI (men) xi) ODI (women)

Each of these formats have 7 tabs which are

— Analyze batsman

— Analyze bowlers

— Analyze match

— Head-to-head

— Team vs all other teams

— Rank batsmen

— Rank bowlers

Within these 11 x 7 = 77 tabs you can analyze batsmen, bowlers, matches, head-to-head, team vs all other teams and rank players for T20 and ODI. In addition all plots have been made interactive so there is a lot more information that you can get from these charts

Try out the interactive GooglyPlusPlus2021 now!!!

You can fork/clone the Shiny app from Github at GooglyPlusPlus2021

Below I have randomly included some charts for different formats to show how you can interact with them

a) Batsman Analysis – Runs vs Deliveries (IPL)

Mouse-over/Hover

The plot below gives the number of runs scored by David Warner vs Deliveries faced.

b) Batsman Analysis – Runs vs Deliveries (IPL) (prediction)

Since a 2nd order regression line,with confidence intervals(shaded area), has been fitted in the above plot, we can predict the runs given the ‘balls faced’ as below

Click ‘Toggle Spike lines’ (use palette on top-right)

By using hover(mouse-over) on the curve we can determine the predicted number of runs Warner will score given a certain number of deliveries

c) Bowler Analysis – Wickets against opposition – Intl. T20 (women)

Jhulan Goswami’s wickets against opposition countries in Intl. T20 (women)

d) Bowler Analysis (Predict bowler wickets) IPL – (non-interactive**)

Note: Some plots are non-interactive, like the one below which predicts the number of wickets Bumrah will take based on number of deliveries bowled

e) Match Analysis – Batsmen Partnership -Intl. T20 (men)

India vs England batting partnership between Virat Kohli & Shikhar Dhawan in all matches between England and India

f) Match Analysis – Worm chart (Super Smash T20) SSM

i) Worm chart of Auckland vs Northern Districts (29 Jan 2021).

ii) The final cross-over happens around the 2nd delivery of the 19th over (18.2) as Northern Districts over-takes Auckland to win the match.

g) Head-to-head – Team batsmen vs bowlers (Bangladesh batsmen against Afghanistan bowlers) Intl. T20 (men)

Batting performance of Shakib-al-Hasan (Bangladesh) against Afghanistan bowlers in Intl. T20 (men)

h) Head-to-head – Team batsmen vs bowlers (Bangladesh batsmen against Afghanistan bowlers) Intl. T20 (men)Filter

Double click on Shakib-al-Hasan on the legend to get the performance of Shakib-al-Hasan against Afghanistan bowlers

Avoiding the clutter

i) Head-to-head – Team bowler vs batsmen (Chennai Super Kings bowlers vs Mumbai Indians batsmen) – IPL

If you choose the above option the resulting plot is very crowded as shown below

To get the performance of Mumbai Indian (MI) batsmen (Rohit Sharma & Kieron Pollard) against Chennai Super Kings (CSK) bowlers in all matches do as told below

Steps to avoid clutter in stacked bar plots

1) This can be avoided by selectively choosing to filter out the batsmen we are interested in. say RG Sharma and Kieron Pollard. Then double-ciick RG Sharma, this is will bring up the chart with only RG Sharma as below

2) Now add additional batsmen you are interested in by single-clicking. In the example below Kieron Pollard is added

You can continue to add additional players that you are interested by single clicking.

j) Head-to-head (Performance of Indian batsmen vs Australian bowlers)- ODI

In the plot V Kohli, MS Dhoni and SC Ganguly have been selected for their performance against Australian bowlers (use toggle spike lines)

k) Overall Performance – PSL batting partnership against all teams (Fakhar Zaman)

The plot below shows Fakhar Zaman (Lahore Qalanders) partnerships with other teammates in all matches in PSL.

l) Win-loss against all teams (CPL)

Win-loss chart of Jamaica Talawallahs (CPL) in all matches against all opposition

m) Team batting partnerships against all teams for India (ODI Women)

Batting partnerships of Indian ODI women against all other teams

n) Ranking of batsmen (IPL 2021)

Finally here is the latest ranking of IPL batsmen for IPL 2021 (can be done for all other T20 formats)

o) Ranking of bowlers (IPL 2021)

Clone/download the Shiny app from Github at GooglyPlusPlus2021

So what are you waiting for? Go ahead and try out GooglyPlusPlus2021!

Knock yourself out!

Enjoy enjaami!!!

See also

  1. Deconstructing Convolutional Neural Networks with Tensorflow and Keras
  2. Deep Learning from first principles in Python, R and Octave – Part 6
  3. Cricketr learns new tricks : Performs fine-grained analysis of players
  4. Big Data 6: The T20 Dance of Apache NiFi and yorkpy
  5. Using Linear Programming (LP) for optimizing bowling change or batting lineup in T20 cricket
  6. Practical Machine Learning with R and Python – Part 6
  7. Introducing QCSimulator: A 5-qubit quantum computing simulator in R
  8. Simulating an oscillating revoluteJoint in Android
  9. Benford’s law meets IPL, Intl. T20 and ODI cricket
  10. De-blurring revisited with Wiener filter using OpenCV

To see all posts click Index of posts

GooglyPlusPlus2021 with IPL 2021, as-it-happens!

A brand new season of IPL 2021 is on its way, and I intend to keep my Shiny app GooglyPlusPlus updated with all the analysis “as-it-happens”. I had written a post earlier Big Data 7: yorkr waltzes with Apache NiFi, where I used my R package yorkr in a NiFi pipeline to automate downloading, converting and generating appropriate data for GooglyPlusPlus. However, using Apache NiFi for daily updates would be an overkill. So, I have created a ‘big bash script’ (with shell,R,python scripts) and scheduled daily with CRON, which will get daily updates from Cricsheet, convert the yaml files, generate the necessary data files for GooglyPlusPlus in an automated way, using my R package yorkr, and integrate with the Shiny app

Now, you should be able to do detailed analysis of batsmen, bowlers, IPL matches, IPL teams and also do the ranking of the batsmen and bowlers as new data is added on a daily basis. Also remember that GooglyPlusPlus2021 can do similar analysis for all T20 formats (Intl. T20 (men,women), BBL, NTB, PSL, CPL, WBB etc.)

Check out GooglyPlusPlus2021-IPL 2021

I will be keeping the app updated as data becomes available after the match. Do check it out. Here are some random analysis of the the completed 29 matches (see included matches in table below)

You can download/clone the code for GooglyPlusPlus2021 from Github at gpp2021-1

You can perform analysis of the completed matches in the ‘IPL Match’ tab as shown below

A) Mumbai Indians-Royal Challengers Bangalore-2021-04-09

a) Match scorecard – Mumbai Indians

Note: The scorecards are computed in real time.

b) Batting Partnerships – Royal Challengers Bangalore

c) Bowling Wicket Kind – Royal Challengers Bangalore

B) Chennai Super Kings vs Delhi Capitals – 2021-04-10

d) Batting Partnerships (table) – Delhi Capitals

e) Match Worm Graph

C) Kolkata Knight Riders vs Sunrisers Hyderabad – 2021-04-11

f) Batsmen vs Bowlers

D) Final ranks of IPL 2021 batsmen

E) Final ranks of IPL 2021 bowlers

Incidentally GooglyPlusPlus2021 has also been updated with all matches PSL 2021. Here is a sample

PSL 2021

F) Quetta Gladiators vs Lahore Qalanders – 22-02-2021

G) Ranks of batsmen PSL 2021

H) Ranks bowlers PSL 2021

Important note :

Note: 1) The GooglyPlusPlus2021 Shiny app also includes functions for performing the following analysis namely

  • detailed analysis of batsmen
  • detailed analysis of bowlers
  • match analysis
  • Head-to-head between 2 IPL teams
  • Analysis of IPL team against all other teams
  • Ranking of batsmen based on number of years and matches played
  • Ranking of bowlers based on number of years and matches played

So do check the other tabs of this app

2) GooglyPlusPlus2021 can do similar analysis for other T20 games like Intl. T20 (men,women), BBL, NTB, PSL and so on.

Give GooglyPlusPlus2021 a spin!!

Download/clone the code for GooglyPlusPlus2021 from Github at gpp2021-1

GooglyPlusPlus2021 has been updated with all the completed 29 matches

Mumbai Indians-Royal Challengers Bangalore-2021-04-09Chennai Super Kings-Delhi Capitals-2021-04-10
Kolkata Knight Riders-Sunrisers Hyderabad-2021-04-11Punjab Kings-Rajasthan Royals-2021-04-12
Mumbai Indians-Kolkata Knight Riders-2021-04-13Royal Challengers Bangalore-Sunrisers Hyderabad-2021-04-14
Delhi Capitals-Rajasthan Royals-2021-04-15Punjab Kings-Chennai Super Kings-2021-04-16
Mumbai Indians-Sunrisers Hyderabad-2021-04-17Royal Challengers Bangalore-Kolkata Knight Riders-2021-04-18
Punjab Kings-Delhi Capitals-2021-04-18Chennai Super Kings-Rajasthan Royals-2021-04-19
Mumbai Indians-Delhi Capitals-2021-04-20Punjab Kings-Sunrisers Hyderabad-2021-04-21
Chennai Super Kings-Kolkata Knight Riders-2021-04-21Rajasthan Royals-Royal Challengers Bangalore-2021-04-22
Mumbai Indians-Punjab Kings-2021-04-23Kolkata Knight Riders-Rajasthan Royals-2021-04-24
Chennai Super Kings-Royal Challengers Bangalore-2021-04-25Delhi Capitals-Sunrisers Hyderabad-2021-04-25
Punjab Kings-Kolkata Knight Riders-2021-04-26Royal Challengers Bangalore-Delhi Capitals-2021-04-27
Sunrisers Hyderabad-Chennai Super Kings-2021-04-28Rajasthan Royals-Mumbai Indians-2021-04-29
Kolkata Knight Riders-Delhi Capitals-2021-04-29Punjab Kings-Royal Challengers Bangalore-2021-04-30.RData
Chennai Super Kings-Mumbai Indians-2021-05-01Rajasthan Royals-Sunrisers Hyderabad-2021-05-02
Punjab Kings-Delhi Capitals-2021-05-02

Watch this space!

Also see

  1. Introducing GooglyPlusPlus!!!
  2. Introducing cricket package yorkr: Part 1- Beaten by sheer pace!
  3. Cricketr adds team analytics to its repertoire!!!
  4. Deep Learning from first principles in Python, R and Octave – Part 3
  5. Introducing QCSimulator: A 5-qubit quantum computing simulator in R
  6. Practical Machine Learning with R and Python – Part 5

To see all posts click Index of posts

GooglyPlusPlus2021 bubbles up top T20 players in all formats!

“Would you tell me, please, which way I ought to go from here?”
“That depends a good deal on where you want to get to.”
“I don’t much care where –”
“Then it doesn’t matter which way you go.”
                              Lewis Carroll, Alice in Wonderland

This latest version of GooglyPlusPlus2021 has the following updates

– brand new strategy and algorithm for ranking T20 players in any format (IPL, BBL, NTB, PSL etc.)

– integrates the Caribbean Premier League T20 into the app

– includes the latest BBL matches in 2020-2021

– includes all the latest Natwest T20 matches 2020

– has a new and better user interface

Interestingly the new Ranking algorithm has come just before the IPL auction. Check out who’s who in IPL T20 by taking GooglyPlusPlus2021 for ride!!!

Try out GooglyPlusPlus2021!!

You can clone/fork the code from Github at GooglyPlusPlus2021

1) Ranking Algorithm

In my last post GooglyPlusPlus2021 interactively ranks T20 batsmen and bowlers!!!  I had shown how by changing the number of matches played we see that KL Rahul and Rishabh Pant move above Virat Kohli. That set me thinking. So, I redesigned the ranking so that we start to identify newer players earlier.  This is how the new ranking works.

a) Ranking T20 batsmen :

I have the following controls

a) Since year : This tells the year range  to consider for the  batsmen. This slider moves from right to left

b) Matches played : Number of matches played by the batsman in the year range. This moves left to right

c) Mode : The ranking algorithm sorts on and ranks on 2 columns namely Runs and Strike rate. The mode tells whether to consider Runs over Strike rate of Strike rate over runs. 

By default, the control for “Since year” will be set to the ‘beginning of time’ which for IPL data is 2008. If you move the ‘Since year’ since year slider to the left, the ‘Matches played’ slider will move to the right and the corresponding maximum value of matches played will be set appropriately.

b) Ranking T20 Bowlers:

This has the following controls

a) Since year : This tells the year range to consider for  bowler in IPL. Moves right to left.

b) Matches played : Number of matches played by the bowler in the year range. Move left to right.

c) Mode 1: The bowlers are sorted and ranked on 2 columns namely Wickets and Economy rate. The mode tells whether to consider Wickets over Economy rate or Economy rate over wickets when ranking the bowlers. 

By default, the “Since year” will be set to the year when the T20 data is available. If you move the ‘Since year’ since year slider to the left, the ‘Matches played’ slider will move to the right and the corresponding maximum value of matches played will be set appropriately.

2) Strategy for ranking:

Here is the rationale and philosophy behind these controls

The ranking in my earlier post GooglyPlusPlus2021 interactively ranks T20 batsmen and bowlers!!!  is based on sorting batsmen and bowlers from the start of IPL tournament. Hence we will find players who have played a lot of matches. So in the bowler ranking you will SL Malinga who no longer plays IPL.

Hence I decided that the user should be allowed to choose the start year to consider for the ranking. We can move the “Since year’ slider from right to left to choose the data for the year range we are interested in e.g. (2014-2020, or 2017-2020). Changing the ‘Since year’ slider will obviously change the maximum matches played by any player. However the user can move this slider right to left and decide the number of matches to be considered for a batsman or bowler.  Finally, the ‘Mode’. will allow the user to choose whether the list if batsmen should be ranked first by average runs and then average strike rate or vice versa. In the case of bowlers, the choice is whether to first sort by number of wickets and then economy rate or vice versa. 

The consideration for ‘Mode’ and “Mode1′ is that while runs and wickets are important for batsman or bowler, it is clear the Strike rate and Economy rate become critical in ‘death overs‘. Batsmen who can accelerate the scoring rate when needed and bowlers who can put the brakes on during death overs is of paramount importance in T20 cricket.

Let me walk through a  few scenarios in IPL T20. The same functionality is also available in alll other T20 formats (Intl. T20 (men, women), BBL, NTB, PSL, WBB, CPL etc.)

3) Ranking IPL batsmen

Note: Those players who are underlined in red are the shooting stars**

a) Scenario 1

These are  the consistent and reliable players

Since year = 2013, Matches played = 95 and Mode = ‘Runs over Strike Rate’

 

 

b) Scenario 2

Since year = 2015, Matches played = 67 and Mode = ‘Runs over Strike Rate’

c) Scenario 3

We can turn the above over its head and choose Mode = “Strike rate over Runs’

Since year = 2015, Matches played = 67 and Mode = ‘Strike Rate over Runs’

d) Scenario 4: 

Since year = 2018, Matches played = 35 and Mode = ‘Runs over Strike Rate’

4) Ranking IPL bowlers

e) Scenario 5:

Since year = 2016, Matches played = 55 and Mode1 = ‘ Wickets over Economy Rate’

f) Scenario 6

Since year = 2018, Matches played = 33 and Mode1 = ‘ Economy Rate over wickets’. Note the economy rate is sorted in ascending order

Note:  Incidentally the ranking of IPL and other T20 players in my earlier posts is a special case, when you consider all matches from the beginning of time (or since the data is available, rather than choosing a range from later years).

Note 1: Personally, if I had to form a team, I would choose

– at least 2-3 batsmen who are reliable and have good average runs and a good strike rate

– 2 batsmen who can be used to accelerate strike rate during critical junctures or at death overs

– 2-3 bowlers who have a great record of wicket taking with good economy rate (2 + 1 pace/spin)

– 2 bowlers who have good economy rate over wickets

– 2 all rounders with good batting and bowling average

– 1 wicketkeeper batsmen

The key point is how many matches would you consider as a minimum for reliability or strike rate. More is good but not always better as you could miss out on rising stars* who may be risky but good picks and possibly cost less, since the team composition will also depend on the available budget for each team. You could also check other T20 formats for good T20 players. Creating a T20 team in IPL is an optimization problem where the objective is to maximize the runs, strike rate for batsmen,  or maximizing the wickets taken, while minimizing the economy rate for bowlers with the constraint of the overall budget of the team.

Note 2: The ranking algorithm has been included for all T20 formats in GooglyPlusPlus2020. See below

5) International T20 Batsmen (men) ranking

Since year = 2012, Matches played = 50 and Mode = ‘Runs over Strike Rate’

 

6) International T20 Bowlers (men) ranking

Since year = 2013, Matches played = 33 and Mode1 = ‘Wickets over Economy Rate ‘

Here are some top class T20 bowlers

 

7) International T20 Batsmen (women) ranking

Since year = 2015, Matches played = 29 and Mode = ‘Runs over Strike Rate’. Smriti Mandhana makes it to the top 3 in Runs over Strike Rate!

 

 

and 2nd when Strike rate over runs is considered!

 

8) Integrating Carribean Premier League T20

In this version I have also integrated Carribean Premier League (CPL). It took me about 3 -4  hours of focused work to setup the data and the associated code. Like every other T20 format,  CPL league has access to 100+ functions of yorkr. So you can do analysis of CPL batsmen, bowlers, CPL matches, CPL head-to-head confrontation and CPL team against all other teams besides the ranking function. You can also generate batting and bowling scorecard for matches, for a team against all other teams and the overall scorecard in all matches against all other teams. Here is a random sample

a) C Munro – Cumulative Average Runs

 

b) RR Emrit – Bowler’s wickets against opposition

 

c) Head-to-head (Barbados Tridents vs Antigua Hawkbill all matches)

 

d) Rank CPL batsmen 

Since year= 2016, Matches played = 41 Mode = Strike rate over Runs

Nicholas Pooran tops the list

9) BBL 2020-21

GooglyPlusPlus2021 now has the latest Big Bash League matches of 2020-21, in fact “hot off the pitch“. So you should be able to do all the analysis on current BBL data namely batsmen and bowler analysis, match analysis, team analysis vs another team or against all other teams, and finally ranking of batsmen and bowlers. Here is a sample with yesterday’s match

a) Match scorecard -Perth Scorchers- Sydney Sixers 06 Feb 2021 (Final)

 

b) Predict runs of batsman – CA Lynn

 

10) Natwest T20 Blast 2020

I noticed recently the Cricsheet  has more data. Now NTB data includes all matches till 2020. This data has been incorporated into NTB and you should be able to use all the yorkr functions to analyze batsmen, bowlers, teams, team-vs-team and team vs all other teams, besides the ranking functions. Here are a couple below

a) Head-to-head Derbyshire vs Essex all matches

 

b) Team Batsmen vs Bowlers – All matches all opposition Leicestershire (MJ Cosgrove)

 

Do check out the various functions of GooglyPlusPlus2021. Take a look at the ranks of the T20 batsmen and bowlers. Hope you have a good time!

Take GooglyPlusPlus2021 for a test ride!!

Feel free to clone/fork the code from Github at GooglyPlusPlus2021

Also see

  1. Cricketr adds team analytics to its repertoire!!!
  2. Introducing cricket package yorkr: Part 2-Trapped leg before wicket
  3. Introducing cricpy:A python package to analyze performances of cricketers
  4. Big Data-5: kNiFi-ing through cricket data with yorkpy
  5. Using Linear Programming (LP) for optimizing bowling change or batting lineup in T20 cricket
  6. See all 72+ posts on cricket in R, Python and Octave
  7. Deep Learning from first principles in Python, R and Octave – Part 5
  8. Practical Machine Learning with R and Python – Part 3
  9. Understanding Neural Style Transfer with Tensorflow and Keras

To see all post click Index of posts

 

 

GooglyPlusPlus2021 interactively ranks T20 batsmen and bowlers!!!

Every time I think that I have my R packages or Shiny apps all wrapped up, I find another idea trots up and knocks at my door. Since I intend to keep GooglyPlusPlus current with the latest data, I decided to include the ranking functions in my Shiny app GooglyPlusPlus.

Fortunately, since GooglyPlusPlus is based on my R package ‘yorkr‘ (see Introducing cricket package yorkr: Beaten by sheer pace!), I could make the necessary changes to the ranking functions in the package, so that it could be incorporated into my latest Shiny app GooglyPlusPlus2021!! To know how to use GooglyPlusPlus see my post Introducing GooglyPlusPlus

Note: GooglyPlusPlus can analyze batsmen, bowlers, matches and teams.

Take GooglyPlusPlus2021 for a test drive!!!

You can clone/fork GooglyPlusPlus2021 from Github

Here are a few scenarios from GooglyPlusPlus2021

A) Ranking batsmen

Ranking IPL batsmen (minMatches = 80) – The following table shows the ranking of IPL players who have played 80 matches or more

B) Identifying batsmen of potential and promise

Ranking IPL batsmen (minMatches =70) –  If we reduce the minimum number of matches played to 70, then we see it pushes up KL Rahul above Kohli.

Ranking IPL batsmen (minMatches =60) – When the slider is moved to 60, we see that Rishabh Pant has a better mean average and mean strike rate and is also ranked above Kohli. We can identify promising players this way. However, it is also likely that some players may be just a bright flash in the pan

C) Ranking T20 bowlers (men)

D) Ranking NTB Batsmen

GooglyPlusPlus2021 can rank all T20 formats (IPL, BBL, Intl. T20 (men), Intl. T20 (women), NTB, PSL and WBB. Do give it a try!

Also remember that GooglyPlusPlus2021 includes close to 100+ functions which enable it to perform analysis of batsmen, bowlers, T20 matches, head-to-head confrontation of T20 teams and overall performance of T20 teams . To know more about GooglyPlusPlus2021 see Introducing GooglyPlusPlus

You can download the code for this app from Github at GooglyPlusPlus2021

Do give GooglyPlusPlus2021 a spin!!

I do have some other ideas also which I will be incorporating  into GooglyPlusPlus2021.

Watch this space!!

Also see
1. Deep Learning from first principles in Python, R and Octave – Part 7
2. A method to crowd source pothole marking on (Indian) roads
3. Big Data 7: yorkr waltzes with Apache NiFi
4. Understanding Neural Style Transfer with Tensorflow and Keras
5. Revisiting World Bank data analysis with WDI and gVisMotionChart
6. Natural language processing: What would Shakespeare say?
7. Introducing QCSimulator: A 5-qubit quantum computing simulator in R
8. Introducing cricpy:A python package to analyze performances of cricketers
9. Simulating an Edge Shape in Android

To see all posts click Index of posts

GooglyPlusPlus 2020!!

I have updated my GooglyPlusPlus Shiny app with data from latest IPL 2020. GooglyPlusPlus  2020 is also based on my R package yorkr.  To know more about yorkr (see Revitalizing R package yorkr.) Now you should be able to analyze IPL matches, teams, players upto IPL 2020. Note: My latest GooglyPlusPlus 2020 can analyze all formats of T20 matches. yorkr uses data from Cricsheet

There are 5 tabs in each of the T20 formats

i) Analyze T20 batsmen ii) Analyze T20 bowlers. iii) Analyze T20 match iv) Analyze T20 team

vs another T20 team v) Analyze overall performance of T20 against all other teams

I plan to update GooglyPlusPlus  at least twice a year  to keep it abreast of all the latest data of all T20 formats

In GooglyPlusPlus 2020 you can check out IPL data upto 2020, besides other T20 formats like BBL, PSL, NTB, WBBL, Intl. T20 etc.

Try out GooglyPlusPlus 2020 Shiny app!!

You can clone/fork the code from Github GooglyPlusPlus2020

Important note: My earlier app GooglyPlusPlus handled all T20 formats including ODI (men and women). Due to an issue with Shiny, I could not include ODI matches in GooglyPlusPlus 2020

Here are some snapshots from GooglyPlusPlus 2020

A. Batting – Runs vs Deliveries (Shreyas Iyer)

 

 

B. Batting – Cumulative Batting Average (Shubman Gill)

 

C. Bowling – Mean Economy Rate (T. Natarajan)

 

D. Bowling -Bowler’s wickets against opposition (N A Saini)

E. Match scorecard – CSK vs DC 2020-10-17

The scorecards batting and bowling are computed on the fly for all T20 matches

 

F. Match – Batsmen vs Bowlers (DD vs KKR 2015-04-20)

 

G. Head-to-head: MI vs  KXIP all matches – Batting scorecard

H. Overall team performance- Team Bowler Wicket kind: Rajasthan Royals

Clone/fork the code from Github GooglyPlusPlus2020

Do take GooglyPlusPlus 2020 for a drive! While I have highlighted only IPL T20, because I have updated with the latest data, GooglyPlusPlus 2020 can also handle other T20 formats like BBL, Natwest, PSL, Intl. T20 (men &women) and WBB

 

Hope you have fun!

Also see

1.Big Data 7: yorkr waltzes with Apache NiFi

2. Deep Learning from first principles in Python, R and Octave – Part 6

3. Deconstructing Convolutional Neural Networks with Tensorflow and Keras

4. Sea shells on the seashore

5. Practical Machine Learning with R and Python – Part 3

6. Benford’s law meets IPL, Intl. T20 and ODI cricket

To see all posts click Index of posts

Rank IPL batsmen and bowlers post IPL 2020

Introduction

This post ranks IPL batsmen and bowlers post IPL 2020 season based on my R package yorkr. To know more about yorkr see Revisting R package yorkrAnalysis of IPL T20 matches with yorkr templates and others posts on this R package in Index of posts

library(yorkr)

1. Convert YAML files to match data

Convert all the match data as YAML file into .RData

#convertAllYaml2RDataframesT20("ipl","IPLMatches")

2. Rank the IPL Batsmen post IPL 2020

The function below ranks the IPL batsmen post IPL 2020. Note: We can specify the minimum number of matches that should have played by the batsmen for the ranking. By varying this parameter we can identify upcoming stars versus those who are more consistent.

dir="/Users/tvganesh/backup/software/cricket-package/yorkr-cricsheet/ipl2020/IPLMatches"
odir="/Users/tvganesh/backup/software/cricket-package/yorkr-cricsheet/ipl2020/IPLBattingBowlingDetails"


rankIPLBatsmen(dir=dir,odir=odir,minMatches=60)
## [1] "Chennai Super Kings"
## [1] "Delhi Capitals"
## [1] "Deccan Chargers"
## [1] "Delhi Daredevils"
## [1] "Kings XI Punjab"
## [1] "Kochi Tuskers Kerala"
## [1] "Kolkata Knight Riders"
## [1] "Mumbai Indians"
## [1] "Pune Warriors"
## [1] "Rajasthan Royals"
## [1] "Royal Challengers Bangalore"
## [1] "Sunrisers Hyderabad"
## [1] "Gujarat Lions"
## [1] "Rising Pune Supergiants"
## [1] "Chennai Super Kings-BattingDetails.RData"
## [1] "Delhi Capitals-BattingDetails.RData"
## [1] "Deccan Chargers-BattingDetails.RData"
## [1] "Delhi Daredevils-BattingDetails.RData"
## [1] "Kings XI Punjab-BattingDetails.RData"
## [1] "Kochi Tuskers Kerala-BattingDetails.RData"
## [1] "Kolkata Knight Riders-BattingDetails.RData"
## [1] "Mumbai Indians-BattingDetails.RData"
## [1] "Pune Warriors-BattingDetails.RData"
## [1] "Rajasthan Royals-BattingDetails.RData"
## [1] "Royal Challengers Bangalore-BattingDetails.RData"
## [1] "Sunrisers Hyderabad-BattingDetails.RData"
## [1] "Gujarat Lions-BattingDetails.RData"
## [1] "Rising Pune Supergiants-BattingDetails.RData"
## # A tibble: 65 x 4
##    batsman        matches meanRuns meanSR
##    <chr>            <int>    <dbl>  <dbl>
##  1 DA Warner          146     37.5   128.
##  2 CH Gayle           132     36.4   134.
##  3 SE Marsh            67     35.9   120.
##  4 KL Rahul            73     34.2   126.
##  5 RR Pant             68     31.8   133.
##  6 V Kohli            190     31.6   118.
##  7 AB de Villiers     155     30.5   136.
##  8 F du Plessis        79     30.4   118.
##  9 S Dhawan           174     30.0   115.
## 10 Q de Kock           64     29.8   119.
## # … with 55 more rows
rankIPLBatsmen(dir=dir,odir=odir,minMatches=70)
## [1] "Chennai Super Kings"
## [1] "Delhi Capitals"
## [1] "Deccan Chargers"
## [1] "Delhi Daredevils"
## [1] "Kings XI Punjab"
## [1] "Kochi Tuskers Kerala"
## [1] "Kolkata Knight Riders"
## [1] "Mumbai Indians"
## [1] "Pune Warriors"
## [1] "Rajasthan Royals"
## [1] "Royal Challengers Bangalore"
## [1] "Sunrisers Hyderabad"
## [1] "Gujarat Lions"
## [1] "Rising Pune Supergiants"
## [1] "Chennai Super Kings-BattingDetails.RData"
## [1] "Delhi Capitals-BattingDetails.RData"
## [1] "Deccan Chargers-BattingDetails.RData"
## [1] "Delhi Daredevils-BattingDetails.RData"
## [1] "Kings XI Punjab-BattingDetails.RData"
## [1] "Kochi Tuskers Kerala-BattingDetails.RData"
## [1] "Kolkata Knight Riders-BattingDetails.RData"
## [1] "Mumbai Indians-BattingDetails.RData"
## [1] "Pune Warriors-BattingDetails.RData"
## [1] "Rajasthan Royals-BattingDetails.RData"
## [1] "Royal Challengers Bangalore-BattingDetails.RData"
## [1] "Sunrisers Hyderabad-BattingDetails.RData"
## [1] "Gujarat Lions-BattingDetails.RData"
## [1] "Rising Pune Supergiants-BattingDetails.RData"
## # A tibble: 51 x 4
##    batsman        matches meanRuns meanSR
##    <chr>            <int>    <dbl>  <dbl>
##  1 DA Warner          146     37.5   128.
##  2 CH Gayle           132     36.4   134.
##  3 KL Rahul            73     34.2   126.
##  4 V Kohli            190     31.6   118.
##  5 AB de Villiers     155     30.5   136.
##  6 F du Plessis        79     30.4   118.
##  7 S Dhawan           174     30.0   115.
##  8 AM Rahane          124     29.6   105.
##  9 SS Iyer             77     29.3   111.
## 10 G Gambhir          155     29     110.
## # … with 41 more rows

3. Rank IPL bowlers post IPL 2020

The function ranks IPL bowlers post IPL 2020. We can specify the minimum number of matches that should have been played by the bowlers

dir="/Users/tvganesh/backup/software/cricket-package/yorkr-cricsheet/ipl2020/IPLMatches"
odir="/Users/tvganesh/backup/software/cricket-package/yorkr-cricsheet/ipl2020/IPLBattingBowlingDetails"
rankIPLBowlers(dir=dir,odir=odir,minMatches=60)
## [1] "Chennai Super Kings"
## [1] "Delhi Capitals"
## [1] "Deccan Chargers"
## [1] "Delhi Daredevils"
## [1] "Kings XI Punjab"
## [1] "Kochi Tuskers Kerala"
## [1] "Kolkata Knight Riders"
## [1] "Mumbai Indians"
## [1] "Pune Warriors"
## [1] "Rajasthan Royals"
## [1] "Royal Challengers Bangalore"
## [1] "Sunrisers Hyderabad"
## [1] "Gujarat Lions"
## [1] "Rising Pune Supergiants"
## [1] "Chennai Super Kings-BowlingDetails.RData"
## [1] "Delhi Capitals-BowlingDetails.RData"
## [1] "Deccan Chargers-BowlingDetails.RData"
## [1] "Delhi Daredevils-BowlingDetails.RData"
## [1] "Kings XI Punjab-BowlingDetails.RData"
## [1] "Kochi Tuskers Kerala-BowlingDetails.RData"
## [1] "Kolkata Knight Riders-BowlingDetails.RData"
## [1] "Mumbai Indians-BowlingDetails.RData"
## [1] "Pune Warriors-BowlingDetails.RData"
## [1] "Rajasthan Royals-BowlingDetails.RData"
## [1] "Royal Challengers Bangalore-BowlingDetails.RData"
## [1] "Sunrisers Hyderabad-BowlingDetails.RData"
## [1] "Gujarat Lions-BowlingDetails.RData"
## [1] "Rising Pune Supergiants-BowlingDetails.RData"
## # A tibble: 21 x 4
##    bowler          matches totalWickets meanER
##    <chr>             <int>        <dbl>  <dbl>
##  1 SL Malinga          120          184   6.99
##  2 SP Narine           117          143   6.82
##  3 Harbhajan Singh     131          134   7.11
##  4 DJ Bravo             91          125   8.20
##  5 YS Chahal            97          124   7.73
##  6 B Kumar              90          121   7.40
##  7 JJ Bumrah            91          119   7.35
##  8 R Ashwin             92           98   6.81
##  9 RA Jadeja           102           91   8.04
## 10 PP Chawla            85           87   8.02
## # … with 11 more rows
rankIPLBowlers(dir=dir,odir=odir,minMatches=50)
## [1] "Chennai Super Kings"
## [1] "Delhi Capitals"
## [1] "Deccan Chargers"
## [1] "Delhi Daredevils"
## [1] "Kings XI Punjab"
## [1] "Kochi Tuskers Kerala"
## [1] "Kolkata Knight Riders"
## [1] "Mumbai Indians"
## [1] "Pune Warriors"
## [1] "Rajasthan Royals"
## [1] "Royal Challengers Bangalore"
## [1] "Sunrisers Hyderabad"
## [1] "Gujarat Lions"
## [1] "Rising Pune Supergiants"
## [1] "Chennai Super Kings-BowlingDetails.RData"
## [1] "Delhi Capitals-BowlingDetails.RData"
## [1] "Deccan Chargers-BowlingDetails.RData"
## [1] "Delhi Daredevils-BowlingDetails.RData"
## [1] "Kings XI Punjab-BowlingDetails.RData"
## [1] "Kochi Tuskers Kerala-BowlingDetails.RData"
## [1] "Kolkata Knight Riders-BowlingDetails.RData"
## [1] "Mumbai Indians-BowlingDetails.RData"
## [1] "Pune Warriors-BowlingDetails.RData"
## [1] "Rajasthan Royals-BowlingDetails.RData"
## [1] "Royal Challengers Bangalore-BowlingDetails.RData"
## [1] "Sunrisers Hyderabad-BowlingDetails.RData"
## [1] "Gujarat Lions-BowlingDetails.RData"
## [1] "Rising Pune Supergiants-BowlingDetails.RData"
## # A tibble: 28 x 4
##    bowler          matches totalWickets meanER
##    <chr>             <int>        <dbl>  <dbl>
##  1 SL Malinga          120          184   6.99
##  2 SP Narine           117          143   6.82
##  3 Harbhajan Singh     131          134   7.11
##  4 DJ Bravo             91          125   8.20
##  5 YS Chahal            97          124   7.73
##  6 B Kumar              90          121   7.40
##  7 JJ Bumrah            91          119   7.35
##  8 R Ashwin             92           98   6.81
##  9 RA Jadeja           102           91   8.04
## 10 PP Chawla            85           87   8.02
## # … with 18 more rows
  1. Designing a Social Web Portal
  2. ntroducing QCSimulator: A 5-qubit quantum computing simulator in R
  3. Understanding Neural Style Transfer with Tensorflow and Keras
  4. Big Data-5: kNiFi-ing through cricket data with yorkpy
  5. Programming languages in layman’s language

To see all posts click Index of posts

Benford’s law meets IPL, Intl. T20 and ODI cricket

“To grasp how different a million is from a billion, think about it like this: A million seconds is a little under two weeks; a billion seconds is about thirty-two years.”

“One of the pleasures of looking at the world through mathematical eyes is that you can see certain patterns that would otherwise be hidden.”

               Steven Strogatz, Prof at Cornell University

Introduction

Within the last two weeks, I was introduced to Benford’s Law by 2 of my friends. Initially, I looked it up and Google and was quite intrigued by the law. Subsequently another friends asked me to check the ‘Digits’ episode, from the “Connected” series on Netflix by Latif Nasser, which I strongly recommend you watch.

Benford’s Law also called the Newcomb–Benford law, the law of anomalous numbers, or the First Digit Law states that, when dealing with quantities obtained from Nature, the frequency of appearance of each digit in the first significant place is logarithmic. For example, in sets that obey the law, the number 1 appears as the leading significant digit about 30.1% of the time, the number 2 about 17.6%, number 3 about 12.5% all the way to the number 9 at 4.6%. This interesting logarithmic pattern is observed in most natural datasets from population densities, river lengths, heights of skyscrapers, tax returns etc. What is really curious about this law, is that when we measure the lengths of rivers, the law holds perfectly regardless of the units used to measure. So the length of the rivers would obey the law whether we measure in meters, feet, miles etc. There is something almost mystical about this law.

The law has also been used widely to detect financial fraud, manipulations in tax statements, bots in twitter, fake accounts in social networks, image manipulation etc. In this age of deep fakes, the ability to detect fake images will assume paramount importance. While deviations from Benford Law do not always signify fraud, to large extent they point to an aberration. Prof Nigrini, of Cape Town used this law to identify financial discrepancies in Enron’s financial statement resulting in the infamous scandal. Also the 2009 Iranian election was found to be fradulent as the first digit percentages did not conform to those specified by Benford’s Law.

While it cannot be said with absolute certainty, marked deviations from Benford’s law could possibly indicate that there has been manipulation of natural processes. Possibly Benford’s law could be used to detect large scale match-fixing in cricket tournaments. However, we cannot look at this in isolation and the other statistical and forensic methods may be required to determine if there is fraud. Here is an interesting paper Promises and perils of Benford’s law

A set of numbers is said to satisfy Benford’s law if the leading digit d (d ∈ {1, …, 9}) occurs with probability

P(d)=log_{10}(1+1/d)

This law also works for number in other bases, in base b >=2

P(d)=log_{b}(1+1/d)

Interestingly, this law also applies to sports on the number of point scored in basketball etc. I was curious to see if this applied to cricket. Previously, using my R package yorkr, I had already converted all T20 data and ODI data from Cricsheet which is available at yorkrData2020, I wanted to check if Benford’s Law worked on the runs scored, or deliveries faced by batsmen at team level or at a tournament level (IPL, Intl. T20 or ODI).

Thankfully, R has a package benford.analysis to check for data behaviour in accordance to Benford’s Law, and I have used this package in my post

This post is also available in RPubs as Benford’s Law meets IPL, Intl. T20 and ODI

library(data.table)
library(reshape2)
library(dplyr)
library(benford.analysis)
library(yorkr)

In this post, I have randomly check data with Benford’s law. The fully converted dataset is available in yorkrData2020 which I have included above. You can try on any dataset including ODI (men,women),Intl T20(men,women),IPL,BBL,PSL,NTB and WBB.

1. Check the runs distribution by Royal Challengers Bangalore

We can see the behaviour is as expected with Benford’s law, with minor deviations

load("/Users/tvganesh/backup/software/cricket-package/yorkr-cricsheet/yorkrData2020/ipl/iplBattingBowlingDetails/Royal Challengers Bangalore-BattingDetails.RData")
rcbRunsTrends = benford(battingDetails$runs, number.of.digits = 1, discrete = T, sign = "positive") 
rcbRunsTrends
## 
## Benford object:
##  
## Data: battingDetails$runs 
## Number of observations used = 1205 
## Number of obs. for second order = 99 
## First digits analysed = 1
## 
## Mantissa: 
## 
##    Statistic  Value
##         Mean  0.458
##          Var  0.091
##  Ex.Kurtosis -1.213
##     Skewness -0.025
## 
## 
## The 5 largest deviations: 
## 
##   digits absolute.diff
## 1      1         14.26
## 2      7         13.88
## 3      9          8.14
## 4      6          5.33
## 5      4          4.78
## 
## Stats:
## 
##  Pearson's Chi-squared test
## 
## data:  battingDetails$runs
## X-squared = 5.2091, df = 8, p-value = 0.735
## 
## 
##  Mantissa Arc Test
## 
## data:  battingDetails$runs
## L2 = 0.0022852, df = 2, p-value = 0.06369
## 
## Mean Absolute Deviation (MAD): 0.004941381
## MAD Conformity - Nigrini (2012): Close conformity
## Distortion Factor: -18.8725
## 
## Remember: Real data will never conform perfectly to Benford's Law. You should not focus on p-values!

2. Check the ‘balls played’ distribution by Royal Challengers Bangalore

load("/Users/tvganesh/backup/software/cricket-package/yorkr-cricsheet/yorkrData2020/ipl/iplBattingBowlingDetails/Royal Challengers Bangalore-BattingDetails.RData")
rcbBallsPlayedTrends = benford(battingDetails$ballsPlayed, number.of.digits = 1, discrete = T, sign = "positive") 
plot(rcbBallsPlayedTrends)

 

3. Check the runs distribution by Chennai Super Kings

The trend seems to deviate from the expected behavior to some extent in the number of digits for 5 & 7.

load("/Users/tvganesh/backup/software/cricket-package/yorkr-cricsheet/yorkrData2020/ipl/iplBattingBowlingDetails/Chennai Super Kings-BattingDetails.RData")
cskRunsTrends = benford(battingDetails$runs, number.of.digits = 1, discrete = T, sign = "positive") 
cskRunsTrends
## 
## Benford object:
##  
## Data: battingDetails$runs 
## Number of observations used = 1054 
## Number of obs. for second order = 94 
## First digits analysed = 1
## 
## Mantissa: 
## 
##    Statistic  Value
##         Mean  0.466
##          Var  0.081
##  Ex.Kurtosis -1.100
##     Skewness -0.054
## 
## 
## The 5 largest deviations: 
## 
##   digits absolute.diff
## 1      5         27.54
## 2      2         18.40
## 3      1         17.29
## 4      9         14.23
## 5      7         14.12
## 
## Stats:
## 
##  Pearson's Chi-squared test
## 
## data:  battingDetails$runs
## X-squared = 22.862, df = 8, p-value = 0.003545
## 
## 
##  Mantissa Arc Test
## 
## data:  battingDetails$runs
## L2 = 0.002376, df = 2, p-value = 0.08173
## 
## Mean Absolute Deviation (MAD): 0.01309597
## MAD Conformity - Nigrini (2012): Marginally acceptable conformity
## Distortion Factor: -17.90664
## 
## Remember: Real data will never conform perfectly to Benford's Law. You should not focus on p-values!

4. Check runs distribution in all of Indian Premier League (IPL)

battingDF <- NULL
teams <-c("Chennai Super Kings","Deccan Chargers","Delhi Daredevils",
          "Kings XI Punjab", 'Kochi Tuskers Kerala',"Kolkata Knight Riders",
          "Mumbai Indians", "Pune Warriors","Rajasthan Royals",
          "Royal Challengers Bangalore","Sunrisers Hyderabad","Gujarat Lions",
          "Rising Pune Supergiants")


setwd("/Users/tvganesh/backup/software/cricket-package/yorkr-cricsheet/yorkrData2020/ipl/iplBattingBowlingDetails")
for(team in teams){
  battingDetails <- NULL
  val <- paste(team,"-BattingDetails.RData",sep="")
  print(val)
  tryCatch(load(val),
           error = function(e) {
             print("No data1")
             setNext=TRUE
           }
           
           
  )
  details <- battingDetails
  battingDF <- rbind(battingDF,details)
}
## [1] "Chennai Super Kings-BattingDetails.RData"
## [1] "Deccan Chargers-BattingDetails.RData"
## [1] "Delhi Daredevils-BattingDetails.RData"
## [1] "Kings XI Punjab-BattingDetails.RData"
## [1] "Kochi Tuskers Kerala-BattingDetails.RData"
## [1] "Kolkata Knight Riders-BattingDetails.RData"
## [1] "Mumbai Indians-BattingDetails.RData"
## [1] "Pune Warriors-BattingDetails.RData"
## [1] "Rajasthan Royals-BattingDetails.RData"
## [1] "Royal Challengers Bangalore-BattingDetails.RData"
## [1] "Sunrisers Hyderabad-BattingDetails.RData"
## [1] "Gujarat Lions-BattingDetails.RData"
## [1] "Rising Pune Supergiants-BattingDetails.RData"
trends = benford(battingDF$runs, number.of.digits = 1, discrete = T, sign = "positive") 
trends
## 
## Benford object:
##  
## Data: battingDF$runs 
## Number of observations used = 10129 
## Number of obs. for second order = 123 
## First digits analysed = 1
## 
## Mantissa: 
## 
##    Statistic   Value
##         Mean  0.4521
##          Var  0.0856
##  Ex.Kurtosis -1.1570
##     Skewness -0.0033
## 
## 
## The 5 largest deviations: 
## 
##   digits absolute.diff
## 1      2        159.37
## 2      9        121.48
## 3      7         93.40
## 4      8         83.12
## 5      1         61.87
## 
## Stats:
## 
##  Pearson's Chi-squared test
## 
## data:  battingDF$runs
## X-squared = 78.166, df = 8, p-value = 1.143e-13
## 
## 
##  Mantissa Arc Test
## 
## data:  battingDF$runs
## L2 = 5.8237e-05, df = 2, p-value = 0.5544
## 
## Mean Absolute Deviation (MAD): 0.006627966
## MAD Conformity - Nigrini (2012): Acceptable conformity
## Distortion Factor: -20.90333
## 
## Remember: Real data will never conform perfectly to Benford's Law. You should not focus on p-values!

5. Check Benford’s law in India matches

setwd("/Users/tvganesh/backup/software/cricket-package/yorkr-cricsheet/yorkrData2020/t20/t20BattingBowlingDetails")
load("India-BattingDetails.RData")

indiaTrends = benford(battingDetails$runs, number.of.digits = 1, discrete = T, sign = "positive") 
plot(indiaTrends)

 

6. Check Benford’s law in all of Intl. T20

setwd("/Users/tvganesh/backup/software/cricket-package/yorkr-cricsheet/yorkrData2020/t20/t20BattingBowlingDetails")
teams <-c("Australia","India","Pakistan","West Indies", 'Sri Lanka',
          "England", "Bangladesh","Netherlands","Scotland", "Afghanistan",
          "Zimbabwe","Ireland","New Zealand","South Africa","Canada",
          "Bermuda","Kenya","Hong Kong","Nepal","Oman","Papua New Guinea",
          "United Arab Emirates","Namibia","Cayman Islands","Singapore",
          "United States of America","Bhutan","Maldives","Botswana","Nigeria",
          "Denmark","Germany","Jersey","Norway","Qatar","Malaysia","Vanuatu",
          "Thailand")

for(team in teams){
  battingDetails <- NULL
  val <- paste(team,"-BattingDetails.RData",sep="")
  print(val)
  tryCatch(load(val),
           error = function(e) {
             print("No data1")
             setNext=TRUE
           }
           
           
  )
  details <- battingDetails
  battingDF <- rbind(battingDF,details)
  
}
intlT20Trends = benford(battingDF$runs, number.of.digits = 1, discrete = T, sign = "positive") 
intlT20Trends
## 
## Benford object:
##  
## Data: battingDF$runs 
## Number of observations used = 21833 
## Number of obs. for second order = 131 
## First digits analysed = 1
## 
## Mantissa: 
## 
##    Statistic  Value
##         Mean  0.447
##          Var  0.085
##  Ex.Kurtosis -1.158
##     Skewness  0.018
## 
## 
## The 5 largest deviations: 
## 
##   digits absolute.diff
## 1      2        361.40
## 2      9        276.02
## 3      1        264.61
## 4      7        210.14
## 5      8        198.81
## 
## Stats:
## 
##  Pearson's Chi-squared test
## 
## data:  battingDF$runs
## X-squared = 202.29, df = 8, p-value < 2.2e-16
## 
## 
##  Mantissa Arc Test
## 
## data:  battingDF$runs
## L2 = 5.3983e-06, df = 2, p-value = 0.8888
## 
## Mean Absolute Deviation (MAD): 0.007821098
## MAD Conformity - Nigrini (2012): Acceptable conformity
## Distortion Factor: -24.11086
## 
## Remember: Real data will never conform perfectly to Benford's Law. You should not focus on p-values!

Conclusion

Maths rules our lives, more than we are aware, more that we like to admit. It is there in all of nature. Whether it is the recursive patterns of Mandelbrot sets, the intrinsic notion of beauty through the golden ratio, the murmuration of swallows, the synchronous blinking of fireflies or in the almost univerality of Benford’s law on natural datasets, mathematics govern us.

Isn’t it strange that while we humans pride ourselves of freewill, the runs scored by batsmen in particular formats conform to Benford’s rule for the first digits. It almost looks like, the runs that will be scored is almost to extent predetermined to fall within specified ranges obeying Benford’s law. So much for choice.

Something to be pondered over!

Also see

  1. Introducing GooglyPlusPlus!!!
  2. Deconstructing Convolutional Neural Networks with Tensorflow and Keras
  3. Going deeper into IBM’s Quantum Experience!
  4. Experiments with deblurring using OpenCV
  5. Big Data 6: The T20 Dance of Apache NiFi and yorkpy
  6. Deep Learning from first principles in Python, R and Octave – Part 4
  7. Practical Machine Learning with R and Python – Part 4
  8. Re-introducing cricketr! : An R package to analyze performances of cricketers
  9. Bull in a china shop – Behind the scenes in Android

Big Data 7: yorkr waltzes with Apache NiFi

In this post, I construct an end-to-end Apache NiFi pipeline with my R package yorkr. This post is a mirror of my earlier post Big Data-5: kNiFing through cricket data with yorkpy based on my Python package yorkpy. The  Apache NiFi Data Pipeilne  flows all the way from the source, where the data is obtained, all the way  to target analytics output. Apache NiFi was created to automate the flow of data between systems.  NiFi dataflows enable the automated and managed flow of information between systems. This post automates the flow of data from Cricsheet, from where the zip file it is downloaded, unpacked, processed, transformed and finally T20 players are ranked.

This post uses the functions of my R package yorkr to rank IPL players. This is a example flow, of a typical Big Data pipeline where the data is ingested from many diverse source systems, transformed and then finally insights are generated. While I execute this NiFi example with my R package yorkr, in a typical Big Data pipeline where the data is huge, of the order of 100s of GB, we would be using the Hadoop ecosystem with Hive, HDFS Spark and so on. Since the data is taken from Cricsheet, which are few Megabytes, this approach would suffice. However if we hypothetically assume that there are several batches of cricket data that are being uploaded to the source, of different cricket matches happening all over the world, and the historical data exceeds several GBs, then we could use a similar Apache NiFi pattern to process the data and generate insights. If the data is was large and distributed across the Hadoop cluster , then we would need to use SparkR or SparklyR to process the data.

This is shown below pictorially

While this post displays the ranks of IPL batsmen, it is possible to create a cool dashboard using UI/UX technologies like AngularJS/ReactJS.  Take a look at my post Big Data 6: The T20 Dance of Apache NiFi and yorkpy where I create a simple dashboard of multiple analytics

My R package yorkr can handle both men’s and women’s ODI, and all formats of T20 in Cricsheet namely Intl. T20 (men’s, women’s), IPL, BBL, Natwest T20, PSL, Women’s BBL etc. To know more details about yorkr see Revitalizing R package yorkr

The code can be forked from Github at yorkrWithApacheNiFi

You can take a look at the live demo of the NiFi pipeline at yorkr waltzes with Apache NiFi

 

Basic Flow

1. Overall flow

The overall NiFi flow contains 2 Process Groups a) DownloadAnd Unpack. b) Convert and Rank IPL batsmen. While it appears that the Process Groups are disconnected, they are not. The first process group downloads the T20 zip file, unpacks the. zip file and saves the YAML files in a specific folder. The second process group monitors this folder and starts processing as soon the YAML files are available. It processes the YAML converting it into dataframes before storing it as CSV file. The next  processor then does the actual ranking of the batsmen before writing the output into IPLrank.txt

1.1 DownloadAndUnpack Process Group

This process group is shown below

 

1.1.1 GetT20Data

The GetT20Data Processor downloads the zip file given the URL

The ${T20data} variable points to the specific T20 format that needs to be downloaded. I have set this to https://cricsheet.org/downloads/ipl.zip. This could be set any other data set. In fact we could have parallel data flows for different T20/ Sports data sets and generate

1.1.2 SaveUnpackedData

This processor stores the YAML files in a predetermined folder, so that the data can be picked up  by the 2nd Process Group for processing

 

1.2 ProcessAndRankT20Players Process Group

This is the second process group which converts the YAML files to pandas dataframes before storing them as. CSV files. The RankIPLPlayers will then read all the CSV files, stack them and then proceed to rank the IPL players. The Process Group is shown below

 

1.2.1 ListFile and FetchFile Processors

The left 2 Processors ListFile and FetchFile get all the YAML files from the folder and pass it to the next processor

1.2.2 convertYaml2DataFrame Processor

The convertYaml2DataFrame Processor uses the ExecuteStreamCommand which call Rscript. The Rscript invoked the yorkr function convertYaml2DataframeT20() as shown below

 

I also use a 16 concurrent tasks to convert 16 different flowfiles at once

 

library(yorkr)
args<-commandArgs(TRUE)
convertYaml2RDataframeT20(args[1], args[2], args[3])

1.2.3 MergeContent Processor

This processor’s only job is to trigger the rankIPLPlayers when all the FlowFiles have merged into 1 file.

1.2.4 RankT20Players

This processor is an ExecuteStreamCommand Processor that executes a Rscript which invokes a yorrkr function rankIPLT20Batsmen()

library(yorkr)
args<-commandArgs(TRUE)

rankIPLBatsmen(args[1],args[2],args[3])

1.2.5 OutputRankofT20Player Processor

This processor writes the generated rank to an output file.

 

1.3 Final Ranking of IPL T20 players

The Nodejs based web server picks up this file and displays on the web page the final ranks (the code is based on a good youtube for reading from file)

[1] "Chennai Super Kings"
[1] "Deccan Chargers"
[1] "Delhi Daredevils"
[1] "Kings XI Punjab"
[1] "Kochi Tuskers Kerala"
[1] "Kolkata Knight Riders"
[1] "Mumbai Indians"
[1] "Pune Warriors"
[1] "Rajasthan Royals"
[1] "Royal Challengers Bangalore"
[1] "Sunrisers Hyderabad"
[1] "Gujarat Lions"
[1] "Rising Pune Supergiants"
[1] "Chennai Super Kings-BattingDetails.RData"
[1] "Deccan Chargers-BattingDetails.RData"
[1] "Delhi Daredevils-BattingDetails.RData"
[1] "Kings XI Punjab-BattingDetails.RData"
[1] "Kochi Tuskers Kerala-BattingDetails.RData"
[1] "Kolkata Knight Riders-BattingDetails.RData"
[1] "Mumbai Indians-BattingDetails.RData"
[1] "Pune Warriors-BattingDetails.RData"
[1] "Rajasthan Royals-BattingDetails.RData"
[1] "Royal Challengers Bangalore-BattingDetails.RData"
[1] "Sunrisers Hyderabad-BattingDetails.RData"
[1] "Gujarat Lions-BattingDetails.RData"
[1] "Rising Pune Supergiants-BattingDetails.RData"
# A tibble: 429 x 4
   batsman     matches meanRuns meanSR
   <chr>         <int>    <dbl>  <dbl>
 1 DA Warner       130     37.9   128.
 2 LMP Simmons      29     37.2   106.
 3 CH Gayle        125     36.2   134.
 4 HM Amla          16     36.1   108.
 5 ML Hayden        30     35.9   129.
 6 SE Marsh         67     35.9   120.
 7 RR Pant          39     35.3   135.
 8 MEK Hussey       59     33.8   105.
 9 KL Rahul         59     33.5   128.
10 MN van Wyk        5     33.4   112.
# … with 419 more rows

 

Conclusion

This post demonstrated an end-to-end pipeline with Apache NiFi and R package yorkr. You can this pipeline and generated different analytics using the various functions of yorkr and display them on a dashboard.

Hope you enjoyed with post!

 

See also
1. The mechanics of Convolutional Neural Networks in Tensorflow and Keras
2. Deep Learning from first principles in Python, R and Octave – Part 7
3. Fun simulation of a Chain in Android
4. Natural language processing: What would Shakespeare say?
5. TWS-4: Gossip protocol: Epidemics and rumors to the rescue
6. Cricketr learns new tricks : Performs fine-grained analysis of players
7. Introducing QCSimulator: A 5-qubit quantum computing simulator in R
8. Practical Machine Learning with R and Python – Part 5
9. Cricpy adds team analytics to its arsenal!!

To see posts click Index of posts