# Revitalizing R package yorkr

There is nothing so useless as doing efficiently that which should not be done at all. Peter Drucker

The most important thing in communication is to hear what isn’t being said. Peter Drucker

“Work expands to fill the time available for its completion.” Corollary: “Expenditure rises to meet income.” Parkinson’s law

## Introduction

“Operation successful!!!the Programmer Surgeon in me, thought to himself. What should have been a routine surgery, turned out to be a major operation in the end, which involved several grueling hours. The surgeon looked at the large chunks of programming logic in the operation tray, which had been surgically removed, as they had outlived their utility and had partly become dysfunctional. The surgeon glanced at the new, concise code logic which had replaced the earlier somewhat convoluted logic, with a smile of satisfaction,

To, those who tuned in late, I am referring to my R package yorkr which I had created in many years ago, in early 2016. The package had worked well for quite some time on data from Cricsheet. Cricsheet went into a hiatus in late 2017-2018, and came alive back in 2019. Unfortunately, a key function in the package, started to malfunction. The diagnosis was that the format of the YAML files had changed, in newer files, which resulted in the problem. I had got mails from users mentioning that yorkr was not converting the new YAML files. This was on my to do list for a long time, and a week or two back, I decided to “bite the bullet” and fix the issue. I hoped the fix would be trivial but it was anything but. Finally, I took the hard decision of re-designing the core of the yorkr package, which involved converting YAML files to RData (dataframes). Also, since it has been a while since I did R code, having done more of Python stuff in recent times, I had to jog my memory with my earlier 2 posts Essential R and R vs Python

I spent many hours, tweaking and fixing the new logic so that it worked on the older and new files. Finally, I am happy to say that the new code is much more compact and probably less error prone.

I also had to ensure that the converted files performed exactly on all the other yorkr functions. I ran all the my yorkr functions in my yorkr posts on ODI, Intl. T20 and IPL and made sure the results were identical. (Phew!!)

The changes will be available in CRAN in yorkr_0.0.8

Do take a look at my yorkr posts. All the functions work correctly. Do use help, as I have changed a few functions. I will have my posts reflect the correct usage, but some function or other may slip the cracks.

1. One Day Internationals ODI-Part1ODI-Part2ODI-Part3ODI-Part4
2. International T20s – T20-Part1,T20-Part2,T20-Part3,T20-Part4
3. Indian Premier League IPL-Part1IPL-Part2,IPL-Part3IPL-Part4

While making the changes, I also touched up some functions and made them more user friendly (added additional arguments etc). But by and large, yorkr is still yorkr and is intact.It just sports some spanking, new YAML conversion logic.

Note:

1. The code is available in Github yorkr
2. This RMarkdown has been published at RPubs Revitalizing yorkr
3. I have already converted the YAML files for ODI, Intl T20 and IPL. You can access and download the converted data from Github at yorkrData2020
setwd("/Users/tvganesh/backup/software/cricket-package/yorkr-cricsheet/yorkrgit")
install.packages("yorkr_0.0.8.tar.gz",repos = NULL, type="source")
library(yorkr)

Below I rank batsmen and bowlers in ODIs, T20 and IPL based on the data from Cricsheet.

## 1a. Rank ODI Batsmen

dir="/Users/tvganesh/backup/software/cricket-package/yorkr-cricsheet/yorkrData2020/odi/odiMenMatches"
odir="/Users/tvganesh/backup/software/cricket-package/yorkr-cricsheet/yorkrData2020/odi/odiBattingBowlingDetails"

rankODIBatsmen(dir=dir,odir=odir,minMatches=50)

## # A tibble: 151 x 4
##    batsman        matches meanRuns meanSR
##    <chr>            <int>    <dbl>  <dbl>
##  1 Babar Azam          52     50.2   87.2
##  2 SD Hope             51     48.7   71.0
##  3 V Kohli            207     48.4   79.4
##  4 HM Amla            159     46.6   82.4
##  5 DA Warner          114     46.1   88.0
##  6 AB de Villiers     190     45.5   94.5
##  7 JE Root            108     44.9   82.5
##  8 SR Tendulkar        96     43.9   77.1
##  9 IJL Trott           63     43.1   68.9
## 10 Q de Kock          106     42.0   82.7
## # … with 141 more rows

## 1b. Rank ODI Bowlers

dir="/Users/tvganesh/backup/software/cricket-package/yorkr-cricsheet/yorkrData2020/odi/odiMenMatches"
odir="/Users/tvganesh/backup/software/cricket-package/yorkr-cricsheet/yorkrData2020/odi/odiBattingBowlingDetails"

rankODIBowlers(dir=dir,odir=odir,minMatches=30)
## # A tibble: 265 x 4
##    bowler           matches totalWickets meanER
##    <chr>              <int>        <dbl>  <dbl>
##  1 SL Malinga           191          308   5.25
##  2 MG Johnson           142          238   4.73
##  3 Shakib Al Hasan      157          214   4.72
##  4 Shahid Afridi        166          213   4.69
##  5 JM Anderson          143          207   4.96
##  6 KMDN Kulasekara      161          190   4.94
##  7 SCJ Broad            115          189   5.31
##  8 DW Steyn             114          188   4.96
##  9 Mashrafe Mortaza     139          180   4.97
## 10 Saeed Ajmal          106          180   4.17
## # … with 255 more rows

## 2a. Rank T20 Batsmen

dir="/Users/tvganesh/backup/software/cricket-package/yorkr-cricsheet/yorkrData2020/t20/t20MenMatches"
odir="/Users/tvganesh/backup/software/cricket-package/yorkr-cricsheet/yorkrData2020/t20/t20BattingBowlingDetails"

rankT20Batsmen(dir=dir,odir=odir,minMatches=50)
## # A tibble: 43 x 4
##    batsman          matches meanRuns meanSR
##    <chr>              <int>    <dbl>  <dbl>
##  1 V Kohli               61     39.0   132.
##  3 CH Gayle              50     31.1   124.
##  4 BB McCullum           69     30.7   126.
##  5 PR Stirling           66     29.6   116.
##  6 MJ Guptill            70     29.6   125.
##  7 DA Warner             75     29.1   128.
##  8 AD Hales              50     28.1   120.
##  9 TM Dilshan            78     26.7   105.
## 10 RG Sharma             72     26.4   120.
## # … with 33 more rows

## 2b. Rank T20 Bowlers

dir="/Users/tvganesh/backup/software/cricket-package/yorkr-cricsheet/yorkrData2020/t20/t20MenMatches"
odir="/Users/tvganesh/backup/software/cricket-package/yorkr-cricsheet/yorkrData2020/t20/t20BattingBowlingDetails"

rankT20Bowlers(dir=dir,odir=odir,,minMatches=30)

## # A tibble: 153 x 4
##    bowler          matches totalWickets meanER
##    <chr>             <int>        <dbl>  <dbl>
##  1 SL Malinga           78          115   7.39
##  2 Shahid Afridi        89           98   6.80
##  3 Saeed Ajmal          62           92   6.30
##  4 Umar Gul             56           87   7.40
##  5 KMDN Kulasekara      56           72   7.25
##  6 TG Southee           55           69   8.68
##  7 DJ Bravo             60           69   8.41
##  8 DW Steyn             47           69   7.00
##  9 Shakib Al Hasan      57           69   6.82
## 10 SCJ Broad            55           68   7.83
## # … with 143 more rows

## 3a. Rank IPL Batsmen

dir="/Users/tvganesh/backup/software/cricket-package/yorkr-cricsheet/yorkrData2020/ipl/iplMatches"
odir="/Users/tvganesh/backup/software/cricket-package/yorkr-cricsheet/yorkrData2020/ipl/iplBattingBowlingDetails"

rankIPLBatsmen(dir=dir,odir=odir,,minMatches=50)
## # A tibble: 69 x 4
##    batsman        matches meanRuns meanSR
##    <chr>            <int>    <dbl>  <dbl>
##  1 DA Warner          130     37.9   128.
##  2 CH Gayle           125     36.2   134.
##  3 SE Marsh            67     35.9   120.
##  4 MEK Hussey          59     33.8   105.
##  5 KL Rahul            59     33.5   128.
##  6 V Kohli            175     31.6   119.
##  7 AM Rahane          116     30.7   108.
##  8 AB de Villiers     141     30.3   135.
##  9 F du Plessis        65     29.4   117.
## 10 S Dhawan           140     29.0   114.
## # … with 59 more rows

## 3a. Rank IPL Bowlers

dir="/Users/tvganesh/backup/software/cricket-package/yorkr-cricsheet/yorkrData2020/ipl/iplMatches"
odir="/Users/tvganesh/backup/software/cricket-package/yorkr-cricsheet/yorkrData2020/ipl/iplBattingBowlingDetails"

rankIPLBowlers(dir=dir,odir=odir,,minMatches=30)
## # A tibble: 143 x 4
##    bowler          matches totalWickets meanER
##    <chr>             <int>        <dbl>  <dbl>
##  1 SL Malinga          120          184   6.99
##  2 SP Narine           108          137   6.71
##  3 Harbhajan Singh     131          134   7.11
##  4 DJ Bravo             85          118   8.18
##  5 B Kumar              86          116   7.43
##  6 YS Chahal            82          102   7.85
##  7 R Ashwin             92           98   6.81
##  8 JJ Bumrah            76           91   7.47
##  9 PP Chawla            85           87   8.02
## 10 RA Jadeja            89           85   7.93
## # … with 133 more rows

##Conclusion

Go ahead and give yorkr a spin once yorkr_0.0.8 is available in CRAN. I hope you have fun. Do get back to me if you have any issues.

I’ll be back. Watch this space!!

You may also like

To see all posts click Index of posts

# Deconstructing Convolutional Neural Networks with Tensorflow and Keras

I have been very fascinated by how Convolution Neural  Networks have been able to, so efficiently,  do image classification and image recognition CNN’s have been very successful in in both these tasks. A good paper that explores the workings of a CNN Visualizing and Understanding Convolutional Networks  by Matthew D Zeiler and Rob Fergus. They show how through a reverse process of convolution using a deconvnet.

In their paper they show how by passing the feature map through a deconvnet ,which does the reverse process of the convnet, they can reconstruct what input pattern originally caused a given activation in the feature map

In the paper they say “A deconvnet can be thought of as a convnet model that uses the same components (filtering, pooling) but in reverse, so instead of mapping pixels to features, it does the opposite. An input image is presented to the CNN and features  activation computed throughout the layers. To examine a given convnet activation, we set all other activations in the layer to zero and pass the feature maps as input to the attached deconvnet layer. Then we successively (i) unpool, (ii) rectify and (iii) filter to reconstruct the activity in the layer beneath that gave rise to the chosen activation. This is then repeated until input pixel space is reached.”

I started to scout the internet to see how I can implement this reverse process of Convolution to understand what really happens under the hood of a CNN.  There are a lot of good articles and blogs, but I found this post Applied Deep Learning – Part 4: Convolutional Neural Networks take the visualization of the CNN one step further.

This post takes VGG16 as the pre-trained network and then uses this network to display the intermediate visualizations.  While this post was very informative and also the visualizations of the various images were very clear, I wanted to simplify the problem for my own understanding.

Hence I decided to take the MNIST digit classification as my base problem. I created a simple 3 layer CNN which gives close to 99.1% accuracy and decided to see if I could do the visualization.

As mentioned in the above post, there are 3 major visualisations

1. Feature activations at the layer
2. Visualisation of the filters
3. Visualisation of the class outputs

Feature Activation – This visualization the feature activation at the 3 different layers for a given input image. It can be seen that first layer  activates based on the edge of the image. Deeper layers activate in a more abstract way.

Visualization of the filters: This visualization shows what patterns the filters respond maximally to. This is implemented in Keras here

To do this the following is repeated in a loop

• Choose a loss function that maximizes the value of a convnet filter activation
• Do gradient ascent (maximization) in input space that increases the filter activation

Visualizing Class Outputs of the MNIST Convnet: This process is similar to determining the filter activation. Here the convnet is made to generate an image that represents the category maximally.

You can access the Google colab notebook here – Deconstructing Convolutional Neural Networks in Tensoflow and Keras

import numpy as np
import pandas as pd
import os
import tensorflow as tf
import matplotlib.pyplot as plt
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D, Input
from keras.models import Model
from sklearn.model_selection import train_test_split
from keras.utils import np_utils
Using TensorFlow backend.
In [0]:
mnist=tf.keras.datasets.mnist
# Set training and test data and labels
In [0]:
#Normalize training data
X =np.array(training_images).reshape(training_images.shape[0],28,28,1)
# Normalize the images by dividing by 255.0
X = X/255.0
X.shape
# Use one hot encoding for the labels
Y = np_utils.to_categorical(training_labels, 10)
Y.shape
# Split training data into training and validation data in the ratio of 80:20
X_train, X_validation, y_train, y_validation = train_test_split(X,Y,test_size=0.20, random_state=42)
In [4]:
# Normalize test data
X_test =np.array(test_images).reshape(test_images.shape[0],28,28,1)
X_test=X_test/255.0
#Use OHE for the test labels
Y_test = np_utils.to_categorical(test_labels, 10)
X_test.shape
Out[4]:
(10000, 28, 28, 1)

# Display data

Display the training data and the corresponding labels

In [5]:
print(training_labels[0:10])
f, axes = plt.subplots(1, 10, sharey=True,figsize=(10,10))
for i,ax in enumerate(axes.flat):
ax.axis('off')
ax.imshow(X[i,:,:,0],cmap="gray")

# Create a Convolutional Neural Network

The CNN consists of 3 layers

• Conv2D of size 28 x 28 with 24 filters
• Perform Max pooling
• Conv2D of size 14 x 14 with 48 filters
• Perform max pooling
• Conv2d of size 7 x 7 with 64 filters
• Flatten
• Use Dense layer with 128 units
• Perform 25% dropout
• Perform categorical cross entropy with softmax activation function
In [0]:
num_classes=10
inputs = Input(shape=(28,28,1))
x = MaxPooling2D(pool_size=(2, 2))(x)
x = Conv2D(48, (3, 3), padding='same',activation='relu')(x)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = Conv2D(64, (3, 3), padding='same',activation='relu')(x)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = Flatten()(x)
x = Dense(128, activation='relu')(x)
x = Dropout(0.25)(x)
output = Dense(num_classes,activation="softmax")(x)

model = Model(inputs,output)

model.compile(loss='categorical_crossentropy',
metrics=['accuracy'])

# Summary of CNN

Display the summary of CNN

In [7]:
model.summary()
Model: "model_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_1 (InputLayer)         (None, 28, 28, 1)         0
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 28, 28, 24)        240
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 14, 14, 24)        0
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 14, 14, 48)        10416
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 7, 7, 48)          0
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 7, 7, 64)          27712
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 3, 3, 64)          0
_________________________________________________________________
flatten_1 (Flatten)          (None, 576)               0
_________________________________________________________________
dense_1 (Dense)              (None, 128)               73856
_________________________________________________________________
dropout_1 (Dropout)          (None, 128)               0
_________________________________________________________________
dense_2 (Dense)              (None, 10)                1290
=================================================================
Total params: 113,514
Trainable params: 113,514
Non-trainable params: 0
_________________________________________________________________

# Perform Gradient descent and validate with the validation data

In [8]:
epochs = 20
batch_size=256
history = model.fit(X_train,y_train,
epochs=epochs,
batch_size=batch_size,
validation_data=(X_validation,y_validation))
————————————————
acc = history.history[ ‘accuracy’ ]
val_acc = history.history[ ‘val_accuracy’ ]
loss = history.history[ ‘loss’ ]
val_loss = history.history[‘val_loss’ ]
epochs = range(len(acc)) # Get number of epochs
#————————————————
# Plot training and validation accuracy per epoch
#————————————————
plt.plot ( epochs, acc,label=”training accuracy” )
plt.plot ( epochs, val_acc, label=’validation acuracy’ )
plt.title (‘Training and validation accuracy’)
plt.legend()
plt.figure()
#————————————————
# Plot training and validation loss per epoch
#————————————————
plt.plot ( epochs, loss , label=”training loss”)
plt.plot ( epochs, val_loss,label=”validation loss” )
plt.title (‘Training and validation loss’ )
plt.legend()
Test model on test data
f, axes = plt.subplots(1, 10, sharey=True,figsize=(10,10))
for i,ax in enumerate(axes.flat):
ax.axis(‘off’)
ax.imshow(X_test[i,:,:,0],cmap=”gray”)
l=[]
for i in range(10):
x=X_test[i].reshape(1,28,28,1)
y=model.predict(x)
m = np.argmax(y, axis=1)
print(m)
[7]
[2]
[1]
[0]
[4]
[1]
[4]
[9]
[5]
[9]

# Generate the filter activations at the intermediate CNN layers

In [12]:
img = test_images[51].reshape(1,28,28,1)
fig = plt.figure(figsize=(5,5))
print(img.shape)
plt.imshow(img[0,:,:,0],cmap="gray")
plt.axis('off')

# Display the activations at the intermediate layers

This displays the intermediate activations as the image passes through the filters and generates these feature maps

In [13]:
layer_names = ['conv2d_4', 'conv2d_5', 'conv2d_6']

layer_outputs = [layer.output for layer in model.layers if 'conv2d' in layer.name]
activation_model = Model(inputs=model.input,outputs=layer_outputs)
intermediate_activations = activation_model.predict(img)
images_per_row = 8
max_images = 8

for layer_name, layer_activation in zip(layer_names, intermediate_activations):
print(layer_name,layer_activation.shape)
n_features = layer_activation.shape[-1]
print("features=",n_features)
n_features = min(n_features, max_images)
print(n_features)

size = layer_activation.shape[1]
print("size=",size)
n_cols = n_features // images_per_row
display_grid = np.zeros((size * n_cols, images_per_row * size))

for col in range(n_cols):
for row in range(images_per_row):
channel_image = layer_activation[0,:, :, col * images_per_row + row]

channel_image -= channel_image.mean()
channel_image /= channel_image.std()
channel_image *= 64
channel_image += 128
channel_image = np.clip(channel_image, 0, 255).astype('uint8')
display_grid[col * size : (col + 1) * size,
row * size : (row + 1) * size] = channel_image
scale = 2. / size
plt.figure(figsize=(scale * display_grid.shape[1],
scale * display_grid.shape[0]))
plt.axis('off')
plt.title(layer_name)
plt.grid(False)
plt.imshow(display_grid, aspect='auto', cmap='viridis')

plt.show()

It can be seen that at the higher layers only abstract features of the input image are captured
# To fix the ImportError: cannot import name 'imresize' in the next cell. Run this cell. Then comment and restart and run all
#!pip install scipy==1.1.0

## Visualize the pattern that the filters respond to maximally

• Choose a loss function that maximizes the value of the CNN filter in a given layer
• Start from a blank input image.
• Do gradient ascent in input space. Modify input values so that the filter activates more
• Repeat this in a loop.
In [14]:
from vis.visualization import visualize_activation, get_num_filters
from vis.utils import utils
from vis.input_modifiers import Jitter

max_filters = 24
selected_indices = []
vis_images = [[], [], [], [], []]
i = 0
selected_filters = [[0, 3, 11, 15, 16, 17, 18, 22],
[8, 21, 23, 25, 31, 32, 35, 41],
[2, 7, 11, 14, 19, 26, 35, 48]]

# Set the layers
layer_name = ['conv2d_4', 'conv2d_5', 'conv2d_6']
# Set the layer indices
layer_idx = [1,3,5]
for layer_name,layer_idx in zip(layer_name,layer_idx):

# Visualize all filters in this layer.
if selected_filters:
filters = selected_filters[i]
else:
# Randomly select filters
filters = sorted(np.random.permutation(get_num_filters(model.layers[layer_idx]))[:max_filters])
selected_indices.append(filters)

# Generate input image for each filter.
# Loop through the selected filters in each layer and generate the activation of these filters
for idx in filters:
img = visualize_activation(model, layer_idx, filter_indices=idx, tv_weight=0.,
input_modifiers=[Jitter(0.05)], max_iter=300)
vis_images[i].append(img)

# Generate stitched image palette with 4 cols so we get 2 rows.
stitched = utils.stitch_images(vis_images[i], cols=4)
plt.figure(figsize=(20, 30))
plt.title(layer_name)
plt.axis('off')
stitched = stitched.reshape(1,61,127,1)
plt.imshow(stitched[0,:,:,0])
plt.show()
i += 1
from vis.utils import utils
new_vis_images = [[], [], [], [], []]
i = 0
layer_name = ['conv2d_4', 'conv2d_5', 'conv2d_6']
layer_idx = [1,3,5]
for layer_name,layer_idx in zip(layer_name,layer_idx):

# Generate input image for each filter.
for j, idx in enumerate(selected_indices[i]):
img = visualize_activation(model, layer_idx, filter_indices=idx,
seed_input=vis_images[i][j], input_modifiers=[Jitter(0.05)], max_iter=300)
#img = utils.draw_text(img, 'Filter {}'.format(idx))
new_vis_images[i].append(img)

stitched = utils.stitch_images(new_vis_images[i], cols=4)
plt.figure(figsize=(20, 30))
plt.title(layer_name)
plt.axis('off')
print(stitched.shape)
stitched = stitched.reshape(1,61,127,1)
plt.imshow(stitched[0,:,:,0])
plt.show()
i += 1

## Visualizing Class Outputs

Here the CNN will generate the image that maximally represents the category. Each of the output represents one of the digits as can be seen below

In [16]:
from vis.utils import utils
from keras import activations
codes = '''
zero 0
one 1
two 2
three 3
four 4
five 5
six 6
seven 7
eight 8
nine 9
'''
layer_idx=10
initial = []
images = []
tuples = []
# Swap softmax with linear for better visualization
model.layers[layer_idx].activation = activations.linear
model = utils.apply_modifications(model)
for line in codes.split('\n'):
if not line:
continue
name, idx = line.rsplit(' ', 1)
idx = int(idx)
img = visualize_activation(model, layer_idx, filter_indices=idx,
tv_weight=0., max_iter=300, input_modifiers=[Jitter(0.05)])

initial.append(img)
tuples.append((name, idx))

i = 0
for name, idx in tuples:
img = visualize_activation(model, layer_idx, filter_indices=idx,
seed_input = initial[i], max_iter=300, input_modifiers=[Jitter(0.05)])
#img = utils.draw_text(img, name) # Unable to display text on gray scale image
i += 1
images.append(img)

stitched = utils.stitch_images(images, cols=4)
plt.figure(figsize=(20, 20))
plt.axis('off')
stitched = stitched.reshape(1,94,127,1)
plt.imshow(stitched[0,:,:,0])

plt.show()

In the grid below the class outputs show the MNIST digits to which output responds to maximally. We can see the ghostly outline
of digits 0 – 9. We can clearly see the outline if 0,1, 2,3,4,5 (yes, it is there!),6,7, 8 and 9. If you look at this from a little distance the digits are clearly visible. Isn’t that really cool!!

## Conclusion:

It is really interesting to see the class outputs which show the image to which the class output responds to maximally. In the
post Applied Deep Learning – Part 4: Convolutional Neural Networks the class output show much more complicated images and is worth a look. It is really interesting to note that the model has adjusted the filter values and the weights of the fully connected network to maximally respond to the MNIST digits

## Also see

To see all posts click Index of posts

# Understanding Neural Style Transfer with Tensorflow and Keras

Neural Style Transfer (NST)  is a fascinating area of Deep Learning and Convolutional Neural Networks. NST is an interesting technique, in which the style from an image, known as the ‘style image’ is transferred to another image ‘content image’ and we get a third a image which is a generated image which has the content of the original image and the style of another image.

NST can be used to reimagine how famous painters like Van Gogh, Claude Monet or a Picasso would have visualised a scenery or architecture. NST uses Convolutional Neural Networks (CNNs) to achieve this artistic style transfer from one image to another. NST was originally implemented by Gati et al., in their paper Neural Algorithm of Artistic Style. Convolutional Neural Networks have been very successful in image classification image recognition et cetera. CNN networks have also been have also generated very interesting pictures using Neural Style Transfer which will be shown in this post. An interesting aspect of CNN’s is that the first couple of layers in the CNN capture basic features of the image like edges and  pixel values. But as we go deeper into the CNN, the network captures higher level features of the input image.

To get started with Neural Style transfer  we will be using the VGG19 pre-trained network. The VGG19 CNN is a compact pre-trained your network which can be used for performing the NST. However, we could have also used Resnet or InceptionV3 networks for this purpose but these are very large networks. The idea of using a network trained on a different task and applying it to a new task is called transfer learning.

What needs to be done to transfer the style from one of the image to another image. This brings us to the question – What is ‘style’? What is it that distinguishes Van Gogh’s painting or Picasso’s cubist art. Convolutional Neural Networks capture basic features in the lower layers and much more complex features in the deeper layers.  Style can be computed by taking the correlation of the feature maps in a layer L. This is my interpretation of how style is captured.  Since style  is intrinsic to  the image, it  implies that the style feature would exist across all the filters in a layer. Hence, to pick up this style we would need to get the correlation of the filters across channels of a lawyer. This is computed mathematically, using the Gram matrix which calculates the correlation of the activation of a the filter by the style image and generated image

To transfer the style from one image to the content image we need to do two parallel operations while doing forward propagation
– Compute the content loss between the source image and the generated image
– Compute the style loss between the style image and the generated image
– Finally we need to compute the total loss

In order to get transfer the style from the ‘style’ image to the ‘content ‘image resulting in a  ‘generated’  image  the total loss has to be minimised. Therefore backward propagation with gradient descent  is done to minimise the total loss comprising of the content and style loss.

Initially we make the Generated Image ‘G’ the same as the source image ‘S’

The content loss at layer ‘l’

$L_{content} = 1/2 \sum_{i}^{j} ( F^{l}_{i,j} - P^{l}_{i,j})^{2}$

where $F^{l}_{i,j}$ and $P^{l}_{i,j}$ represent the activations at layer ‘l’ in a filter i, at position ‘j’. The intuition is that the activations will be same for similar source and generated image. We need to minimise the content loss so that the generated stylized image is as close to the original image as possible. An intermediate layer of VGG19 block5_conv2 is used

The Style layers that are are used are

style_layers = [‘block1_conv1’,
‘block2_conv1’,
‘block3_conv1’,
‘block4_conv1’,
‘block5_conv1’]
To compute the Style Loss the Gram matrix needs to be computed. The Gram Matrix is computed by unrolling the filters as shown below (source: Convolutional Neural Networks by Prof Andrew Ng, Coursera). The result is a matrix of size $n_{c}$ x $n_{c}$ where $n_{c}$ is the number of channels
The above diagram shows the filters of height $n_{H}$ and width $n_{W}$ with $n_{C}$ channels
The contribution of layer ‘l’ to style loss is given by
$L^{'}_{style} = \frac{\sum_{i}^{j} (G^{2}_{i,j} - A^l{i,j})^2}{4N^{2}_{l}M^{2}_{l}}$
where $G_{i,j}$  and $A_{i,j}$ are the Gram matrices of the style and generated images respectively. By minimising the distance in the gram matrices of the style and generated image we can ensure that generated image is a stylized version of the original image similar to the style image
The total loss is given by
$L_{total} = \alpha L_{content} + \beta L_{style}$
Back propagation with gradient descent works to minimise the content loss between the source and generated image, while the style loss tries to minimise the discrepancies in the style of the style image and generated image. Running through forward and backpropagation through several epochs successfully transfers the style from the style image to the source image.
You can check the Notebook at Neural Style Transfer

Note: The code in this notebook is largely based on the Neural Style Transfer tutorial from Tensorflow, though I may have taken some changes from other blogs. I also made a few changes to the code in this tutorial, like removing the scaling factor, or the class definition (Personally, I belong to the old school (C language) and am not much in love with the ‘self.”..All references are included below

Note: Here is a interesting thought. Could we do a Neural Style Transfer in music? Imagine Carlos Santana playing ‘Hotel California’ or Brian May style in ‘Another brick in the wall’. While our first reaction would be that it may not sound good as we are used to style of these songs, we may be surprised by a possible style transfer. This is definitely music to the ears!

Here are few runs from this

## A) Run 1

1. Neural Style Transfer – a) Content Image – My portrait.  b) Style Image – Wassily Kadinsky Oil on canvas, 1913, Vassily Kadinsky’s composition

2. Result of Neural Style Transfer

2) Run 2

a) Content Image – Portrait of my parents b) Style Image –  Vincent Van Gogh’s ,Starry Night Oil on canvas 1889

2. Result of Neural Style Transfer

## Run 3

1.  Content Image – Caesar 2 (Masai Mara- 20 Jun 2018).  Style Image – The Great Wave at Kanagawa – Katsushika Hokosai, 1826-1833

2. Result of Neural Style Transfer

## Run 4

1.   Content Image – Junagarh Fort , Rajasthan   Sep 2016              b) Style Image – Le Pont Japonais by Claude Monet, Oil on canvas, 1920

2. Result of Neural Style Transfer

Neural Style Transfer is a very ingenious idea which shows that we can segregate the style of a painting and transfer to another image.

### References

1. A Neural Algorithm of Artistic Style, Leon A. Gatys, Alexander S. Ecker, Matthias Bethge
2. Neural style transfer
3. Neural Style Transfer: Creating Art with Deep Learning using tf.keras and eager execution
4. Convolutional Neural Network, DeepLearning.AI Specialization, Prof Andrew Ng
5. Intuitive Guide to Neural Style Transfer

To see all posts click Index of posts

# The mechanics of Convolutional Neural Networks in Tensorflow and Keras

Convolutional Neural Networks (CNNs), have been very popular in the last decade or so. CNNs have been used in multiple applications like image recognition, image classification, facial recognition, neural style transfer etc. CNN’s have been extremely successful in handling these kind of problems. How do they work? What makes them so successful? What is the principle behind CNN’s ?

Note: this post is based on two Coursera courses I did, namely namely Deep Learning specialisation by Prof Andrew Ng and Tensorflow Specialisation by  Laurence Moroney.

In this post I show you how CNN’s work. To understand how CNNs work, we need to understand the concept behind machine learning algorithms. If you take a simple machine learning algorithm in which you are trying to do multi-class classification using softmax or binary classification with the sigmoid function, for a set of for a set of input features against a target variable we need to create an objective function of the input features versus the target variable. Then we need to minimise this objective function, while performing gradient descent, such that the cost  is the lowest. This will give the set of weights for the different variables in the objective function.

The central problem in ML algorithms is to do feature selection, i.e.  we need to find the set of features that actually influence the target.  There are various methods for doing features selection – best fit, forward fit, backward fit, ridge and lasso regression. All these methods try to pick out the predictors that influence the output most, by making the weights of the other features close to zero. Please look at my post – Practical Machine Learning in R and Python – Part 3, where I show you the different methods for doing features selection.

In image classification or Image recognition we need to find the important features in the image. How do we do that? Many years back, have played around with OpenCV.  While working with OpenCV I came across are numerous filters like the Sobel ,the Laplacian, Canny, Gaussian filter et cetera which can be used to identify key features of the image. For example the Canny filter feature can be used for edge detection, Gaussian for smoothing, Sobel for determining the derivative and we have other filters for detecting vertical or horizontal edges. Take a look at my post Computer Vision: Ramblings on derivatives, histograms and contours So for handling images we need to apply these filters to pick  out the key features of the image namely the edges and other features. So rather than using the entire image’s pixels against the target class we can pick out the features from the image and use that as predictors of the target output.

Note: that in Convolutional Neural Network, fixed filter values like the those shown above  are not used directly. Rather the filter values are learned through back propagation and gradient descent as shown below.

In CNNs the filter values are considered to be weights which are then learned and updated in each forward/backward propagation cycle much like the way a fully connected Deep Learning Network learns the weights of the network.

Here is a short derivation of the most important parts of how a CNNs work

The convolution of a filter F with the input X can be represented as.

Convolving we get

This the forward propagation as it passes through a non-linear function like Relu

To go through back propagation we need to compute the $\partial L$  at every node of Convolutional Neural network

The loss with respect to the output is $\partial L/\partial O$. $\partial O/\partial X$ & $\partial O/\partial F$ are the local derivatives

We need these local derivatives because we can learn the filter values using gradient descent

where $\alpha$ is the learning rate. Also $\partial L/\partial X$ is the loss which is back propagated to the previous layers. You can see the detailed derivation of back propagation in my post Deep Learning from first principles in Python, R and Octave – Part 3 in a L-layer, multi-unit Deep Learning network.

In the fully connected layers the weights associated with each connection is computed in every cycle of forward and backward propagation using gradient descent. Similarly, the filter values are also computed and updated in each forward and backward propagation cycle. This is done so as to minimize the loss at the output layer.

By using the chain rule and simplifying the back propagation for the Convolutional layers we get these 2 equations. The first equation is used to learn the filter values and the second is used pass the loss to layers before

(for the detailed derivation see Convolutions and Backpropagations

An important aspect of performing convolutions is to reduce the size of  the flattened image that is passed into the fully connected DL network. Successively convolving with 2D filters and doing a max pooling helps to reduce the size of the features that we can use for learning the images. Convolutions also enable a sparsity of connections  as you can see in the diagram below. In the LeNet-5 Convolution Neural Network of Yann Le Cunn, successive convolutions reduce the image size from 28 x 28=784 to 120 flattened values.

Here is an interesting Deep Learning problem. Convolutions help in picking out important features of images and help in image classification/ detection. What would be its equivalent if we wanted to identify the Carnatic ragam of a song? A Carnatic ragam is roughly similar to Western scales (major, natural, melodic, blues) with all its modes Lydian, Aeolion, Phyrgian etc. Except in the case of the ragams, it is more nuanced, complex and involved. Personally, I can rarely identify a ragam on which a carnatic song is based (I am tone deaf when it comes to identifying ragams). I have come to understand that each Carnatic ragam has its own character, which is made up of several melodic phrases which are unique to that flavor of a ragam. What operation like convolution would be needed so that we can pick out these unique phrases in a Carnatic ragam? Of course, we would need to use it in Recurrent Neural Networks with LSTMs as a song is a time sequence of notes to identify sequences. Nevertheless, if there was some operation with which we can pick up the distinct, unique phrases from a song and then run it through a classifier, maybe we would be able to identify the ragam of the song.

Below I implement 3 simple CNN using the Dogs vs Cats Dataset from Kaggle. The first CNN uses regular Convolutions a Fully connected network to classify the images. The second approach uses Image Augmentation. For some reason, I did not get a better performance with Image Augumentation. Thirdly I use the pre-trained Inception v3 network.

# 1. Basic Convolutional Neural Network in Tensorflow & Keras

You can view the Colab notebook here – Cats_vs_dogs_1.ipynb

Here some important parts of the notebook

## Create CNN Model

• Use 3 Convolution + Max pooling layers with 32,64 and 128 filters respectively
• Flatten the data
• Have 2 Fully connected layers with 128, 512 neurons with relu activation
• Use sigmoid for binary classification
In [0]:
model = tf.keras.models.Sequential([
tf.keras.layers.Conv2D(32,(3,3),activation='relu',input_shape=(150,150,3)),
tf.keras.layers.MaxPooling2D(2,2),
tf.keras.layers.Conv2D(64,(3,3),activation='relu'),
tf.keras.layers.MaxPooling2D(2,2),
tf.keras.layers.Conv2D(128,(3,3),activation='relu'),
tf.keras.layers.MaxPooling2D(2,2),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(128,activation='relu'),
tf.keras.layers.Dense(512,activation='relu'),
tf.keras.layers.Dense(1,activation='sigmoid')
])

## Print model summary

In [13]:
model.summary()
Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d (Conv2D)              (None, 148, 148, 32)      896
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 74, 74, 32)        0
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 72, 72, 64)        18496
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 36, 36, 64)        0
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 34, 34, 128)       73856
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 17, 17, 128)       0
_________________________________________________________________
flatten (Flatten)            (None, 36992)             0
_________________________________________________________________
dense (Dense)                (None, 128)               4735104
_________________________________________________________________
dense_1 (Dense)              (None, 512)               66048
_________________________________________________________________
dense_2 (Dense)              (None, 1)                 513
=================================================================
Total params: 4,894,913
Trainable params: 4,894,913
Non-trainable params: 0
_________________________________________________________________

## Use the Adam Optimizer with binary cross entropy

loss='binary_crossentropy',
metrics=['accuracy'])

• Do Gradient Descent for 15 epochs
history=model.fit(train_generator,
validation_data=validation_generator,
steps_per_epoch=100,
epochs=15,
validation_steps=50,
verbose=2)
Epoch 1/15
100/100 - 13s - loss: 0.6821 - accuracy: 0.5425 - val_loss: 0.6484 - val_accuracy: 0.6131
Epoch 2/15
100/100 - 13s - loss: 0.6227 - accuracy: 0.6456 - val_loss: 0.6161 - val_accuracy: 0.6394
Epoch 3/15
100/100 - 13s - loss: 0.5975 - accuracy: 0.6719 - val_loss: 0.5558 - val_accuracy: 0.7206
Epoch 4/15
100/100 - 13s - loss: 0.5480 - accuracy: 0.7241 - val_loss: 0.5431 - val_accuracy: 0.7138
Epoch 5/15
100/100 - 13s - loss: 0.5182 - accuracy: 0.7447 - val_loss: 0.4839 - val_accuracy: 0.7606
Epoch 6/15
100/100 - 13s - loss: 0.4773 - accuracy: 0.7781 - val_loss: 0.5029 - val_accuracy: 0.7506
Epoch 7/15
100/100 - 13s - loss: 0.4466 - accuracy: 0.7972 - val_loss: 0.4573 - val_accuracy: 0.7912
Epoch 8/15
100/100 - 13s - loss: 0.4395 - accuracy: 0.7997 - val_loss: 0.4252 - val_accuracy: 0.8119
Epoch 9/15
100/100 - 13s - loss: 0.4314 - accuracy: 0.8019 - val_loss: 0.4931 - val_accuracy: 0.7481
Epoch 10/15
100/100 - 13s - loss: 0.4309 - accuracy: 0.7969 - val_loss: 0.4203 - val_accuracy: 0.8109
Epoch 11/15
100/100 - 13s - loss: 0.4329 - accuracy: 0.7916 - val_loss: 0.4189 - val_accuracy: 0.8069
Epoch 12/15
100/100 - 13s - loss: 0.4248 - accuracy: 0.8050 - val_loss: 0.4476 - val_accuracy: 0.7925
Epoch 13/15
100/100 - 13s - loss: 0.3868 - accuracy: 0.8306 - val_loss: 0.3900 - val_accuracy: 0.8236
Epoch 14/15
100/100 - 13s - loss: 0.3710 - accuracy: 0.8328 - val_loss: 0.4520 - val_accuracy: 0.7900
Epoch 15/15
100/100 - 13s - loss: 0.3654 - accuracy: 0.8353 - val_loss: 0.3999 - val_accuracy: 0.8100

## Plot results

• Plot training and validation accuracy

• Plot training and validation loss

#-----------------------------------------------------------
# Retrieve a list of list results on training and test data
# sets for each training epoch
#-----------------------------------------------------------
acc      = history.history[     'accuracy' ]
val_acc  = history.history[ 'val_accuracy' ]
loss     = history.history[    'loss' ]
val_loss = history.history['val_loss' ]

epochs   = range(len(acc)) # Get number of epochs

#------------------------------------------------
# Plot training and validation accuracy per epoch
#------------------------------------------------
plt.plot  ( epochs,     acc,label="training accuracy" )
plt.plot  ( epochs, val_acc, label='validation acuracy' )
plt.title ('Training and validation accuracy')
plt.legend()

plt.figure()

#------------------------------------------------
# Plot training and validation loss per epoch
#------------------------------------------------
plt.plot  ( epochs,     loss , label="training loss")
plt.plot  ( epochs, val_loss,label="validation loss" )
plt.title ('Training and validation loss'   )
plt.legend()

# 2. CNN with Image Augmentation

You can check the Cats_vs_Dogs_2.ipynb

Including the important parts of this implementation below

## Use Image Augumentation

Use Image Augumentation to improve performance

• Use the same model parameters as before
• Perform the following image augmentation
• width, height shift
• shear and zoom

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.optimizers import RMSprop
from tensorflow.keras.preprocessing.image import ImageDataGenerator
model = tf.keras.models.Sequential([
tf.keras.layers.Conv2D(32,(3,3),activation='relu',input_shape=(150,150,3)),
tf.keras.layers.MaxPooling2D(2,2),
tf.keras.layers.Conv2D(64,(3,3),activation='relu'),
tf.keras.layers.MaxPooling2D(2,2),
tf.keras.layers.Conv2D(128,(3,3),activation='relu'),
tf.keras.layers.MaxPooling2D(2,2),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(128,activation='relu'),
tf.keras.layers.Dense(512,activation='relu'),
tf.keras.layers.Dense(1,activation='sigmoid')
])

train_datagen = ImageDataGenerator(
rescale=1./255,
#rotation_range=90,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2)
#horizontal_flip=True,
#fill_mode='nearest')

validation_datagen = ImageDataGenerator(rescale=1./255)
#
train_generator = train_datagen.flow_from_directory(train_dir,
batch_size=32,
class_mode='binary',
target_size=(150, 150))
# --------------------
# Flow validation images in batches of 20 using test_datagen generator
# --------------------
validation_generator =  validation_datagen.flow_from_directory(validation_dir,
batch_size=32,
class_mode  = 'binary',
target_size = (150, 150))

loss='binary_crossentropy',
metrics=['accuracy'])
Found 20000 images belonging to 2 classes.
Found 5000 images belonging to 2 classes.

history=model.fit(train_generator,
validation_data=validation_generator,
steps_per_epoch=100,
epochs=15,
validation_steps=50,
verbose=2)
Epoch 1/15
100/100 - 27s - loss: 0.5716 - accuracy: 0.6922 - val_loss: 0.4843 - val_accuracy: 0.7744
Epoch 2/15
100/100 - 27s - loss: 0.5575 - accuracy: 0.7084 - val_loss: 0.4683 - val_accuracy: 0.7750
Epoch 3/15
100/100 - 26s - loss: 0.5452 - accuracy: 0.7228 - val_loss: 0.4856 - val_accuracy: 0.7665
Epoch 4/15
100/100 - 27s - loss: 0.5294 - accuracy: 0.7347 - val_loss: 0.4654 - val_accuracy: 0.7812
Epoch 5/15
100/100 - 27s - loss: 0.5352 - accuracy: 0.7350 - val_loss: 0.4557 - val_accuracy: 0.7981
Epoch 6/15
100/100 - 26s - loss: 0.5136 - accuracy: 0.7453 - val_loss: 0.4964 - val_accuracy: 0.7621
Epoch 7/15
100/100 - 27s - loss: 0.5249 - accuracy: 0.7334 - val_loss: 0.4959 - val_accuracy: 0.7556
Epoch 8/15
100/100 - 26s - loss: 0.5035 - accuracy: 0.7497 - val_loss: 0.4555 - val_accuracy: 0.7969
Epoch 9/15
100/100 - 26s - loss: 0.5024 - accuracy: 0.7487 - val_loss: 0.4675 - val_accuracy: 0.7728
Epoch 10/15
100/100 - 27s - loss: 0.5015 - accuracy: 0.7500 - val_loss: 0.4276 - val_accuracy: 0.8075
Epoch 11/15
100/100 - 26s - loss: 0.5002 - accuracy: 0.7581 - val_loss: 0.4193 - val_accuracy: 0.8131
Epoch 12/15
100/100 - 27s - loss: 0.4733 - accuracy: 0.7706 - val_loss: 0.5209 - val_accuracy: 0.7398
Epoch 13/15
100/100 - 27s - loss: 0.4999 - accuracy: 0.7538 - val_loss: 0.4109 - val_accuracy: 0.8075
Epoch 14/15
100/100 - 27s - loss: 0.4550 - accuracy: 0.7859 - val_loss: 0.3770 - val_accuracy: 0.8288
Epoch 15/15
100/100 - 26s - loss: 0.4688 - accuracy: 0.7688 - val_loss: 0.4764 - val_accuracy: 0.7786

## Plot results

• Plot training and validation accuracy
• Plot training and validation loss
In [15]:
import matplotlib.pyplot as plt
#-----------------------------------------------------------
# Retrieve a list of list results on training and test data
# sets for each training epoch
#-----------------------------------------------------------
acc      = history.history[     'accuracy' ]
val_acc  = history.history[ 'val_accuracy' ]
loss     = history.history[    'loss' ]
val_loss = history.history['val_loss' ]

epochs   = range(len(acc)) # Get number of epochs

#------------------------------------------------
# Plot training and validation accuracy per epoch
#------------------------------------------------
plt.plot  ( epochs,     acc,label="training accuracy" )
plt.plot  ( epochs, val_acc, label='validation acuracy' )
plt.title ('Training and validation accuracy')
plt.legend()

plt.figure()

#------------------------------------------------
# Plot training and validation loss per epoch
#------------------------------------------------
plt.plot  ( epochs,     loss , label="training loss")
plt.plot  ( epochs, val_loss,label="validation loss" )
plt.title ('Training and validation loss'   )
plt.legend()

# Implementation using Inception Network V3

The implementation is in the Colab notebook Cats_vs_Dog_3.ipynb

This is implemented as below

## Use Inception V3

import os

from tensorflow.keras import layers
from tensorflow.keras import Model

from tensorflow.keras.applications.inception_v3 import InceptionV3
pre_trained_model = InceptionV3(input_shape = (150, 150, 3),
include_top = False,
weights = 'imagenet')

for layer in pre_trained_model.layers:
layer.trainable = False

# pre_trained_model.summary()

last_layer = pre_trained_model.get_layer('mixed7')
print('last layer output shape: ', last_layer.output_shape)
last_output = last_layer.output
87916544/87910968 [==============================] - 1s 0us/step
last layer output shape:  (None, 7, 7, 768)

## Use Layer 7 of Inception Network

• Use Image Augumentation
In [0]:
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.optimizers import RMSprop
from tensorflow.keras.preprocessing.image import ImageDataGenerator
# Flatten the output layer to 1 dimension
x = layers.Flatten()(last_output)
# Add a fully connected layer with 1,024 hidden units and ReLU activation
x = layers.Dense(1024, activation='relu')(x)
# Add a dropout rate of 0.2
x = layers.Dropout(0.2)(x)
# Add a final sigmoid layer for classification
x = layers.Dense  (1, activation='sigmoid')(x)

model = Model( pre_trained_model.input, x)
#train_datagen = ImageDataGenerator( rescale = 1.0/255. )
#validation_datagen = ImageDataGenerator( rescale = 1.0/255. )

train_datagen = ImageDataGenerator(
rescale=1./255,
#rotation_range=90,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2)
#horizontal_flip=True,
#fill_mode='nearest')

validation_datagen = ImageDataGenerator(rescale=1./255)
#
train_generator = train_datagen.flow_from_directory(train_dir,
batch_size=32,
class_mode='binary',
target_size=(150, 150))
# --------------------
# Flow validation images in batches of 20 using test_datagen generator
# --------------------
validation_generator =  validation_datagen.flow_from_directory(validation_dir,
batch_size=32,
class_mode  = 'binary',
target_size = (150, 150))

loss='binary_crossentropy',
metrics=['accuracy'])
Found 20000 images belonging to 2 classes.
Found 5000 images belonging to 2 classes.

## Fit model

history=model.fit(train_generator,
validation_data=validation_generator,
steps_per_epoch=100,
epochs=15,
validation_steps=50,
verbose=2)
Epoch 1/15
100/100 - 31s - loss: 0.5961 - accuracy: 0.8909 - val_loss: 0.1919 - val_accuracy: 0.9456
Epoch 2/15
100/100 - 30s - loss: 0.2002 - accuracy: 0.9259 - val_loss: 0.1025 - val_accuracy: 0.9550
Epoch 3/15
100/100 - 30s - loss: 0.1618 - accuracy: 0.9366 - val_loss: 0.0920 - val_accuracy: 0.9581
Epoch 4/15
100/100 - 29s - loss: 0.1442 - accuracy: 0.9381 - val_loss: 0.0960 - val_accuracy: 0.9600
Epoch 5/15
100/100 - 30s - loss: 0.1402 - accuracy: 0.9381 - val_loss: 0.0703 - val_accuracy: 0.9794
Epoch 6/15
100/100 - 30s - loss: 0.1437 - accuracy: 0.9413 - val_loss: 0.1090 - val_accuracy: 0.9531
Epoch 7/15
100/100 - 30s - loss: 0.1325 - accuracy: 0.9428 - val_loss: 0.0756 - val_accuracy: 0.9670
Epoch 8/15
100/100 - 29s - loss: 0.1341 - accuracy: 0.9491 - val_loss: 0.0625 - val_accuracy: 0.9737
Epoch 9/15
100/100 - 29s - loss: 0.1186 - accuracy: 0.9513 - val_loss: 0.0934 - val_accuracy: 0.9581
Epoch 10/15
100/100 - 29s - loss: 0.1171 - accuracy: 0.9513 - val_loss: 0.0642 - val_accuracy: 0.9727
Epoch 11/15
100/100 - 29s - loss: 0.1018 - accuracy: 0.9591 - val_loss: 0.0930 - val_accuracy: 0.9606
Epoch 12/15
100/100 - 29s - loss: 0.1190 - accuracy: 0.9541 - val_loss: 0.0737 - val_accuracy: 0.9719
Epoch 13/15
100/100 - 29s - loss: 0.1223 - accuracy: 0.9494 - val_loss: 0.0740 - val_accuracy: 0.9695
Epoch 14/15
100/100 - 29s - loss: 0.1158 - accuracy: 0.9516 - val_loss: 0.0659 - val_accuracy: 0.9744
Epoch 15/15
100/100 - 29s - loss: 0.1168 - accuracy: 0.9591 - val_loss: 0.0788 - val_accuracy: 0.9669

## Plot results

• Plot training and validation accuracy
• Plot training and validation loss
In [14]:
import matplotlib.pyplot as plt
#-----------------------------------------------------------
# Retrieve a list of list results on training and test data
# sets for each training epoch
#-----------------------------------------------------------
acc      = history.history[     'accuracy' ]
val_acc  = history.history[ 'val_accuracy' ]
loss     = history.history[    'loss' ]
val_loss = history.history['val_loss' ]

epochs   = range(len(acc)) # Get number of epochs

#------------------------------------------------
# Plot training and validation accuracy per epoch
#------------------------------------------------
plt.plot  ( epochs,     acc,label="training accuracy" )
plt.plot  ( epochs, val_acc, label='validation acuracy' )
plt.title ('Training and validation accuracy')
plt.legend()

plt.figure()

#------------------------------------------------
# Plot training and validation loss per epoch
#------------------------------------------------
plt.plot  ( epochs,     loss , label="training loss")
plt.plot  ( epochs, val_loss,label="validation loss" )
plt.title ('Training and validation loss'   )
plt.legend()

I intend to do some interesting stuff with Convolutional Neural Networks.

Watch this space!

To see all posts click Index of posts

# Big Data 6: The T20 Dance of Apache NiFi and yorkpy

“I don’t count my sit-ups. I only start counting once it starts hurting. ”

“Hard work beats talent when talent doesn’t work hard.”

Tim Notke

In my previous post Big Data 5: kNiFI-ing through cricket data with Apache NiFi and yorkpy, I created a Big Data Pipeline that takes raw data in YAML format from a Cricsheet to processing and ranking IPL T20 players. In that post I had mentioned that we could create a similar pipeline to create a real time dashboard of IPL Analytics. I could have have done this but I needed to know how to create a Web UI. After digging and poking around, I have been able to create a simple Web UI running off Apache Web server. This UI uses basic JQuery and CSS to display a real time IPL T20 dashboard. As in my previous post, this is an end-2-end Big Data pipeline which can handle large data sets at scheduled times, process them and generate real time dashboards.

We could imagine an inter-galactic T20 championship league where T20 data comes in every hour or sooner and we need to perform analytics to see if us earthlings are any better than people with pointy heads  or little green men. The NiFi pipeline could be used as-is, however the yorkpy package would have to be rewritten in Pyspark. That is in another eon, though.

My package yorkpy has around ~45+ functions which fall in the following main categories

1. Pitching yorkpy . short of good length to IPL – Part 1 :Class 1: This includes functions that convert the yaml data of IPL matches into Pandas dataframe which are then saved as CSV. This part can perform analysis of individual IPL matches.
2. Pitching yorkpy.on the middle and outside off-stump to IPL – Part 2 :Class 2:This part includes functions to create a large data frame for head-to-head confrontation between any 2IPL teams says CSK-MI, DD-KKR etc, which can be saved as CSV. Analysis is then performed on these team-2-team confrontations.
3. Pitching yorkpy.swinging away from the leg stump to IPL – Part 3 Class 3:The 3rd part includes the performance of any IPL team against all other IPL teams. The data can also be saved as CSV.
4. Pitching yorkpy … in the block hole – Part 4 :Class 4: This part performs analysis of individual IPL batsmen and bowlers

Watch the live demo of the end-2-end NiFi pipeline at ‘The T20 Dance

You can download the NiFi template and associated code from Github at  T20 Dance

The Apache NiFi Pipeline is shown below

## 1. T20 Dance – Overall NiFi Pipeline

There are 5 process groups

## 2. ListAndConvertYaml2DataFrames

This post starts with having the YAML files downloaded and unpacked from Cricsheet.  The individual YAML files are converted into Pandas dataframes and saved as CSV. A concurrency of 12 is used to increase performance and process YAML files in parallel. The processor MergeContent creates a merged content to signal the completion of conversion and triggers the other Process Groups through a funnel.

## 3. Analyse individual IPL T20 matches

This Process Group ‘Analyse T20 matches’  used the yorkpy’s Class 1 functions which can perform analysis of individual IPL T20 matches. The matchWorm() and matchScorecard() functions are used, through any other function could have been used. The Process Group is shown below

## 4. Analyse performance of an IPL team in all matches against another IPL team

This Process Group ‘Analyse performance of IPL team in all matched against another IPL team‘ does analysis in all matches between any 2 IPL teams (Class 2) as shown below

## 5. Analyse performance of IPL team in all matches against all other IPL teams

This uses Class 3 functions. Individual data sets for each IPL team versus all other IPL teams is created before Class 3 yorkpy functions are invoked. This is included below

## 6. Analyse performances of IPL batsmen and bowlers

This Process Group uses Class 4 yorkpy functions. The match CSV files are processed to get batting and bowling details before calling the individual functions as shown below

## 7. IPL T20 Dashboard

The IPL T20 Dashboard is shown

## Conclusion

This NiFI pipeline was done for IPL T20 however, it could be done for any T20 format like Intl T20, BBL, Natwest etc which are posted in Cricsheet. Also, only a subset of the yorkpy functions were used. There is a much wider variety of functions available.

Hope the T20 dance got your foot a-tapping!

To see all posts click Index of posts

# Big Data-5: kNiFi-ing through cricket data with yorkpy

“The temptation to form premature theories upon insufficient data is the bane of our profession.”

Sherlock Holmes in the Valley of fear by Arthur Conan Doyle

“If we have data, let’s look at data. If all we have are opinions, let’s go with mine.”

Jim Barksdale, former CEO Netscape

In this post I use  Apache NiFi Dataflow Pipeline along with my Python package yorkpy to crunch through cricket data from Cricsheet. The Data Pipelne  flows all the way from the source  to target analytics output. Apache NiFi was created to automate the flow of data between systems.  NiFi dataflows enable the automated and managed flow of information between systems. This post automates the flow of data from Cricsheet, from where the zip file it is downloaded, unpacked, processed, transformed and finally T20 players are ranked.

While this is a straight forward example of what can be done, this pattern can be applied to real Big Data systems. For example hypothetically, we could consider that we get several parallel streams of  cricket data or for that matter any sports related data. There could be parallel Data flow pipelines that get the data from the sources. This would then be  followed by data transformation modules and finally a module for generating analytics. At the other end a UI based on AngularJS or ReactJS could display the results in a cool and awesome way.

Incidentally, the NiFi pipeline that I discuss in this post, is a simplistic example, and does not use the Big Data stack like HDFS, Hive, Spark etc. Nevertheless, the pattern used, has all the modules for a Big Data pipeline namely ingestion, unpacking, transformation and finally analytics. This NiF pipeline demonstrates the flow using the regular file system of Mac and my python based package yorkpy. The concepts mentioned could be used in a real Big Data scenario which has much fatter pipes of data coming. If  this was the case the NiFi pipeline would utilize  HDFS/Hive for storing the ingested data and Pyspark/Scala for the transformation and analytics and other related technologies.

A pictorial representation is given below

In the diagram above each of the vertical boxes could be any technology from the ever proliferating Big Data stack namely HDFS, Hive, Spark, Sqoop, Kafka, Impala and so on.  Such a dataflow automation could be created when any big sporting event happens, as long as the data generated large, and there is a need for dynamic and automated reporting. The UI could be based on AngularJS/ReactJS and could display analytical tables and charts.

This post demonstrates one such scenario in which IPL T20 data is downloaded from Cricsheet site, unpacked and stored in a specific directory. This dataflow automation is based on my yorkpy package. To know more about the yorkpy package  see Pitching yorkpy … short of good length to IPL – Part 1  and the associated parts. The zip file, from Cricsheet, contains individual IPL T20 matches in YAML format. The convertYaml2DataframeT20() function is used to convert the YAML files into Pandas dataframes before storing them as CSV files. After this done, the function rankIPLT20batting() function is used to perform the overall ranking of the T20 players. My yorkpy Python package has about ~ 50+ functions that perform various analytics on any T20 data for e.g it has the following classes of functions

• analyze T20 matches
• analyze performance of a T20 team in all matches against another T20 team
• analyze performance of a T20 team against all other T20 teams
• analyze performance of T20 batsman and bowlers
• rank T20 batsmen and bowlers

The functions of yorkpy generate tables or charts. While this post demonstrates one scenario, we could use any of the yorkpy T20 functions, generate the output and display on a widget in the UI display, created with cool technologies like AngularJS/ReactJS,  possibly in near real time as data keeps coming in.,

To use yorkpy with NiFI the following packages have to be installed in your environment

-pip install yorkpy
-pip install pyyaml
-pip install pandas
-yum install python-devel (equivalent in Windows)
-pip install matplotlib
-pip install seaborn
-pip install sklearn
-pip install datetime

I have created a video of the NiFi Pipeline with the real dataflow fro source to the ranked IPL T20 batsmen. Take a look at RankingT20PlayersWithNiFiYorkpy

You can clone/fork the NiFi template from rankT20withNiFiYorkpy

The NiFi Data Flow Automation is shown below

## 1. Overall flow

The overall NiFi flow contains 2 Process Groups a) DownloadAnd Unpack. b) Convert and Rank IPL batsmen. While it appears that the Process Groups are disconnected, they are not. The first process group downloads the T20 zip file, unpacks the. zip file and saves the YAML files in a specific folder. The second process group monitors this folder and starts processing as soon the YAML files are available. It processes the YAML converting it into dataframes before storing it as CSV file. The next  processor then does the actual ranking of the batsmen before writing the output into IPLrank.txt

This process group is shown below

The ${T20data} variable points to the specific T20 format that needs to be downloaded. I have set this to https://cricsheet.org/downloads/ipl.zip. This could be set any other data set. In fact we could have parallel data flows for different T20/ Sports data sets and generate #### 1.1.2 SaveUnpackedData This processor stores the YAML files in a predetermined folder, so that the data can be picked up by the 2nd Process Group for processing ### 1.2 ProcessAndRankT20Players Process Group This is the second process group which converts the YAML files to pandas dataframes before storing them as. CSV files. The RankIPLPlayers will then read all the CSV files, stack them and then proceed to rank the IPL players. The Process Group is shown below #### 1.2.1 ListFile and FetchFile Processors The left 2 Processors ListFile and FetchFile get all the YAML files from the folder and pass it to the next processor #### 1.2.2 convertYaml2DataFrame Processor The convertYaml2DataFrame Processor uses the ExecuteStreamCommand which call a python script. The Python script invoked the yorkpy function convertYaml2Dataframe() as shown below The${convertYaml2Dataframe} variable points to the python file below which invoked the yorkpy function yka.convertYaml2PandasDataframeT20()

import yorkpy.analytics as yka
import argparse
parser = argparse.ArgumentParser(description='convert')
args=parser.parse_args()
yamlFile=args.yamlFile
yka.convertYaml2PandasDataframeT20(yamlFile,"/Users/tvganesh/backup/software/nifi/ipl","/Users/tvganesh/backup/software/nifi/ipldata")

### Reshape and rescale

# Reshape the array
x_train <- array_reshape(x_train, c(nrow(x_train), 784))
x_test <- array_reshape(x_test, c(nrow(x_test), 784))
# Rescale
x_train <- x_train / 255
x_test <- x_test / 255

### Convert out put to One Hot encoded format

y_train <- to_categorical(y_train, 10)
y_test <- to_categorical(y_test, 10)

### Fit the model

Use the softmax activation for recognizing 10 digits and categorical cross entropy for loss

model <- keras_model_sequential()
model %>%
layer_dense(units = 256, activation = 'relu', input_shape = c(784)) %>%
layer_dense(units = 128, activation = 'relu') %>%
layer_dense(units = 10, activation = 'softmax') # Use softmax

model %>% compile(
loss = 'categorical_crossentropy',
optimizer = optimizer_rmsprop(),
metrics = c('accuracy')
)

### Fit the model

Note: A smaller number of epochs has been used. For better performance increase number of epochs

history <- model %>% fit(
x_train, y_train,
epochs = 5, batch_size = 128,
validation_data = list(x_test,y_test)
)

### Plot the accuracy and loss for training and test data

plot(history)

Conclusion
This post shows how to use Tensorflow and Keras in both Python & R
Hope you have fun with Tensorflow!!

To see all posts click Index of posts