Bend it like Bluemix, MongoDB with auto-scaling – Part 3

In this last post of this series, I test the performance of Bluemix & MongoDB against concurrent queries and deletes to the cloud based app with Mongo DB, with auto-scaling on. Before I started these series of tests I moved the Overload policy a couple of notches higher and made it scale out if memory utilization > 75% for 120 secs and < 30% for 120 secs (from the earlier 55% memory utilization) as shown below.

The code for bluemixMongo app can be forked from Devops at bluemixMongo or can be cloned from GitHub at bluemix-mongo-autoscale. The multi-mechanize scripts can be downloaded from GitHub at multi-mechanize Before starting the testing I checked the current number of documents inserted by the concurrent inserts (see Bend it like Bluemix., MongoDB using Auto-scaling – Part 2). The total number as determined by checking the logs was 1380 Sure enough with the scaling policy change after 2 minutes the number of instanced dropped from 3 to 2

1. Querying the bluemixMongo app with Multi-mechanize

The Multi-mechanize Python script used for querying the bluemixMongo app simply invokes the app’s userlist URL (resp=br.open(“http://bluemixmongo.mybluemix.net/userlist/”)

v_user.py

def run(self): # create a Browser instance br = mechanize.Browser() # don"t bother with robots.txt br.set_handle_robots(False) # start the timer start_timer = time.time() #print("Display userlist") # Display 5 random documents resp=br.open("http://bluemixmongo.mybluemix.net/userlist/") assert("Example Mongo Page" in resp.get_data()) # stop the timer latency = time.time() - start_timer self.custom_timers["Userlist"] = latency r = random.uniform(1, 2) time.sleep(r) self.custom_timers['Example_Timer'] = r

The configuration setup for this script creates 2 sets of 10 concurrent threads

config.cfg
run_time = 300 rampup = 0 results_ts_interval = 10 progress_bar = on console_logging = off xml_report = off [user_group-1] threads = 10 script = v_user.py [user_group-2] threads = 10 script = v_user.py

The corresponding userlist.js for querying the app is shown below. Here the query is constructed by creating a ‘RegularExpression’ with a random Firstname, consisting of a random letter and a random number. Also the query is also limited to 5 documents.

function(callback) { // Display a random set of 5 records based on a regular expression made with random letter, number var randnum = Math.floor((Math.random() * 10) + 1); var alpha = ['A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P','Q','R','S','T','U','V','X','Y','Z']; var randletter = alpha[Math.floor(Math.random() * alpha.length)]; var val = randletter + ".*" + randnum + ".*"; // Limit the display to 5 documents var results = collection.find({"FirstName": new RegExp(val)}).limit(5).toArray(function(err, items){ if(err) { console.log(err + " Error getting items for display"); } else { res.render('userlist', { "userlist" : items }); // end res.render } //end else db.close(); // Ensure that the open connection is closed }); // end toArray function callback(null, 'two'); }

2. Running the userlist query

The following screenshot shows the userlist query being executed concurrently with Multi-mechanize. Note that the number of instances also drops down to 1

3. Deleting documents with Multi-mechanize

The multi-mechanize script for deleting a document is shown below. This script calls the URL with resp = br.open(“http://bluemixmongo.mybluemix.net/remuser”). No values are required to be entered into the form and the ‘submit’ is simulated.

v_user.py def run(self): # create a Browser instance br = mechanize.Browser() # don"t bother with robots.txt br.set_handle_robots(False) br.addheaders = [("User-agent", "Mozilla/5.0Compatible")] # start the timer start_timer = time.time() # submit the request resp = br.open("http://bluemixmongo.mybluemix.net/remuser") #resp = br.open("http://localhost:3000/remuser") resp.read() # stop the timer latency = time.time() - start_timer # store the custom timer self.custom_timers["Load_Front_Page"] = latency # think-time time.sleep(2) # select first (zero-based) form on page br.select_form(nr=0) # set form field br.form["firstname"] = "" br.form["lastname"] = "" br.form["mobile"] = "" # start the timer start_timer = time.time() # submit the form resp = br.submit() resp.read() print("Removed") # stop the timer latency = time.time() - start_timer # store the custom timer self.custom_timers["Delete"] = latency # think-time time.sleep(2)

config.cfg

The config file is set to start 2 sets of 10 concurrent threads and execute for 300 secs

[global] run_time = 300 rampup = 0 results_ts_interval = 10 progress_bar = on console_logging = off xml_report = off [user_group-1] threads = 10 script = v_user.py [user_group-2] threads = 10 script = v_user.py ;

deleteuser.js

This Node.js script does a findOne() document and does a remove with the ‘justOne’ set to true

collection.findOne(function(err, item) { // Delete just a single record collection.remove(item, {justOne:true},(function (err, doc) { if (err) { // If it failed, return error res.send("There was a problem removing the information to the database."); } else { // If it worked redirect to userlist res.location("userlist"); // And forward to success page res.redirect("userlist"); } })); }); collection.find().toArray(function(err, items) { console.log("Length =----------------" + items.length); db.close(); }); callback(null, 'two');

4. Running the deleteuser multimechanize script

The output of the script executing and the reduction of the number of instances because of the change in the memory utilization policy is shown

5. Multi-mechanize

As mentioned in the previous posts

The multi-mechanize commands are executed as follows
To create a new project
multimech-newproject.exe userlist
This will create 2 folders a) results b) test_scripts and the file c) config.cfg. The v_user.py needs to be updated as required

To run the script
multimech-run.exe userlist

The details of the response times for the query is shown below .

More details on latency and throughput for the queries and the deletes are included in the results folder of multi-mechanize

6. Autoscaling The details of the auto-scaling service is shown below

a. Scaling Metric Statistics

b. Scaling history

7. Monitoring and Analytics (M & A) The output from M & A is shown below

a. Performance Monitoring

b. Log Analysis output The log analysis give a detailed report on the calls made to the app, the console log output and other useful information

The series of the 3 posts Bend it like Bluemix, MongoDB with auto-scaling demonstrated the ability of the cloud to expand and shrink based on the load on the cloud.An important requirement for Cloud Architects is design applications that can scale horizontally without impacting the performance while keeping the costs optimum. The real challenge to auto-scaling is the need to make the application really distributed as opposed to the monolithic architectures we are generally used to. I hope to write another post on creating simple distributed application later.

Hasta la Vista!

Also see
1. Bend it like Bluemix, MongoDB with autoscaling – Part 1
2. Bend it like Bluemix, MongoDB with autoscaling – Part 2

Disclaimer: This article represents the author’s viewpoint only and doesn’t necessarily represent IBM’s positions, strategies or opinions

Bend it like Bluemix, MongoDB using Auto-scaling – Part 2!

This post takes off from my previous post Bend it like Bluemix, MongoDB using Auto-scale – Part 1! In this post I generate traffic using Multi-Mechanize a performance test framework and check out the auto-scaling on Bluemix, besides also doing some rudimentary check on the latency and throughput for this test application. In this particular post I generate concurrent threads which insert documents into MongoDB.

Note: As mentioned in my earlier post this is more of a prototype and the typical situation when architecting cloud applications. Clearly I have not optimized my cloud app (bluemixMongo) for maximum efficiency. Also this a simple 2 tier application with a rudimentary Web interface and a NoSQL DB at This is more of a Proof of Concept (PoC) for the auto-scaling service on Bluemix.

As earlier mentioned the bluemixMongo app is a modification of my earlier post Spicing up a IBM Bluemix cloud app with MongoDB and NodeExpress. The bluemixMongo cloud app that was used for this auto-scaling test can be forked from Devops at bluemixMongo or from GitHib at bluemix-mongo-autoscale. The Multi-mechanize config file, scripts and results can be found at GitHub in multi-mechanize

The document to be inserted into MongoDB consists of 3 fields – Firstname, Lastname and Mobile. To simulate the insertion of records into MongoDB I created a Multi-Mechanize script that will generate random combination of letters and numbers for the First and Last names and a random 9 digit number for the mobile. The code for this script is shown below

1. The snippet below measure the latency for loading the ‘New User’ page

v_user.py
def run(self): # create a Browser instance br = mechanize.Browser() # don"t bother with robots.txt br.set_handle_robots(False) print("Rendering new user") br.addheaders = [("User-agent", "Mozilla/5.0Compatible")] # start the timer start_timer = time.time() # submit the request resp = br.open("http://bluemixmongo.mybluemix.net/newuser") #resp = br.open("http://localhost:3000/newuser") resp.read() # stop the timer latency = time.time() - start_timer # store the custom timer self.custom_timers["Load Add User Page"] = latency # think-time time.sleep(2)

The script also measures the time taken to submit the form containing the Firstname, Lastname and Mobile

# select first (zero-based) form on page br.select_form(nr=0) # Create random Firstname a = (''.join(random.choice(string.ascii_uppercase) for i in range(5))) b = (''.join(random.choice(string.digits) for i in range(5))) firstname = a + b # Create random Lastname a = (''.join(random.choice(string.ascii_uppercase) for i in range(5))) b = (''.join(random.choice(string.digits) for i in range(5))) lastname = a + b # Create a random mobile number mobile = (''.join(random.choice(string.digits) for i in range(9))) # set form field br.form["firstname"] = firstname br.form["lastname"] = lastname br.form["mobile"] = mobile # start the timer start_timer = time.time() # submit the form resp = br.submit() print("Submitted.") resp.read() # stop the timer latency = time.time() - start_timer # store the custom timer self.custom_timers["Add User"] = latency

2. The config.cfg file is setup to generate 2 asynchronous thread pools of 10 threads for about 400 seconds

config.cfg
run_time = 400 rampup = 0 results_ts_interval = 10 progress_bar = on console_logging = off xml_report = off [user_group-1] threads = 10 script = v_user.py [user_group-2] threads = 10 script = v_user.py

3. The code to add a new user in the app (adduser.js) uses the ‘async’ Node module to enforce sequential processing.

adduser.js
async.series([ function(callback) { collection = db.collection('phonebook', function(error, response) { if( error ) { return; // Return immediately } else { console.log("Connected to phonebook"); } }); callback(null, 'one'); }, function(callback) // Insert the record into the DB collection.insert({ "FirstName" : FirstName, "LastName" : LastName, "Mobile" : Mobile }, function (err, doc) { if (err) { // If it failed, return error res.send("There was a problem adding the information to the database."); } else { // If it worked, redirect to userlist - Display users res.location("userlist"); // And forward to success page res.redirect("userlist") } }); collection.find().toArray(function(err, items) { console.log("**************************>>>>>>>Length =" + items.length); db.close(); // Make sure that the open DB connection is close }); callback(null, 'two'); } ]);

4. To checkout auto-scaling the instance memory was kept at 128 MB. Also the scale-up policy was memory based and based on the memory of the instance exceeding 55% of 128 MB for 120 secs. The scale up based on CPU utilization was to happen when the utilization exceed 80% for 300 secs.

5. Check the auto-scaling policy

6. Initially as seen there is just a single instance

7. At around 48% of the script with around 623 transactions the instance is increased by 1. Note that the available memory is decreased by 640 MB – 128 MB = 512 MB.

8. At around 1324 transactions another instance is added

Note: Bear in mind

a) The memory threshold was artificially brought down to 55% of 128 MB.b) The app itself is not optimized for maximum efficiency

9. The Metric Statistics tab for the Autoscaling service shows this memory breach and the trigger for autoscaling

10. The Scaling history Tab for the Auto-scaling service displays the scale-up and scale-down and the policy rules based on which the scaling happened

11. If you go to the results folder for the Multi-mechanize tool the response and throughput are captured.

The multi-mechanize commands are executed as follows
To create a new project
multimech-newproject.exe adduser
This will create 2 folders a) results b) test_scripts and the file c) config.cfg. The v_user.py needs to be updated as required

To run the script
multimech-run.exe adduser

12.The results are shown below

a) Load Add User page (Latency)

b) Load Add User (Throughput)

c)Load Add User (Latency)

d) Load Add User (Throughput)

The detailed results can be seen at GitHub at multi-mechanize

13. Check the Monitoring and Analytics Page

a) Availability

b) Performance monitoring

So once the auto-scaling happens the application can be fine-tuned and for performance. Obviously one could do it the other way around too.

As can be seen adding NoSQL Databases like MongoDB, Redis, Cloudant DB etc. Setting up the auto-scaling policy is also painless as seen above.

Of course the real challenge in cloud applications is to make them distributed and scalable while keeping the applications themselves lean and mean!

a) Latency, throughput implications for the cloud

b) The many faces of latency

c) Design principles of scalable, distributed systems

Disclaimer: This article represents the author’s viewpoint only and doesn’t necessarily represent IBM’s positions, strategies or opinions

Spicing up a IBM Bluemix cloud app with MongoDB and NodeExpress

In this post I highlight the rudiments for a creating a cloud application on IBM’s PaaS offering Bluemix, using MongoDB and NodeExpress. Clearly Bluemix allows one to fire up a cloud application with a NoSQL database in a matter of a few hours which makes it really attractive. The NodeExpress application was initially created using Enide Studio for Node.js with a local Mongodb server running on my desktop. (Please see my post Elements of CRUD with Node Express and MongoDB) Once you have ironed out the issues in this local application you are ready to deploy on IBM Bluemix.

The code for this Bluemix application can be forked from bluemix-mongo from IBM Devops.

You can also clone the code from GitHub at bluemix-mongo

Here are the key changes that need to be made for running the NodeExpress Webserver with MongoDB as the backend DB

1) Webserver : Setup the port and host for the Webserver.

app.js

var port = (process.env.VCAP_APP_PORT || 1337); var host = (process.env.VCAP_APP_HOST || '0.0.0.0'); var app = express(); app.configure(function(){ app.set('port', port);

As seen above the host & port for the Webserver are obtained from the process.env variable.
2) Routes and Middleware
Setup the routes and invoke them appropriately in app.js
var express = require('express') , routes = require('./routes') , user = require('./routes/user') , userlist = require('./routes/userlist') , newuser = require('./routes/newuser') , adduser = require('./routes/adduser') , changeuser = require('./routes/changeuser') , updateuser = require('./routes/updateuser') , remuser = require('./routes/remuser') , deleteuser = require('./routes/deleteuser')

app.get('/users', user.list); app.get('/helloworld', routes.index); app.get('/userlist', userlist.list); app.get('/newuser', newuser.list); app.post('/adduser',adduser.list); app.get('/changeuser', changeuser.list); app.post('/updateuser', updateuser.list); app.get('/remuser', remuser.list); app.post('/deleteuser',deleteuser.list);

3) Initialize MongoDB database: Create a set of 3 records when the Webserver starts as follows

if (process.env.VCAP_SERVICES) { var env = JSON.parse(process.env.VCAP_SERVICES); if (env['mongodb-2.2']) { var mongo = env['mongodb-2.2'][0]['credentials']; } } else { var mongo = { "username" : "user1", "password" : "secret", "url" : "mongodb://user1:secret@localhost:27017/test" } } var MongoClient = mongodb.MongoClient; var db= MongoClient.connect(mongo.url, function(err, db) { if(err) { log("failed to connect to the database"); } else { log("connected to database"); } var collection = db.collection('phonebook'); //Clear DB and insert 3 records remove(mycallback); var user1 = { "FirstName" : "Tinniam", "LastName" : "Ganesh","Mobile": "916732177728" }; var user2 = { "FirstName" : "Darth", "LastName" : "Vader","Mobile": "6666699999" }; var user3 = { "FirstName" : "Bill", "LastName" : "Shakespeare","Mobile": "8342189991" };

insert(user1,function(err,result){});
insert(user2,function(err,result){});
insert(user3,function(err,result){});
find().toArray(function(err, items) {

});
});

3) Home Page: Setup up a Home page with the CRUD operations when the Bluemix cloud application’s route for e.g. http://bluemix-mongo.mybluemix.net is clicked. This is shown below.

4) Display Users: To display the list of users the route /userlist is invoked. This function gets all the records from the collection and stores them into a toArray element, which is then used for rendering the list of uses with a ‘userlist.jade’ template

userlist.js
var MongoClient = mongodb.MongoClient; var db= MongoClient.connect(mongo.url, function(err, db) { if(err) {

log(“Failed to connect to the database”);

} else {

log(“Connected to database”);

}
var collection = db.collection(‘phonebook’);
//Get all records and display them

find().toArray(function(err, items) {
log(items);
render(‘userlist’, {

“userlist” : items
});
});
});

jade

This template displays the list of users as a table. The code is shown below

extends layout block content h1= "Display the list of Users" p strong Firstname Lastname Mobile table each user, i in userlist tr td #{user.FirstName} td #{user.LastName} td #{user.Mobile} p p a(href='/') Home

Note: A link back to the Home page is included in here at the bottom.

5) Adding a User
There are 2 parts to this
a) Invoking the /newuser route to display the input form through the newuser.jade
b) Invoking the /adduser route to insert the values entered in the form. The changes are shown below
a) app.js
.. newuser = require('./routes/newuser') adduser = require('./routes/adduser') .. app.get('/newuser', newuser.list); app.post('/adduser',adduser.list);

b) newuser.js
exports.list = function(req, res){

render(‘newuser’, { title: ‘Add User’});

};

The newuser jade displays the input form
c) newuser.jade
extends layout block content h1= "Add a User" form#formAddUser(name="adduser",method="post",action="/adduser") input#inputUserFirstName(type="text", placeholder="firstname", name="firstname") input#inputUserLastName(type="text", placeholder="lastname", name="lastname") input#inputUserLastName(type="text", placeholder="mobile", name="mobile") button#btnSubmit(type="submit") submit p p a(href='/') Home

d) adduser.js

The adduser.js gets the mongo url from the process.env.VCAP_SERVICES and setups up the connection to the DB and inserts the values received in the ‘newuser.jade’ form into the database

exports.list = function(req, res) { if (process.env.VCAP_SERVICES) { var env = JSON.parse(process.env.VCAP_SERVICES); if (env['mongodb-2.2']) { var mongo = env['mongodb-2.2'][0]['credentials']; } } else { var mongo = { "username" : "user1", "password" : "secret", "url" : "mongodb://user1:secret@localhost:27017/test" } } // Set up the DB connection var MongoClient = mongodb.MongoClient; var db= MongoClient.connect(mongo.url, function(err, db) { if(err) {

log(“Failed to connect to the database”);

} else {

log(“Connected to database”);

}
// Get our form values. These rely on the “name” attributes
var FirstName = req.body.firstname;
var LastName = req.body.lastname;
var Mobile = req.body.mobile;
// Set our collection
var collection = db.collection(‘phonebook’);
// Insert the record into the DB

insert({

“FirstName” : FirstName,
“LastName” : LastName,
“Mobile” : Mobile
}, function (err, doc) {
if (err) {
// If it failed, return error

send(“There was a problem adding the information to the database.”);

}
else {
// Redirect to userlist – Display users

location(“userlist”);

// And forward to success page

redirect(“userlist”);

}
});
});

If the insert is successful the userlist page is displayed with the new user

6) Updating a User & Deleting a User: Updating and Deleting users follow the same format as Adding a user.

7) index.jade The Home page is built using index.jade with a hyperlink invoking the route for each database operation
extends layout block content h1= title p Welcome to #{title} ul li a(href='/userlist') Display list of users li a(href='/newuser') Add a user li a(href='/changeuser') Update a user li a(href='/remuser') Delete a user

Tip: “Return of the Jade“i : Getting the jade template right is truly an art as Jade is extremely finicky about spaces, tabs, indents and outdents(???). Creating the Jade template had me run into circles. I found out that you can debug the jade template individually by executing

C:> npm install jade -g" and then running C:> jade <template name>
from the command prompt. If the result of the command is “rendered <template name>.html” then you are in luck and you can incorporate this jade template into your views folder for e.g.

C: >jade index.jade rendered index.html

8) Push changes to Bluemix: Once the changes have been made push the changes on to Bluemix with ‘cf’ as follows

cf login -a https://api.ng.bluemix.net cf push bluemix-mongo -p . -m 512M cf create-service mongodb 100 mongodb01 cf bind-service bluemix-mongo mongodb01

The last 2 commands can also be performed through the Bluemix dashboard in which you add the mongodb service to your Node.js app/

8) Files and Logs: In the Bluemix dashboard you can check your logs in the Files and Logs

Important tip: Finally if the application fails to start when you push the application with ‘cf’ for e.g.

cf push <app name> -p . -m 512M .... ..... ----> Writing a custom .npmrc to circumvent npm bugs ----> Installing dependencies ----> Caching node_modules directory for future builds ----> Cleaning up node-gyp and npm artifacts ----> No Procfile found; Adding npm start to new Procfile ----> Building runtime environment ----> Checking and configuring service extensions ----> Uploading droplet (7.6M) of 1 instances running, 1 down of 1 instances running, 1 down of 1 instances running, 1 down of 1 instances running, 1 down
or if it crashes when you click a link then your debugging friend is

cf logs <app name > — recent
This will dump the error that was encountered either while the application was being started of why the application crashed.

You can fork this Bluemix application from bluemix-mongo at IBM Devops or from GitHub at bluemix-mongo

Disclaimer: This article represents the author’s viewpoint only and doesn’t necessarily represent IBM’s positions, strategies or opinions

Find me on Google+

Elements of CRUD with NodeExpress and MongoDB using Enide Studio

In this post I perform basic CRUD operations using NodeExpress and MongoDB with Enide Studio. There is not a whole lot of information in the Web on using Node Express with Enide Studio so the task was kind of difficult. Anyway I managed to get basic CRUD operations to work. The complete code can be cloned from nodeexpress-mongo.

Here it is.

1) To get started create a new Node Express project with Enide Studio File->New->NodeExpress Project.

2) This should create the necessary files, routes and views.

3) Start the MongoDB as follows

mongod –dbpath <folder of project>

4) Start the mongo console. To get started I created 3 records as follows

mongo> db.phonebook.insert({ "FirstName" : "Tinniam", "LastName" : "Ganesh","Mobile": "916732177728" }) db.phonebook.insert({ "FirstName" : "Darth", "Lastname" : "Vader","Mobile": "6666699999" }) db.phonebook.insert({ "FirstName" : "Bill", "Lastname" : "Shakespeare","Mobile": "8342189991" })

5) You can display the added records with

> db.phonebook.find().pretty() { "_id" : ObjectId("53de3f8e0e7f8abf82c0c850"), "FirstName" : "Tinniam", "LastName" : "Ganesh", "Mobile" : "916732177728" } { "_id" : ObjectId("53de3fed0e7f8abf82c0c851"), "FirstName" : "Darth", "Lastname" : "Vader", "Mobile" : "6666699999" } { "_id" : ObjectId("53de40200e7f8abf82c0c852"), "FirstName" : "Bill", "Lastname" : "Shakespeare", "Mobile" : "8342189991"

6) For each of the CRUD operations there are 4 components

– Set the route
– Invoke app.get/app.post as approriate
– Call the necessary route Javascript file
– Use the appropriate Jade constructs for rendering the output

7) Here are the changes for the Display of the users

A) Displaying the User
a) app.js
.. userlist = require('./routes/userlist') .. app.get('/userlist', userlist.list); ..

var mongodb = require(‘mongodb’);

b) userlist.js

/* GET Phone users page. */
exports.list = function(req, res) { // var db = req.db; var MongoClient = mongodb.MongoClient; var db= MongoClient.connect("mongodb://localhost:27017/test", function(err, db) { if(err) {

log(“Failed to connect to the database”);

} else {

log(“Connected to database”);

}
var collection = db.collection(‘phonebook’);

find().toArray(function(err, items) {
log(items);
render(‘userlist’, {

“userlist” : items
});
});
});
};

c) userlist.jade
extends layout block content h1= "Display the list of Users" p strong Firstname Lastname Mobile table each user, i in userlist tr td #{user.FirstName} td #{user.LastName} td #{user.Mobile} p p a(href='http://localhost:3000') Home

This is shown below

B) Adding a User

There are 2 parts to this

a) Invoking the /newuser route to display the input form
b) Invoking the /adduser route to insert the values entered in the form. The changes are shown below

a) app.js
.. newuser = require('./routes/newuser') adduser = require('./routes/adduser') .. app.get('/newuser', newuser.list); app.post('/adduser',adduser.list);

These require the following

b) newuser.js

exports.list = function(req, res){

render(‘newuser’, { title: ‘Add User’});

};

The newuser jade displays the input form

c) newuser.jade

block content h1= "Add a User" form#formAddUser(name="adduser",method="post",action="/adduser") input#inputUserFirstName(type="text", placeholder="firstname", name="firstname") input#inputUserLastName(type="text", placeholder="lastname", name="lastname") input#inputUserLastName(type="text", placeholder="mobile", name="mobile") button#btnSubmit(type="submit") submit p p a(href='http://localhost:3000') Home

The /adduser route inserts the record into the phonebook collection as shown

d) adduser.js

var FirstName = req.body.firstname; var LastName = req.body.lastname; var Mobile = req.body.mobile; // Set our collection var collection = db.collection('phonebook'); // Insert the record into the DB

insert({

“FirstName” : FirstName,
“LastName” : LastName,
“Mobile” : Mobile
}, function (err, doc) {
if (err) {
// If it failed, return error

send(“There was a problem adding the information to the database.”);

}
else {
// If it worked, redirect to userlist – Display users

location(“userlist”);

// And forward to success page

redirect(“userlist”);

}
});

This takes us back to Display users if it successfully added the user

C) Updating a User and Deleting a User are similar to Adding a User

D) Index page

All the actions are included in the index.jade as shown below with hyperlinks

p Welcome to #{title}
ul li a(href='http://localhost:3000/userlist') Display list of users li a(href='http://localhost:3000/newuser') Add a user li a(href='http://localhost:3000/changeuser') Update a user li a(href='http://localhost:3000/remuser') Delete a user

Important tip:
Here is an important note. I found that getting the jade template right extremely frustrating. It is really unforgiving and is very picky up indents, outdents(whatever that is!), tabs and spaces. A quick way to check whether your jade template is fine or not is to install jade in your project directory.

npm install jade –global
You can debug your jade with
If there an error you will get

c:> jade userlist.jade throw e ^ TypeError: userlist.jade:6 4| h1= title 5| ul > 6| each user, i in userlist 7| li 8| #{user.FirstName} Cannot read property 'length' of undefined at eval (eval at <anonymous>

If there are no errors you should see that the html is rendered as below

C: >jade index.jade rendered index.html

You can clone the complete code from GitHub at nodeexpress-mongo

Take a look at
1. Brewing a potion with Bluemix, PostgreSQL & Node.js in the cloud
2. A Bluemix recipe with MongoDB and Node.js
3. Spicing up IBM Bluemix with MongoDB and NodeExpress
4. A Cloud Medley with IBM’s Bluemix, Cloudant and Node.js
5. Rock N’ Roll with Bluemix, Cloudant & NodeExpress

Find me on Google+

A Bluemix recipe with MongoDB and Node.js

Here is a tasty IBM Bluemix recipe with a dash of MongoDB and a pinch of Node.js. This posts shows the steps needed to perform basic CRUD (Create, Remove, Update & Delete) operations on the MongoDB database using REST APIs of PUT,GET, UPDATE & DELETE.

You can fork the code for the below app from Devops at mymongodb

The code can also be cloned from GitHub at mymongodb

For this, the first thing we need to do is to create a Webserver using Node.js as shown below

Webserver

require('http').createServer(function(req, res) { if ( typeof mongodb !== 'undefined' && mongodb ) { // Perform CRUD operations through REST APIs if(req.method == 'POST') { insert_records(req,res); } else if(req.method == 'GET') { list_records(req,res); } else if(req.method == 'PUT') { update_records(req,res); } else if(req.method == 'DELETE') { delete_record(req,res); } } else { res.writeHead(200, {'Content-Type': 'text/plain'}); res.write("No MongoDB service instance is bound.\n"); res.end(); } }).listen(port, host);

The Webserver users the port and host values obtained as shown above to wait for HTTP requests.

The REST API calls are handled by individual Node.js functions to perform the operations of insert, update, delete and select.

Insertions

The code for insertions is shown below. For this a set of 5 documents are created and then inserted using Node.js

Updating documents in Mongodb

For update 2 documents are changed as shown below

var update_records = function(req, res) { var MongoClient = require('mongodb').MongoClient; MongoClient.connect (mongo.url, function(err, db) { // Update var collection = db.collection('books', function(err, collection) { collection.update({book:"Fountainhead"},{$set:{qty:2}}, {w:1},function(err,result) {}); collection.update({book:"Animal Farm"},{$set:{author:"George Orwell"}}, {w:1},function(err,result) {}); console.log("Updated 2 books");

}); // var collection
}); //End MongoClient.connect

}; //End update-records

Deletions

The delete functions requires a callback method which is included

var delete_record = function(req, res) { var MongoClient = require('mongodb').MongoClient; MongoClient.connect (mongo.url, function(err, db) {

//Deleting documents
var collection = db.collection(‘books’, function(err, collection) {

collection.remove({book:”Foundation”},mycallback);
collection.remove({book:”The Da Vinci Code”},{w:1},mycallback);
console.log(‘Deleted 2 books’);

});
}); //End MongoClient.connect
}; //End delete-records

Retrieving documents

To retrieve documents the collection.find.stream() method is used as below

var list_records = function(req, res) { var MongoClient = require('mongodb').MongoClient; MongoClient.connect (mongo.url, function(err, db) { //Retrieve documents var collection = db.collection('books', function(err, collection) { var stream = collection.find().stream(); console.log("Printing values..."); res.writeHead(200, {'Content-Type': 'text/plain'}); stream.on('error', function (err) { console.error(err.stack) });

stream.on(“data”, function(item) {
console.log(item);
res.write(JSON.stringify(item) + “\n”);
});

stream.on(“end”, function() {
console.log(“End”);
res.end();
});
}); //var collection
}); //End MongoClient.connect

The connection between the Node.js & Webserver and the MongoDB is setup using the VCAP_SERVICES as follows

if (process.env.VCAP_SERVICES) { var env = JSON.parse(process.env.VCAP_SERVICES); if (env['mongodb-2.2']) { var mongo = env['mongodb-2.2'][0]['credentials']; } } else { var mongo = { "username" : "user1", "password" : "secret", "url" : "mongodb://user1:secret@localhost:27017/test" } }

To get started you can fork the code for the above Bluemix- MongoDB app from Devops from mymongodb

The code can also be cloned from GitHub at mymongodb

After you have forked the code you can clone the code into a local directory on your machine.

Now use the ‘cf’ command to push the code onto IBM Bluemix as shown. In my case I named the app as mymongodb01

cf login -a https://api.ng.bluemix.net cf push mymongodb01 -p . -m 512M cf create-service mongodb 100 mongodb cf bind-service mymongodb01 mongodb
Instead of the last 2 steps you can also use the Add-service in Bluemix dashboard to add the MongoDB service. (Note: You will have to check Experimental at the top and you will see the service under Data Management.) After the MongoDB service is added check if your app is running in the Bluemix dashboard

If the app is running you can check the CRUD operations on MongoDB using the SureUtils-REST API client extension to Chrome.

The CRUD operations performed are shown below

1.POST + GET

Here 5 documents are inserted in the MongoDB and then displayed subsequently

2.UPDATE + GET

Here 2 records are updated – The quantity of the book ‘Fountainhead’ is set to 2 and the author of ‘Animal Farm’ is set to George Orwell

3.DELETE + GET

In this set 2 book are deleted and the result is displayed

If all things went well you should be able to see the app running.

You can also get the output o the console.login Files and Logs in the Bluemix dashboard.

Important tip: While executing the Buemix app if you run into problems and you app crashes with the “Health decreased” for your app and its colour turning red you can use the recent history of ‘cf” command to debug your problem

PS C:\Users\IBM_ADMIN\git\mymongodb> cf logs mymongodb01 –recent

Connected, dumping recent logs for app mymongodb01 in org tvganesh.85@gmail.com / space dev as tvganesh.85@gmail.com…

…….

2014-07-27T11:34:26.45+0530 [App/0] OUT We are connected to DB 2014-07-27T11:34:26.46+0530 [App/0] OUT Updated 2 books 2014-07-27T11:34:31.17+0530 [App/0] OUT We are connected to DB 2014-07-27T11:34:31.17+0530 [App/0] OUT Updated 2 books 2014-07-27T11:34:31.18+0530 [RTR] OUT mymongodb01.mybluemix.net - [27/07/2014:06:04:31 +0000] "PUT / HTTP/1.1" 200 1 7 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36" 75 .126.70.43:19986 vcap_request_id:4fa831610b6e79455d0257581a51014e response_time:0.009122768 app_id:ed49744f-1a29-4e3f-9e 1c-8b45e85c3310 2014-07-27T11:35:02.75+0530 [App/0] ERR 2014-07-27T11:35:02.75+0530 [App/0] ERR /home/vcap/app/app.js:95 2014-07-27T11:35:02.75+0530 [App/0] ERR MongoClient.connect (mongo.url, function(err, db) { 2014-07-27T11:35:02.75+0530 [App/0] ERR ^ 2014-07-27T11:35:02.75+0530 [App/0] ERR ReferenceError: MongoClient is not defined 2014-07-27T11:35:02.75+0530 [App/0] ERR at delete_record (/home/vcap/app/app.js:95:2) 2014-07-27T11:35:02.75+0530 [App/0] ERR at Server.<anonymous> (/home/vcap/app/app.js:165:12) 2014-07-27T11:35:02.75+0530 [App/0] ERR at Server.emit (events.js:98:17) 2014-07-27T11:35:02.75+0530 [App/0] ERR at HTTPParser.parser.onIncoming (http.js:2108:12) 2014-07-27T11:35:02.75+0530 [App/0] ERR at HTTPParser.parserOnHeadersComplete [as onHeadersComplete] (http.js:121: 23) 2014-07-27T11:35:02.75+0530 [App/0] ERR at Socket.socket.ondata (http.js:1966:22) 2014-07-27T11:35:02.75+0530 [App/0] ERR at TCP.onread (net.js:527:27) 2014-07-27T11:35:02.80+0530 [RTR] OUT mymongodb01.mybluemix.net - [27/07/2014:06:05:02 +0000] "DELETE / HTTP/1.1" Mi ssingResponseStatusCode 0 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.19