A Basic Dashboard

This article will demonstrate how to put together a basic dashboard that can be used to visualize data. The server and client in this article will refer to Windows machines.

Technologies Used

  • Java 7 Update 51 – Create applications from the Java Programming Language
  • Maven 3.0.5 – Project Management system for Java Programs
  • Eclipse – Development Environment to develop Java Programs
  • Git 1.9.4 – Version Control system, Unix command prompt for Windows
  • CopSSH 4.8.0 – SSH system for Windows
  • MongoDB 2.6.2 – Database system
  • Tomcat 7.0.54 – Web Server
  • Jenkins 1.568 – Continuous Integration, Job Management
  • JQuery 1.8.2 – Javascript/Ajax API
  • HighCharts – Javascript Chart Generator

Architecture Overview
We will build a Server which has the following capabilities:

  • Client Access via SSH
  • Hosting source code via Git
  • Accessing shared data from a MongoDB database
  • Hosting a front end webpage to visualize the shared data
  • Running periodic jobs to fetch and refresh the shared data

We will build a Client which has the following capabilities:

  • Accessing the server via SSH
  • Accessing source code via Git
  • Modifying shared data from a MongoDB database
  • Developing a Front End Web Application to visualize the shared data
  • Developing a Back End Java Application to fetch and refresh the shared data

Step 1: Setup the git user on the Server
Start, Right Click Computer, Manage, Select Configuration, Select Local Users and Groups, New user, Add user name “git” and give a password and keep track of it (it will need letters, numbers and a few special characters). Double click git and uncheck “Change password on next login”.

Make the directory “C:\Users\git”.

You will use the username and home directory below to login to the Server from the Client using SSH.

Step 2: Setup CopSSH on the Server
The Server will use CopSSH to allow SSH capability to provide access to the shared Java code (aka source code).

From github.com
From git-scm.com
CopSSH Installer – http://download.cnet.com/CopSSH/3001-7240_4-10218944.html?spi=dd79c53bc9a32dcbcf09df4229e7d536&dlm=0

Press Next through all screens, but enter a password and keep track of it. This will setup ssh as a Windows service.

Open CopSSH, Users Tab, Add, Forward, select git, Forward, enter home directory “C:\Users\git”, Forward, Apply. This verifies that step 1 above worked.

Step 3: Setup Git on the Client and Server
The Client will use Git as a command prompt and for modifying source code.
The Server will use Git as a command prompt and for hosting source code.

Git Installer – https://github.com/msysgit/msysgit/releases/download/Git-1.9.4-preview20140611/Git-1.9.4-preview20140611.exe

After installation, configure Git to be run from the command line:
Right click Computer, Advanced system settings, Environment variables, Under System variables select PATH, Edit, append “;C:\Program Files (x86)\Git\bin”. This will allow git to be run from the command line.

Two scripts are needed to upload and download source code via the ‘git push’ and ‘git clone’ commands. Copy the two scripts “git-upload-pack.exe” and “git-receive-pack.exe” from “C:\Program Files (x86)\Git\libexec\git-core” to “C:\Program Files (x86)\Git\bin”.

TEST SSH CONNECTION
From the client, open Git Bash. Type the command ‘ssh git@my.server.com’. Here my.server.com represents the domain name such as http://www.google.com or it could be your ip address.

When prompted enter the password for the git user that you created above. The Git Bash command prompt should change to show you are logged in (ie git@my.server.com). This step verifies that CopSSH was installed correctly. If you have trouble here, make sure the CopSSH service is running by opening the CopSSH Control Panel.

USE PUBLIC AND PRIVATE KEYS FOR AUTOMATIC AUTHENTICATION
Open Git Bash on the Client. Type the command ‘cd ~/.ssh’. Then, generate the public/private keys by typing the command ‘ssh-keygen’. Enter “id_rsa” as the file name and don’t enter a passphrase.

The .ssh folder should now contain id_rsa and id_rsa.pub. Copy the id_rsa.pub file from the Client to the “C:\Users\git” folder on the server.

Open Git Bash on the Server. Type the command following two commands to append the public key to the authorized_keys file


cd "C:\Users\git"
cat id_rsa.pub >> authorized_keys

Go back to the Git Bash on the Client and type the command ‘ssh git@my.server.com’. This time you should not need to type in a password. (You may need to delete the known_hosts file in the .ssh directory and close and reopen Git Bash)

CREATE THE REPOSITORY FOR YOUR PROJECT
From the Server, Open Git Bash and enter the following commands (replace project with the name of your project):


cd "C:\Users\git"
mkdir project.git
cd project.git
git --bare init

From the Client, Open Git Bash and enter the following commands after moving to a directory where you wish to store your project:


mkdir project
cd project
git init
touch new
git add -A
git commit -m "initial commit"
git remote add origin git@my.server.com:project.git
git push origin master

From now on, any other clients that want access to this code can issue:


git clone git@my.server.com:project.git

Step 4: Setup MongoDB on the Server
The Server will use MongoDB to provide access to the shared data.

From docs.mongodb.org
64 bit MongoDB Installer – https://fastdl.mongodb.org/win32/mongodb-win32-x86_64-2008plus-2.6.2.zip

You can run this command to find out if you need 64 bit or 32 bit:


wmic os get osarchitecture

Copied to server, extracted all the files, copied the bin folder to “C:\mongodb\bin” (and other miscellaneous files moved to mongodb folder).

CONFIGURE AND START MONGO DB SERVER


Open Git Bash on the Server
cd /c/mondodb
mkdir data
mkdir data/db
 
mkdir data/log
echo logpath="C:\mongodb\data\log\mongo.log" > "C:\mongodb\mongod.cfg"
echo dbpath="C:\mongodb\data\db" >> "C:\mongodb\mongod.cfg"
sc.exe create MongoDB binPath= "\"C:\mongodb\bin\mongod.exe\" --service --config=\"C:\mongodb\mongod.cfg\"" DisplayName= "MongoDB 2.6 Standard" start= "auto"

If you reopen the Services from Task Manager, Services Tab, Services button, you will see “MongoDB 2.6 Standard”
You may start MongoDB from the services or enter the command ‘net start MongoDB’.

ADD DATA TO MONGO DB

From docs.mongodb.org


Open Git Bash on the Server
/c/mongodb/bin/mongo.exe
db
(displays test, the database created by default)
use mydb
(creates a database called mydb, switches to that database)
j = { name : "mongo" }
k = { x : 3 }
(creates some data)
db.testData.insert(j)
db.testData.insert(k)
(creates the testData collection and adds the data to the collection)
show collections
(shows the new testData collection)

TEST CONNECTION FROM THE CLIENT

Mongo VUE Installer – http://www.mongovue.com/Installer-1.6.9.zip

Open Mongo VUE, hit Connect, hit plus button, Enter ‘MongoDB’ in Name, add ‘my.server.com’ in Server, add 27017 in Port and ‘testData’ in Database(s). The configuration above created a ‘test’ database by default. There is no username or password.

Hit Save, Connect. You can expand ‘mydb’ database to see the ‘testData’ collection and double click that to see the 2 documents added (name=mongo and x=3).

Step 5: Setup Java on the Client and Server
The Client will use Java to develop the Web Application and Java Application.
The Server will use Java to run Tomcat.

For help, see Getting Started with Java’s Hello World: Part I
From www.oracle.com
Java 7 Update 51 Installer – http://download.oracle.com/otn/java/jdk/7u51-b13/jdk-7u51-windows-x64.exe

After installation, configure Java to be run from the command line:
Right click Computer, Advanced system settings, Environment variables, Under System variables select PATH, Edit, append “;C:\Program Files\Java\jdk1.7.0_51\bin”.

You can test this by opening a command prompt and running the following command ‘java -version’. If it shows 1.7.0_51, Java has been installed correctly.

Step 6: Setup Maven on the Client and Server
The Client will use Java to develop the Web Application and Java Application.
The Server will use Maven to run a periodic job using a Jenkins script.

For help, see Getting Started with Java’s Hello World: Part II
From maven.apache.org
Maven 3.0.5 Installer – http://www.dsgnwrld.com/am/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.zip

After installation, configure Maven to be run from the command line:
Right click Computer, Advanced system settings, Environment variables, Under System variables select PATH, Edit, append “;C:\apache-maven-3.0.5\bin”.

Create a new system variable, JAVA_HOME whose value is “C:\Program Files\Java\jdk1.7.0_51”.

You can test this by opening a command prompt and running the following command ‘maven -version’. If it shows 3.0.5, Maven has been installed correctly.

Step 7: Setup Eclipse on the Client
For help, see Getting Started with Java’s Hello World: Part III

Step 8: Create a Back End Java Project to Add More Data to Mongo DB
For help, see the Getting Started tutorial links in above.

CREATE THE BACK END PROJECT
Create a new Maven project (create a default Java Application, assuming Java, Maven and Eclipse are installed). In the pom.xml file, add the mongo dependency.

ADD THE BACK END PROJECT MAVEN DEPENDENCIES


<!-- Mongo DB -->
<dependency>
    <groupId>org.mongodb</groupId>
    <artifactId>mongo-java-driver</artifactId>
    <version>2.11.3</version>
</dependency>

ADD THE BACK END CONTROLLER CLASS TO POPULATE THE MONGO DB WITH DATA
In the src/main/java folder, add a package and a class:


package my.company.com.BackEnd;

import java.net.UnknownHostException;
import java.util.ArrayList;
import java.util.List;
import com.mongodb.BasicDBObject;
import com.mongodb.DB;
import com.mongodb.DBCollection;
import com.mongodb.DBCursor;
import com.mongodb.DBObject;
import com.mongodb.Mongo;
import com.mongodb.ServerAddress;

public class BackEndController {
	public static void main(String[] args) {
		System.out.println("Starting BackEndController");
		DB mongoDB = createMongoDB("my.server.com", "", "", "mydb");
		DBCollection testCollection = mongoDB.getCollection("test_collection");
		deleteAllEntries(testCollection);
		
		DBObject record1 = new BasicDBObject();
		record1.put("First", "Johnny");
		record1.put("Last", "Smith");
		testCollection.save(record1);
		
		DBObject record2 = new BasicDBObject();
		record2.put("First", "Jane");
		record2.put("Last", "Smith");
		testCollection.save(record2);
		
		System.out.println("Finishing BackEndController");
	}
	
	public static DB createMongoDB(String replicaSetURL, String username, String password, String databaseName) {
		try {
			ArrayList addresses = new ArrayList();
			for (String address: replicaSetURL.split(",")) {
				addresses.add(new ServerAddress(address));
			}
			Mongo mongo = new Mongo(addresses);
			mongo.slaveOk();
			DB mongoDB = mongo.getDB(databaseName);
			mongoDB.authenticate(username, password.toCharArray());
			return mongoDB;
		} catch (UnknownHostException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
			return null;
		}
	}
	
	public static List findAll(DBCollection collection) {
		List objects = new ArrayList();
		DBCursor cursor = collection.find();
		try {
			while(cursor.hasNext()) {
				DBObject object = cursor.next();
				objects.add(object);
			}
		} finally {
			cursor.close();
		}
		return objects;
	}
	
	public static  void deleteAllEntries(DBCollection collection){
		List objects = findAll(collection);
		for(DBObject object: objects) {
			collection.remove(object);
		}
	}
}

Right click, Run As Java. Then, use Mongo VUE to verify the “test_collection” collection was created and the new data records were added.

COMMIT THE CODE FOR THE BACK END PROJECT
From Eclipse, delete this project from the Package Explorer (but don’t delete from your hard drive). Copy the contents to the project folder created above. Then commit this project to the Git central repository, by opening Git Bash from the Client and running the following commands:


Move to the project directory
git status
(Should show all the files that you are about to add for the Back End Project)
git add -A
git commit -m "Adding the Back End project"
git push origin master

This code represents code that is responsible for fetching external data, within the Back End that you wish to display.

Step 9: Integrate Tomcat with Eclipse
For help, see “Setup Server” from Creating a New Web App to Handle File Upload.

Step 10: Create the Front End Web App to Display Data from Mongo DB

CREATE THE FRONT END PROJECT
Create a new Maven project (create a default Java Web Application, assuming Java, Maven, Eclipse and Tomcat are installed).

ADD THE POM DEPENDENCIES FOR THE FRONT END PROJECT
In the pom.xml file, add the mongodb (connect to database), spring (configure web app servlet) and httpclient (connect to server) dependencies.


<!-- Mongo DB -->
<dependency>
    <groupId>org.mongodb</groupId>
    <artifactId>mongo-java-driver</artifactId>
    <version>2.11.3</version>
</dependency>
 
 
<!-- org.springframework.web.servlet -->
<dependency>
    <groupId>org.springframework</groupId>
    <artifactId>spring-webmvc</artifactId>
    <version>3.0.5.RELEASE</version>
</dependency>
 
<!-- HttpClient -->
<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.2.2</version>
</dependency>

ADD THE SERVLET FOR THE FRONT END PROJECT
In the src/main/webapp/WEB-INF folder, edit the web.xml file to contain the following (notice we are using version=2.5):


<web-app xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://java.sun.com/xml/ns/javaee 
          http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
    version="2.5">
 
    <display-name>Archetype Created Web Application</display-name>
 
    <servlet>
        <servlet-name>FrontEndController</servlet-name>
        <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
        <init-param>
            <param-name>contextConfigLocation</param-name>
            <param-value>/WEB-INF/fecontroller-servlet.xml</param-value>
        </init-param>
        <load-on-startup>1</load-on-startup>
    </servlet>
 
    <servlet-mapping>
        <servlet-name>FrontEndController</servlet-name>
        <url-pattern>/fecontroller/*</url-pattern>
    </servlet-mapping>
</web-app>

ADD THE BEANS NEEDED FOR THE SERVLET
In the WEB-INF folder add the file “fecontroller-servlet.xml” containing the following:


<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
                        http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd">
 
    <context:component-scan base-package="my.company.com.FrontEnd" />
 
    <bean
        class="org.springframework.web.servlet.view.InternalResourceViewResolver">
        <property name="prefix">
            <value>/WEB-INF/</value>
        </property>
        <property name="suffix">
            <value>.jsp</value>
        </property>
    </bean>
 
</beans>

ADD THE DISPLAY WEB PAGE FOR THE SERVLET
Add the src/main/webapp/WEB-INF/fecontroller-display.jsp file to have the following (we will substitute the body with something else):


<html>
<body>
<h2>Front End Controller Display</h2>
 
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.8.2/jquery.min.js"></script>
<script src="http://code.highcharts.com/highcharts.js"></script>
<script src="http://code.highcharts.com/modules/exporting.js"></script>
 
<div id="container" style="min-width: 310px; height: 400px; max-width: 600px; margin: 0 auto"></div>
<script type="text/javascript">
$(function () {
    $('#container').highcharts({
        chart: {
            plotBackgroundColor: null,
            plotBorderWidth: 1,//null,
            plotShadow: false
        },
        title: {
            text: 'User Time Percentages'
        },
        tooltip: {
            pointFormat: '{series.name}: <b>{point.percentage:.1f}%</b>'
        },
        plotOptions: {
            pie: {
                allowPointSelect: true,
                cursor: 'pointer',
                dataLabels: {
                    enabled: true,
                    format: '<b>{point.name}</b>: {point.percentage:.1f} %',
                    style: {
                        color: (Highcharts.theme && Highcharts.theme.contrastTextColor) || 'black'
                    }
                }
            }
        },
        series: [{
            type: 'pie',
            name: 'Time share',
            data: [
                ['${firstName}',   65.2],
                ['${secondName}',   34.8],
            ]
        }]
    });
});
</script>
</body>
</html>

We will pull the firstName and secondName from the database, but later all this data can be pulled as desired.

ADD THE REST CONTROLLER FOR THE SERVLET
Add the Front End Controller which will use REST service


package my.company.com.FrontEnd;
import java.net.UnknownHostException;
import java.util.ArrayList;
import java.util.List;
import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.util.EntityUtils;
import org.springframework.stereotype.Controller;
import org.springframework.ui.ModelMap;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import com.mongodb.DB;
import com.mongodb.DBCollection;
import com.mongodb.DBCursor;
import com.mongodb.DBObject;
import com.mongodb.Mongo;
import com.mongodb.ServerAddress;
@Controller
@RequestMapping("/")
public class FrontEndController {
    public static void main(String[] args) {
        HttpClient httpclient = new DefaultHttpClient();
         
        try {
            String url = "http://localhost:8080/FrontEnd/fecontroller/display";
            HttpGet httpGet = new HttpGet(url);
            HttpResponse response = httpclient.execute(httpGet);
            System.out.println("output1:"+EntityUtils.toString(response.getEntity()));
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
  
    @RequestMapping(value="display", method = RequestMethod.GET)
    public String welcome(ModelMap model) {
        DB mongoDB = createMongoDB("my.server.com", "", "", "mydb");
        DBCollection testCollection = mongoDB.getCollection("test_collection");
        List dbObjects = findAll(testCollection);
        String firstName = dbObjects.get(0).get("First") + " " + dbObjects.get(0).get("Last");
        String secondName = dbObjects.get(1).get("First") + " " + dbObjects.get(1).get("Last");
         
        model.addAttribute("firstName", firstName);
        model.addAttribute("secondName", secondName);
        return "fecontroller-display";  //Spring uses InternalResourceViewResolver and return back fecontroller-display.jsp
    }
    public static DB createMongoDB(String replicaSetURL, String username, String password, String databaseName) {
        try {
            ArrayList addresses = new ArrayList();
            for (String address: replicaSetURL.split(",")) {
                addresses.add(new ServerAddress(address));
            }
            Mongo mongo = new Mongo(addresses);
            mongo.slaveOk();
            DB mongoDB = mongo.getDB(databaseName);
            mongoDB.authenticate(username, password.toCharArray());
            return mongoDB;
        } catch (UnknownHostException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
            return null;
        }
    }
     
    public List findAll(DBCollection collection) {
        List objects = new ArrayList();
        DBCursor cursor = collection.find();
        try {
            while(cursor.hasNext()) {
                DBObject object = cursor.next();
                objects.add(object);
            }
        } finally {
            cursor.close();
        }
        return objects;
    }
}

TEST THE SERVLET IN TOMCAT IN ECLIPSE


Open Git Bash and prepare the Front End project
mvn eclipse:eclipse -Dwtpversion=2.0

In the Servers Tab, add the Front End project to Tomcat, Right Click and Start.
You can then open Firefox and view the display at http://localhost:8080/FrontEnd/fecontroller/display

COMMIT THE CODE FOR THE FRONT END PROJECT
From Eclipse, delete this project from the Package Explorer (but don’t delete from your hard drive). Copy the contents to the project folder created above. Then commit this project to the Git central repository, by opening Git Bash from the Client and running the following commands:


Move to the project directory
git status
(Should show all the files that you are about to add for the Back End Project)
git add -A
git commit -m "Adding the Front End project"
git push origin master

This code represents code that is responsible for displaying internal data.

Step 11: Setup Tomcat on the Server to Host the Front End Display
From tomcat.apache.org
Tomcat 7 Installer – http://mirror.symnds.com/software/Apache/tomcat/tomcat-7/v7.0.54/bin/apache-tomcat-7.0.54.exe

After installing the Java jdk, you can use this in the tomcat installation (ie “C:\Program Files\Java\jdk1.7.0_51”).

On the client, Open Git Bash and move to the directory containing the pom.xml file for the Front End project. Enter the command ‘mvn install’ to create the FrontEnd.war file, which will be found in the target directory. Copy this to the Tomcat webapps directory on the Server (ie “C:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps”).

From the Services Tab, find Tomcat, stop, start.

Open Firefox and hit the url, this time using the server host name: http://my.server.com:8080/FrontEnd/fecontroller/display

Step 12: Setup Jenkins on the Server
Note: Up to this point, we have not discussed the Maven Repository. These instructions don’t handle setting up a Maven Repository. This is what automatically pulls in certain maven dependencies when you run the ‘mvn compile’ command from the Git Bash. The local and central repositories are handled by the settings.xml file which can be found at “C\apache-maven-3.0.5\conf\settings.xml”. If you cannot run the ‘mvn exec:java’ command below on the Jenkins machine you will have to find another way to compile the code (such as packaging the libraries with the code).

From jenkins-ci.org
Jenkins 1.568 Installer – http://mirrors.jenkins-ci.org/war/latest/jenkins.war

Copy this war file to the Tomcat webapps directory on the Server and restart tomcat

Open Firefox and hit the url: http://my.server.com:8080/jenkins. You may see “Please wait while Jenkins is getting ready to work”

CONFIGURE THE JENKINS USER FOR SSH
From wiki.jenkins-ci.org

For the jenkins user (by default, “Local System account”), create the .ssh directory (for Local System account, use “C:\Windows\SysWOW64\config\systemprofile\.ssh”).

From above, when you cloned the git repo from the Client, it created a known_hosts file in the .ssh folder (‘cd ~/.ssh’). Copy the id_rsa, id_rsa.pub and known_hosts files into the .ssh directory for the jenkins user on the Server. Remember, when Jenkins is running, it will be a Git Client connecting to the Git Server to pull the source code.

CONFIGURE A JENKINS JOB TO RUN THE BACK END CODE
From the jenkins web console, create new jobs, enter “BackEnd” as name, “build a free-style software project”

Select build periodically, enter cron expression “5 8 * * 6” (don’t include quotes) to build every Saturday and 8:05am. This represents having data collected automatically on a regular basis.

Add build step, Execute Windows batch command, enter this script


echo "Removing old project directory"
rm -rf project 
 
echo "Checking out master branch"
git clone git@my.server.com:project.git
cd project/Backend
git log -1
 
echo "Running Backend"
mvn exec:java -Dexec.mainClass="my.company.com.BackEnd.BackEndController"

On the job, click build now. Verify that it ran successfully.

Step 13: Test the whole system
On the client, edit the backend controller to insert “JJJ Andrews” as the first name (these lines for record1 can be found in the main method of BackEndController.java:


        DBObject record1 = new BasicDBObject();
        record1.put("First", "JJJ");
        record1.put("Last", "Andrews");
        testCollection.save(record1);

Save the code and commit via Git Bash


git add -A
git commit -m "Test backend data change"
git push origin master

Although this change would automatically be run on Saturday morning, we will manually run it through Jenkins

Click build now, from the Firefox on the Jenkins terminal.

You should now see “JJJ Andrews”.

image2014-6-20 15-19-26

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s