K8PetStore: A scalable kubernetes application with automated load generation (#3137).

examples/k8petstore/README.md

+## Welcome to k8PetStore
+
+This is a follow up to the Guestbook example, which implements a slightly more real world demonstration using 
+
+the same application architecture.
+
+- It leverages the same components (redis, Go REST API) as the guestbook application
+- It comes with visualizations for graphing whats happening in Redis transactions, along with commandline printouts of transaction throughput
+- It is hackable : you can build all images from the files is in this repository (With the exception of the data generator, which is apache bigtop).
+- It generates massive load using a semantically rich, realistic transaction simulator for petstores
+
+This application will run a web server which returns REDIS records for a petstore application.
+It is meant to simulate and test high load on kubernetes or any other docker based system.
+
+If you are new to kubernetes, and you haven't run guestbook yet, 
+
+you might want to stop here and go back and run guestbook app first.  
+
+The guestbook tutorial will teach you alot about the basics of kubernetes, and we've tried not to be redundant here.
+
+## Architecture of this SOA
+
+A diagram of the overall architecture of this application can be seen in arch.dot (you can paste the contents in any graphviz viewer, including online ones such as http://sandbox.kidstrythisathome.com/erdos/.
+
+## Docker image dependencies
+
+Reading this section is optional, only if you want to rebuild everything from scratch.
+
+This project depends on three docker images which you can build for yourself and save
+in your dockerhub "dockerhub-name".
+
+Since these images are already published under other parties like redis, jayunit100, and so on,
+so you don't need to build the images to run the app. 
+
+If you do want to build the images, you will need to build and push these 3 docker images.
+
+- dockerhub-name/bigpetstore-load-generator, which generates transactions for the database.
+- dockerhub-name/redis, which is a simple curated redis image.
+- dockerhub-name/k8petstore, which is the web app image.
+
+## Get started with the WEBAPP 
+
+The web app is written in Go, and borrowed from the original Guestbook example by brendan burns.
+
+We have extended it to do some error reporting, persisting of JSON petstore transactions (not much different then guestbook entries),
+
+and supporting of additional REST calls, like LLEN, which returns the total # of transactions in the database.
+
+To run it locally, you simply need to run basic Go commands.  Assuming you have Go set up, do something like: 
+
+```
+#Assuming your gopath is in / (i.e. this is the case, for example, in our Dockerfile).
+go get main
+go build main
+export STATIC_FILES=/tmp/static
+/gopath/bin/main
+```
+
+## Set up the data generator
+
+The web front end provides users an interface for watching pet store transactions in real time as they occur.
+
+To generate those transactions, you can use the bigpetstore data generator.  Alternatively, you could just write a 
+
+shell script which calls "curl localhost:3000/k8petstore/rpush/blahblahblah" over and over again :).  But thats not nearly
+
+as fun, and its not a good test of a real world scenario where payloads scale and have lots of information content. 
+
+Similarly, you can locally run and test the data generator code, which is Java based, you can pull it down directly from 
+
+apache bigtop.
+
+Directions for that are here : https://github.com/apache/bigtop/tree/master/bigtop-bigpetstore/bigpetstore-transaction-queue
+
+You will likely want to checkout the branch 2b2392bf135e9f1256bd0b930f05ae5aef8bbdcb, which is the exact commit which the current k8petstore was tested on.
+
+## Set up REDIS
+
+Install and run redis locally.  This can be done very easily on any Unix system, and redis starts in an insecure mode so its easy 
+
+to develop against.
+
+Install the bigpetstore-transaction-queue generator app locally (optional), but for realistic testing.
+Then, run the go app directly.  You will have to get dependencies using go the first time (will add directions later for that, its easy).
+
+## Now what? 
+
+Once you have done the above 3 steps, you have a working, from source, locally runnable version of the k8petstore app, now, we can try to run it in kubernetes.
+
+## Hacking, testing, benchmarking
+
+Once the app is running, you can go to the location of publicIP:3000 (the first parameter in the script).  In your browser, you should see a chart 
+
+and the k8petstore title page, as well as an indicator of transaction throughput, and so on.  You should be able to modify  
+
+You can modify the HTML pages, add new REST paths to the Go app, and so on.
+
+## Running in kubernetes
+
+Now that you are done hacking around on the app, you can run it in kubernetes.  To do this, you will want to rebuild the docker images (most likely, for the Go web-server app), but less likely for the other images which you are less likely to need to change. Then you will push those images to dockerhub.
+
+Now, how to run the entire application in kubernetes? 
+
+To simplify running this application, we have a single file, k8petstore.sh, which writes out json files on to disk.  This allows us to have dynamic parameters, without needing to worry about managing multiplejson files.
+
+You might want to change it to point to your customized Go image, if you chose to modify things.  
+
+like the number of data generators (more generators will create more load on the redis master).
+
+So, to run this app in kubernetes, simply run `k8petstore.sh`.
+
+## Future
+
+In the future, we plan to add cassandra support.  Redis is a fabulous in memory data store, but it is not meant for truly available and resilient storage.  
+
+Thus we plan to add another tier of queueing, which empties the REDIS transactions into a cassandra store which persists.   
+
+## Questions
+
+For questions on running this app, you can ask on the google containers group.
+
+For questions about bigpetstore, and how the data is generated, ask on the apache bigtop mailing list.
+

examples/k8petstore/Vagrantfile

+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+require 'fileutils'
+
+#$fes = 1
+#$rslavess = 1 
+
+Vagrant.configure("2") do |config|
+
+    config.vm.define "rmaster" do |rm|     
+        rm.vm.provider "docker" do |d|
+            d.vagrant_vagrantfile = "./docker-host/Vagrantfile"  
+            d.build_dir = "redis-master"
+            d.name = "rmaster"
+            d.create_args = ["--privileged=true", "-m", "1g"]
+            #d.ports = [ "6379:6379" ]
+            d.remains_running = true
+        end
+    end
+
+    config.vm.define "frontend" do |fe|
+        fe.vm.provider "docker" do |d|
+            d.vagrant_vagrantfile = "./docker-host/Vagrantfile"
+            d.build_dir = "web-server"
+            d.name = "web-server"
+            d.create_args = ["--privileged=true"]
+            d.remains_running = true
+            d.create_args = d.create_args << "--link" << "rmaster:rmaster"
+            d.ports = ["3000:3000"]
+            d.env = {"REDISMASTER_SERVICE_HOST"=>"rmaster","REDISMASTER_SERVICE_PORT"=>"6379"} 
+        end
+    end
+
+   
+ 
+    ### Todo , add data generator.
+
+
+end

examples/k8petstore/bps-data-generator/README.md

+# How to generate the bps-data-generator container #
+
+This container is maintained as part of the apache bigtop project.
+
+To create it, simply 
+
+`git clone https://github.com/apache/bigtop`
+
+and checkout the last exact version (will be updated periodically).
+
+`git checkout -b aNewBranch 2b2392bf135e9f1256bd0b930f05ae5aef8bbdcb`
+
+then, cd to bigtop-bigpetstore/bigpetstore-transaction-queue, and run the docker file, i.e. 
+
+`Docker build -t -i jayunit100/bps-transaction-queue`.

examples/k8petstore/build-push-containers.sh

+#K8PetStore version is tied to the redis version.  We will add more info to version tag later.
+#Change the 'jayunit100' string below to you're own dockerhub name and run this script.
+#It will build all the containers for this application and publish them to your dockerhub account
+version="r.2.8.19"
+docker build -t jayunit100/k8-petstore-redis:$version ./redis/
+docker build -t jayunit100/k8-petstore-redis-master:$version ./redis-master
+docker build -t jayunit100/k8-petstore-redis-slave:$version ./redis-slave
+docker build -t jayunit100/k8-petstore-web-server:$version ./web-server
+
+docker push jayunit100/k8-petstore-redis:$version
+docker push jayunit100/k8-petstore-redis-master:$version
+docker push jayunit100/k8-petstore-redis-slave:$version
+docker push jayunit100/k8-petstore-web-server:$version

examples/k8petstore/dev/README

+### Local development 
+
+1) Install Go
+
+2) Install Redis
+
+Now start a local redis instance 
+
+```
+redis-server
+```
+
+And run the app 
+
+```
+export GOPATH=~/Development/k8hacking/k8petstore/web-server/
+cd $GOPATH/src/main/
+## Now, you're in the local dir to run the app.  Go get its depenedencies.
+go get
+go run PetStoreBook.go
+```
+
+Once the app works the way you want it to, test it in the vagrant recipe below.  This will gaurantee that you're local environment isn't doing something that breaks the containers at the  versioning level.
+
+### Testing 
+
+This folder can be used by anyone interested in building and developing the k8petstore application.
+
+This is for dev and test.
+
+`vagrant up` gets you a cluster with the app's core components running.
+
+You can rename Vagrantfile_atomic to Vagrantfile if you want to try to test in atomic instead.
+
+** Now you can run the code on the kubernetes cluster with reasonable assurance that any problems you run into are not bugs in the code itself :) *

examples/k8petstore/dev/Vagrantfile

+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+require 'fileutils'
+
+#$fes = 1
+#$rslavess = 1 
+
+Vagrant.configure("2") do |config|
+
+    config.vm.define "rmaster" do |rm|     
+        rm.vm.provider "docker" do |d|
+            d.vagrant_vagrantfile = "./hosts/Vagrantfile"  
+            d.build_dir = "../redis-master"
+            d.name = "rmaster"
+            d.create_args = ["--privileged=true"]
+            #d.ports = [ "6379:6379" ]
+            d.remains_running = true
+        end
+    end
+   
+   puts "sleep 20 to make sure container is up..."
+   sleep(20)    
+   puts "resume"
+
+    config.vm.define "frontend" do |fe|
+        fe.vm.provider "docker" do |d|
+            d.vagrant_vagrantfile = "./hosts/Vagrantfile"
+            d.build_dir = "../web-server"
+            d.name = "web-server"
+            d.create_args = ["--privileged=true"]
+            d.remains_running = true
+            d.create_args = d.create_args << "--link" << "rmaster:rmaster"
+            d.ports = ["3000:3000"]
+            d.env = {"REDISMASTER_SERVICE_HOST"=>"rmaster","REDISMASTER_SERVICE_PORT"=>"6379"} 
+        end
+    end
+
+   
+ 
+    ### Todo , add data generator.
+
+
+end

examples/k8petstore/dev/hosts/Vagrantfile

+VAGRANTFILE_API_VERSION = "2"
+
+Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
+  config.vm.box = "jayunit100/centos7"
+  config.vm.provision "docker"
+  config.vm.provision "shell", inline: "ps aux | grep 'sshd:' | awk '{print $2}' | xargs kill"
+  config.vm.provision "shell", inline: "yum install -y git && service firewalld stop && service docker restart"  
+  config.vm.provision "shell", inline: "docker ps -a | awk '{print $1}' | xargs --no-run-if-empty docker rm -f || ls"
+  config.vm.network :forwarded_port, guest: 3000, host: 3000
+
+end

examples/k8petstore/dev/test.sh

+## First set up the host VM.  That ensures
+## we avoid vagrant race conditions.
+set -x 
+
+cd hosts/ 
+echo "note: the VM must be running before you try this"
+echo "if not already running, cd to hosts and run vagrant up"
+vagrant provision
+#echo "removing containers"
+#vagrant ssh -c "sudo docker rm -f $(docker ps -a -q)"
+cd ..
+
+## Now spin up the docker containers
+## these will run in the ^ host vm above.
+
+vagrant up
+
+## Finally, curl the length, it should be 3 .
+
+x=`curl localhost:3000/llen`
+
+for i in `seq 1 100` do
+    if [ x$x == "x3" ]; then 
+       echo " passed $3 "
+       exit 0
+    else
+       echo " FAIL" 
+    fi
+done
+
+exit 1 # if we get here the test obviously failed.

examples/k8petstore/k8petstore.dot

+ digraph k8petstore {
+          
+    USERS -> publicIP_proxy -> web_server;
+    bps_data_generator -> web_server [arrowhead = crow, label = "http://$FRONTEND_SERVICE_HOST:3000/rpush/k8petstore/{name..address..,product=..."];
+    external -> web_server [arrowhead = crow, label="  http://$FRONTEND_SERVICE_HOST/k8petstore/llen:3000"];
+    web_server -> redis_master [label=" RESP : k8petstore, llen"];
+    redis_master -> redis_slave [arrowhead = crow] [label="replication (one-way)"];
+}
+                                            

examples/k8petstore/k8petstore.sh

+echo "WRITING KUBE FILES , will overwrite the jsons, then testing pods. is kube clean ready to go?"
+
+VERSION="r.2.8.19"
+PUBLIC_IP="127.0.0.1" # ip which we use to access the Web server.
+SECONDS=1000          # number of seconds to measure throughput.
+FE="1"                # amount of Web server  
+LG="1"                # amount of load generators
+SLAVE="1"             # amount of redis slaves 
+
+function create { 
+
+cat << EOF > fe-rc.json
+{
+  "id": "fectrl",
+  "kind": "ReplicationController",
+  "apiVersion": "v1beta1",
+  "desiredState": {
+    "replicas": $FE,
+    "replicaSelector": {"name": "frontend"},
+    "podTemplate": {
+      "desiredState": {
+         "manifest": {
+           "version": "v1beta1",
+           "id": "frontendCcontroller",
+           "containers": [{
+             "name": "frontend-go-restapi",
+             "image": "jayunit100/k8-petstore-web-server:$VERSION",
+           }]
+         }
+       },
+       "labels": {
+         "name": "frontend",
+         "uses": "redis-master"
+       }
+      }},
+  "labels": {"name": "frontend"}
+}
+EOF
+
+cat << EOF > bps-load-gen-rc.json
+{
+  "id": "bpsloadgenrc",
+  "kind": "ReplicationController",
+  "apiVersion": "v1beta1",
+  "desiredState": {
+    "replicas": $LG,
+    "replicaSelector": {"name": "bps"},
+    "podTemplate": {
+      "desiredState": {
+         "manifest": {
+           "version": "v1beta1",
+           "id": "bpsLoadGenController",
+           "containers": [{
+             "name": "bps",
+             "image": "jayunit100/bigpetstore-load-generator",
+             "command": ["sh","-c","/opt/PetStoreLoadGenerator-1.0/bin/PetStoreLoadGenerator http://\$FRONTEND_SERVICE_HOST:3000/rpush/k8petstore/ 4 4 1000 123"]
+         }]
+         }
+       },
+       "labels": {
+         "name": "bps",
+         "uses": "frontend"
+        }
+      }},
+  "labels": {"name": "bpsLoadGenController"}
+}
+EOF
+
+cat << EOF > fe-s.json
+{
+  "id": "frontend",
+  "kind": "Service",
+  "apiVersion": "v1beta1",
+  "port": 3000,
+  "containerPort": 3000,
+  "publicIPs":["$PUBLIC_IP","$PUBLIC_IP2", "10.1.4.89","127.0.0.1","10.1.4.82"],
+  "selector": {
+    "name": "frontend"
+  },
+  "labels": {
+    "name": "frontend"
+  }
+}
+EOF
+
+cat << EOF > rm.json
+{
+  "id": "redismaster",
+  "kind": "Pod",
+  "apiVersion": "v1beta1",
+  "desiredState": {
+    "manifest": {
+      "version": "v1beta1",
+      "id": "redismaster",
+      "containers": [{
+        "name": "master",
+        "image": "jayunit100/k8-petstore-redis-master:$VERSION",
+        "ports": [{
+          "containerPort": 6379,
+          "hostPort": 6379
+        }]
+      }]
+    }
+  },
+  "labels": {
+    "name": "redis-master"
+  }
+}
+EOF
+
+cat << EOF > rm-s.json
+{
+  "id": "redismaster",
+  "kind": "Service",
+  "apiVersion": "v1beta1",
+  "port": 6379,
+  "containerPort": 6379,
+  "selector": {
+    "name": "redis-master"
+  },
+  "labels": {
+    "name": "redis-master"
+  }
+}
+EOF
+
+cat << EOF > rs-s.json
+{
+  "id": "redisslave",
+  "kind": "Service",
+  "apiVersion": "v1beta1",
+  "port": 6379,
+  "containerPort": 6379,
+  "labels": {
+    "name": "redisslave"
+  },
+  "selector": {
+    "name": "redisslave"
+  }
+}
+EOF
+
+cat << EOF > slave-rc.json
+{
+  "id": "redissc",
+  "kind": "ReplicationController",
+  "apiVersion": "v1beta1",
+  "desiredState": {
+    "replicas": $SLAVE,
+    "replicaSelector": {"name": "redisslave"},
+    "podTemplate": {
+      "desiredState": {
+         "manifest": {
+           "version": "v1beta1",
+           "id": "redissc",
+           "containers": [{
+             "name": "slave",
+             "image": "jayunit100/k8-petstore-redis-slave:$VERSION",
+             "ports": [{"containerPort": 6379, "hostPort": 6380}]
+           }]
+         }
+      },
+      "labels": {
+        "name": "redisslave",
+        "uses": "redis-master"
+      }
+    }
+  },
+  "labels": {"name": "redisslave"}
+}
+EOF
+kubectl create -f rm.json --api-version=v1beta1
+kubectl create -f rm-s.json --api-version=v1beta1
+sleep 3 # precaution to prevent fe from spinning up too soon.
+kubectl create -f slave-rc.json --api-version=v1beta1
+kubectl create -f rs-s.json --api-version=v1beta1
+sleep 3 # see above comment.
+kubectl create -f fe-rc.json --api-version=v1beta1 
+kubectl create -f fe-s.json --api-version=v1beta1
+kubectl create -f bps-load-gen-rc.json --api-version=v1beta1
+}
+
+function test { 
+	pass_http=0
+
+	### Test HTTP Server comes up.
+	for i in `seq 1 150`;
+	do
+	    ### Just testing that the front end comes up.  Not sure how to test total entries etc... (yet)
+	    echo "Trying curl ... $i . expect a few failures while pulling images... " 
+	    curl "$PUBLIC_IP:3000" > result
+	    cat result
+	    cat result | grep -q "k8-bps"
+	    if [ $? -eq 0 ]; then
+		echo "TEST PASSED after $i tries !"
+		i=1000
+		break
+	    else	
+		echo "the above RESULT didn't contain target string for trial $i"
+	    fi
+	    sleep 5
+	done
+
+	if [ $i -eq 1000 ]; then
+	   pass_http=-1
+	fi
+
+	pass_load=0 
+
+	### Print statistics of db size, every second, until $SECONDS are up.
+	for i in `seq 1 $SECONDS`;
+	do
+	    echo "curl : $i"
+            curl "$PUBLIC_IP:3000/llen" >> result
+	    sleep 1
+	done
+}
+
+create
+
+test

examples/k8petstore/redis-master/Dockerfile

+#
+# Redis Dockerfile
+#
+# https://github.com/dockerfile/redis
+#
+
+# Pull base image.
+#
+# Just a stub. 
+
+FROM jayunit100/redis:2.8.19
+
+ADD etc_redis_redis.conf /etc/redis/redis.conf
+
+CMD ["redis-server", "/etc/redis/redis.conf"]
+# Expose ports.
+EXPOSE 6379

examples/k8petstore/redis-slave/Dockerfile

+#
+# Redis Dockerfile
+#
+# https://github.com/dockerfile/redis
+#
+
+# Pull base image.
+#
+# Just a stub. 
+
+FROM jayunit100/redis:2.8.19
+
+ADD run.sh /run.sh
+
+CMD /run.sh

examples/k8petstore/redis-slave/run.sh

+#!/bin/bash
+
+# Copyright 2014 Google Inc. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+i
+echo "Note, if you get errors below indicate kubernetes env injection could be faliing..."
+echo "env vars ="
+env
+echo "CHECKING ENVS BEFORE STARTUP........"
+if [ ! "$REDISMASTER_SERVICE_HOST" ]; then
+    echo "Need to set REDIS_MASTER_SERVICE_HOST" && exit 1;
+fi
+if [ ! "$REDISMASTER_PORT" ]; then
+    echo "Need to set REDIS_MASTER_PORT" && exit 1;
+fi
+
+echo "ENV Vars look good, starting !"
+
+redis-server --slaveof ${REDISMASTER_SERVICE_HOST:-$SERVICE_HOST} $REDISMASTER_SERVICE_PORT

examples/k8petstore/redis/Dockerfile

+#
+# Redis Dockerfile
+#
+# https://github.com/dockerfile/redis
+#
+
+# Pull base image.
+FROM dockerfile/ubuntu
+
+# Install Redis.
+RUN \
+  cd /tmp && \
+  # Modify to stay at this version rather then always update.
+
+  #################################################################
+  ###################### REDIS INSTALL ############################
+  wget http://download.redis.io/releases/redis-2.8.19.tar.gz && \
+  tar xvzf redis-2.8.19.tar.gz && \
+  cd redis-2.8.19 && \
+  ################################################################
+  ################################################################
+  make && \
+  make install && \
+  cp -f src/redis-sentinel /usr/local/bin && \
+  mkdir -p /etc/redis && \
+  cp -f *.conf /etc/redis && \
+  rm -rf /tmp/redis-stable* && \
+  sed -i 's/^\(bind .*\)$/# \1/' /etc/redis/redis.conf && \
+  sed -i 's/^\(daemonize .*\)$/# \1/' /etc/redis/redis.conf && \
+  sed -i 's/^\(dir .*\)$/# \1\ndir \/data/' /etc/redis/redis.conf && \
+  sed -i 's/^\(logfile .*\)$/# \1/' /etc/redis/redis.conf
+
+# Define mountable directories.
+VOLUME ["/data"]
+
+# Define working directory.
+WORKDIR /data
+
+ADD etc_redis_redis.conf /etc/redis/redis.conf
+
+# Print redis configs and start.
+# CMD "redis-server /etc/redis/redis.conf"
+
+# Expose ports.
+EXPOSE 6379

examples/k8petstore/web-server/Dockerfile

+FROM google/golang:latest
+
+# Add source to gopath.  This is defacto required for go apps.
+ADD ./src /gopath/src/
+ADD ./static /tmp/static
+ADD ./test.sh /opt/test.sh
+RUN chmod 777 /opt/test.sh
+# $GOPATH/[src/a/b/c]
+# go build a/b/c
+# go run main
+
+# So that we can easily run and install
+WORKDIR /gopath/src/
+
+# Install the code (the executables are in the main dir)  This will get the deps also.
+RUN go get main
+#RUN go build main
+
+# Expected that you will override this in production kubernetes.
+ENV STATIC_FILES /tmp/static
+CMD /gopath/bin/main

negroni @ 1dd3ab0f

+Subproject commit 1dd3ab0ff59e13f5b73dcbf70703710aebe50d2f

redigo @ 535138d7

+Subproject commit 535138d7bcd717d6531c701ef5933d98b1866257

context @ 215affda

+Subproject commit 215affda49addc4c8ef7e2534915df2c8c35c6cd

mux @ 8096f475

+Subproject commit 8096f47503459bcc74d1f4c487b7e6e42e5746b5

simpleredis @ 5292687f

+Subproject commit 5292687f5379e01054407da44d7c4590a61fd3de

examples/k8petstore/web-server/src/main/PetStoreBook.go

+package main
+/*
+Copyright 2014 Google Inc. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+//package main
+
+import (
+    "encoding/json"
+    "net/http"
+    "os"
+    "strings"
+    "fmt"
+    "github.com/codegangsta/negroni"
+    "github.com/gorilla/mux"
+    "github.com/xyproto/simpleredis"
+)
+
+//return the path to static assets (i.e. index.html)
+func pathToStaticContents() (string) {
+    var static_content = os.Getenv("STATIC_FILES");
+    // Take a wild guess.  This will work in dev environment.
+    if static_content == "" {
+        println("*********** WARNING: DIDNT FIND ENV VAR 'STATIC_FILES', guessing your running in dev.")
+        static_content = "../../static/"
+    } else {
+        println("=========== Read ENV 'STATIC_FILES', path to assets : " + static_content);
+    }
+
+    //Die if no the static files are missing.
+    _, err := os.Stat(static_content)
+    if err != nil {
+        println("*********** os.Stat failed on " + static_content + " This means no static files are available.  Dying...");
+        os.Exit(2);
+    }
+    return static_content;
+}
+
+func main() {
+
+    var connection = os.Getenv("REDISMASTER_SERVICE_HOST")+":"+os.Getenv("REDISMASTER_SERVICE_PORT");    
+
+    if connection == ":" {
+        print("WARNING ::: If in kube, this is a failure: Missing env variable REDISMASTER_SERVICE_HOST");
+        print("WARNING ::: Attempting to connect redis localhost.")
+        connection="127.0.0.1:6379";
+    } else {
+        print("Found redis master host "+ os.Getenv("REDISMASTER_SERVICE_PORT"));
+        connection = os.Getenv("REDISMASTER_SERVICE_HOST") + ":" + os.Getenv("REDISMASTER_SERVICE_PORT");
+    }
+
+    println("Now connecting to : " + connection)
+    /**
+    *  Create a connection pool.  ?The pool pointer will otherwise
+    *  not be of any use.?https://gist.github.com/jayunit100/1d00e6d343056401ef00
+    */
+    pool = simpleredis.NewConnectionPoolHost(connection)
+
+    println("Connection pool established : " + connection)
+
+    defer pool.Close()
+
+    r := mux.NewRouter()
+
+    println("Router created ")
+
+    /**
+    * Define a REST path.
+    *  - The parameters (key) can be accessed via mux.Vars.
+    *  - The Methods (GET) will be bound to a handler function.
+    */
+    r.Path("/info").Methods("GET").HandlerFunc(InfoHandler)
+    r.Path("/lrange/{key}").Methods("GET").HandlerFunc(ListRangeHandler)
+    r.Path("/rpush/{key}/{value}").Methods("GET").HandlerFunc(ListPushHandler)
+    r.Path("/llen").Methods("GET").HandlerFunc(LLENHandler)
+
+    //for dev environment, the site is one level up...
+
+    r.PathPrefix("/").Handler(http.FileServer(http.Dir( pathToStaticContents() )))
+
+    r.Path("/env").Methods("GET").HandlerFunc(EnvHandler)
+
+    list := simpleredis.NewList(pool, "k8petstore")
+    HandleError(nil, list.Add("jayunit100"))
+    HandleError(nil, list.Add("tstclaire"))
+    HandleError(nil, list.Add("rsquared"))
+
+    // Verify that this is 3 on startup.
+    infoL := HandleError(pool.Get(0).Do("LLEN","k8petstore")).(int64) 
+    fmt.Printf("\n=========== Starting DB has %d elements \n", infoL)
+    if infoL < 3 {
+        print("Not enough entries in DB.  something is wrong w/ redis querying")
+        print(infoL)
+        panic("Failed ... ")
+    }  
+
+
+    println("===========  Now launching negroni...this might take a second...")
+    n := negroni.Classic()
+    n.UseHandler(r)
+    n.Run(":3000")
+    println("Done ! Web app is now running.")
+
+   
+   
+}
+
+/**
+* the Pool will be populated on startup,
+* it will be an instance of a connection pool.
+* Hence, we reference its address rather than copying.
+*/
+var pool *simpleredis.ConnectionPool
+
+/**
+*  REST
+*  input: key
+*
+*  Writes  all members to JSON.
+*/
+func ListRangeHandler(rw http.ResponseWriter, req *http.Request) {
+    println("ListRangeHandler")
+
+    key := mux.Vars(req)["key"]
+
+    list := simpleredis.NewList(pool, key)
+
+    //members := HandleError(list.GetAll()).([]string)
+    members := HandleError(list.GetLastN(4)).([]string)
+
+    print(members)
+    membersJSON := HandleError(json.MarshalIndent(members, "", "  ")).([]byte)
+
+    print("RETURN MEMBERS = "+string(membersJSON))
+    rw.Write(membersJSON)
+}
+
+func LLENHandler(rw http.ResponseWriter, req *http.Request) {
+    println("=========== LLEN HANDLER")
+
+    infoL := HandleError(pool.Get(0).Do("LLEN","k8petstore")).(int64)
+    fmt.Printf("=========== LLEN is %d ", infoL)
+    lengthJSON := HandleError(json.MarshalIndent(infoL, "", "  ")).([]byte)
+    fmt.Printf("================ LLEN json is %s", infoL)
+    
+    print("RETURN LEN = "+string(lengthJSON))
+    rw.Write(lengthJSON)
+
+}
+
+func ListPushHandler(rw http.ResponseWriter, req *http.Request) {
+    println("ListPushHandler")
+
+    /**
+    *  Expect a key and value as input.
+    *
+    */
+    key := mux.Vars(req)["key"]
+    value := mux.Vars(req)["value"]
+
+    println("New list " + key + " " + value)
+    list := simpleredis.NewList(pool, key)
+    HandleError(nil, list.Add(value))
+    ListRangeHandler(rw, req)
+}
+
+func InfoHandler(rw http.ResponseWriter, req *http.Request) {
+    println("InfoHandler")
+
+    info := HandleError(pool.Get(0).Do("INFO")).([]byte)
+    rw.Write(info)
+}
+
+func EnvHandler(rw http.ResponseWriter, req *http.Request) {
+    println("EnvHandler")
+
+    environment := make(map[string]string)
+    for _, item := range os.Environ() {
+        splits := strings.Split(item, "=")
+        key := splits[0]
+        val := strings.Join(splits[1:], "=")
+        environment[key] = val
+    }
+
+    envJSON := HandleError(json.MarshalIndent(environment, "", "  ")).([]byte)
+    rw.Write(envJSON)
+}
+
+func HandleError(result interface{}, err error) (r interface{}) {
+    if err != nil {
+        print("ERROR :  " + err.Error())
+        //panic(err)
+    }
+    return result
+}

examples/k8petstore/web-server/static/histogram.js

+//var data = [4, 8, 15, 16, 23, 42];
+
+function defaults(){
+
+    Chart.defaults.global.animation = false;
+
+}
+
+function f(data2) {
+
+    defaults();
+
+    // Get context with jQuery - using jQuery's .get() method.
+    var ctx = $("#myChart").get(0).getContext("2d");
+    ctx.width  = $(window).width()*1.5;
+    ctx.width  = $(window).height *.5;
+
+    // This will get the first returned node in the jQuery collection.
+    var myNewChart = new Chart(ctx);
+
+    var data = {
+        labels: Array.apply(null, Array(data2.length)).map(function (_, i) {return i;}),
+        datasets: [
+            {
+                label: "My First dataset",
+                fillColor: "rgba(220,220,220,0.2)",
+                strokeColor: "rgba(220,220,220,1)",
+                pointColor: "rgba(220,220,220,1)",
+                pointStrokeColor: "#fff",
+                pointHighlightFill: "#fff",
+                pointHighlightStroke: "rgba(220,220,220,1)",
+                data: data2
+            }
+        ]
+    };
+
+    var myLineChart = new Chart(ctx).Line(data);
+}
+

examples/k8petstore/web-server/static/index.html

+<!DOCTYPE html>
+<html lang="en">
+<script src="http://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
+<!-- d3 is used by histogram.js
+<script src="http://d3js.org/d3.v3.min.js"  charset="utf-8"></script>
+-->
+<script src="https://raw.githubusercontent.com/nnnick/Chart.js/master/Chart.js"></script>
+<script src="script.js"></script>
+<script src="histogram.js"></script>
+
+  <head>
+    <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
+    <meta charset="utf-8">
+    <meta content="width=device-width" name="viewport">
+    <link href="/style.css" rel="stylesheet">
+    <title>((( - PRODUCTION -))) Guestbook</title>
+  </head>
+  <body>
+
+    <TABLE>
+        <TR>
+            <TD>
+                <div id="k8petstore-entries">
+                    <p>Waiting for database connection...This will get overwritten...</p>
+                </div>
+            </TD>
+
+            <TD>
+                <div id="header">
+                    <h1>k8-bps.</h1>
+                </div><br>
+                <div>
+                    <p><h2 id="k8petstore-host-address"></h2></p>
+                    <p><a href="/env">/env</a>
+                        <a href="/info">/info</a></p>
+                </div>
+            </TD>
+        </TR>
+        <TR >
+            <TD colspan="2">
+                <canvas id="myChart" width="2000" height="600"></canvas>
+            </TD>
+        </TR>
+    </TABLE>
+  </body>
+
+</html>

examples/k8petstore/web-server/static/script.js

+$(document).ready(function() {
+
+  var max_trials=1000
+
+  var headerTitleElement = $("#header h1");
+  var entriesElement = $("#k8petstore-entries");
+  var hostAddressElement = $("#k8petstore-host-address");
+  var currentEntries = []
+
+  var updateEntryCount = function(data, trial) {
+      if(currentEntries.length > 1000)
+        currentEntries.splice(0,100);
+      //console.info("entry count " + data) ;
+      currentEntries[trial]=data ;
+  }
+
+  var updateEntries = function(data) {
+    entriesElement.empty();
+      //console.info("data - > " + Math.random())
+      //uncommend for debugging...
+      //entriesElement.append("<br><br> CURRENT TIME :  "+ $.now() +"<br><br>TOTAL entries :  "+ JSON.stringify(currentEntries)+"<br><br>")
+      var c1 = currentEntries[currentEntries.length-1]
+      var c2 = currentEntries[currentEntries.length-2]
+      entriesElement.append("<br><br> CURRENT TIME :  "+ $.now() +"<br><br>TOTAL entries :  "+ c1 +"<BR>transaction delta " + (c1-c2) +"<br><br>")
+      f(currentEntries);
+    $.each(data, function(key, val) {
+        //console.info(key + " -> " +val);
+        entriesElement.append("<p>" + key + " " + val.substr(0,50) + val.substr(100,150) + "</p>");
+    });
+
+  }
+
+  // colors = purple, blue, red, green, yellow
+  var colors = ["#549", "#18d", "#d31", "#2a4", "#db1"];
+  var randomColor = colors[Math.floor(5 * Math.random())];
+  (
+      function setElementsColor(color) {
+          headerTitleElement.css("color", color);
+      })
+
+        (randomColor);
+
+  hostAddressElement.append(document.URL);
+
+  // Poll every second.
+  (function fetchGuestbook() {
+
+      // Get JSON by running the query, and append
+    $.getJSON("lrange/k8petstore").done(updateEntries).always(
+      function() {
+        setTimeout(fetchGuestbook, 2000);
+      });
+  })();
+
+    (function fetchLength(trial) {
+        $.getJSON("llen").done(
+            function a(llen1){
+                updateEntryCount(llen1, trial)
+            }).always(
+                function() {
+                    // This function is run every 2 seconds.
+                    setTimeout(
+                        function(){
+                            trial+=1 ;
+                            fetchLength(trial);
+                            f();
+                        }, 5000);
+                }
+            )
+    })(0);
+});
+

examples/k8petstore/web-server/static/style.css

+body, input {
+  color: #123;
+  font-family: "Gill Sans", sans-serif;
+}
+
+div {
+  overflow: hidden;
+  padding: 1em 0;
+  position: relative;
+  text-align: center;
+}
+
+h1, h2, p, input, a {
+  font-weight: 300;
+  margin: 0;
+}
+
+h1 {
+  color: #BDB76B;
+  font-size: 3.5em;
+}
+
+h2 {
+  color: #999;
+}
+
+form {
+  margin: 0 auto;
+  max-width: 50em;
+  text-align: center;
+}
+
+input {
+  border: 0;
+  border-radius: 1000px;
+  box-shadow: inset 0 0 0 2px #BDB76B;
+  display: inline;
+  font-size: 1.5em;
+  margin-bottom: 1em;
+  outline: none;
+  padding: .5em 5%;
+  width: 55%;
+}
+
+form a {
+  background: #BDB76B;
+  border: 0;
+  border-radius: 1000px;
+  color: #FFF;
+  font-size: 1.25em;
+  font-weight: 400;
+  padding: .75em 2em;
+  text-decoration: none;
+  text-transform: uppercase;
+  white-space: normal;
+}
+
+p {
+  font-size: 1.5em;
+  line-height: 1.5;
+}
+.chart div {
+    font: 10px sans-serif;
+    background-color: steelblue;
+    text-align: right;
+    padding: 3px;
+    margin: 1px;
+    color: white;
+}

examples/k8petstore/web-server/test.sh

+echo "start test of frontend"
+curl localhost:3000/llen
+curl localhost:3000/llen
+curl localhost:3000/llen
+curl localhost:3000/llen
+curl localhost:3000/llen
+curl localhost:3000/llen
+x=`curl localhost:3000/llen`
+echo "done testing frontend result = $x"