Web Server – Parker Gordon | Software Engineering

Overview

Zookeeper is a distributed sequentially consistent system developed to attack the many tough use cases surrounding distributed systems such as leader election in a cluster, configuration, and distributed locking. For more Zookeeper recipes visit: http://zookeeper.apache.org/doc/current/recipes.html. Zookeeper clusters(ensembles) can be made of of any number of nodes, but typically take the form of a three or five node ensemble with the minority of nodes able to fail and the Zookeeper service able to continue serving traffic.

How we use Zookeeper

On the Search team at Careerbuilder we use Zookeeper as a configuration service replacing static configuration file deployment. We also have plans to move to Solr Cloud, which requires a Zookeeper Ensemble for its election and configuration tasks. Below is a system diagram for the entire Zookeeper/Exhibitor service in AWS with an auto-scaling group and consuming client libraries.

Difficulties with Zookeeper

While Zookeeper provides a high level of reliability and availability through redundancy and leader election patterns, it’s critical use cases introduce a high level of risk for outages and mass failures. To mitigate a lot of the risk, the open source Exhibitor application should be used as a supervisory process to monitor and restart each Zookeeper process, as well as provide data backups, recovery, and auto ensemble configuration. Exhibitor will be explained in more detail later.

Exhibitor: https://github.com/soabase/exhibitor/wiki.

Assumptions

Deployment will take advantage of AWS resources (Currently avoiding containerization)
Exhibitor will be deployed alongside Zookeeper to server as it’s supervisory process and management UI (https://github.com/soabase/exhibitor/wiki)

Deployment

Infrastructure as Code: Frameworks

Infrastructure automation can be achieved through many open and closed source framework providers such as Ansible, Puppet, and Chef. For deployments on the Search team at Careerbuilder we use Chef as the means for setting up environments on self hosted boxes in AWS. The Chef + Ruby combo is a robust tool in managing infrastructure as code and I highly suggest it: https://learn.chef.io/.

Artifacts

To deploy a Zookeeper ensemble the following artifacts and dependencies are required to be installed on each node in the cluster. After each artifact is built they can be uploaded to an artifact repository, but in our case an S3 bucket through a continuous integration build, to be later downloaded and installed by Chef during deployment.

JAVA
- Zookeeper and Exhibitor are JAVA Virtual Machine(JVM) applications therefore JAVA must be installed on all machines running them.
Zookeeper
- Download and Install from https://zookeeper.apache.org/releases.html
Exhibitor
- Exhibitor can be run as a WAR file or with an embedded Jetty Server. After finding the Tomcat/WAR based server cumbersome we chose the Jetty based Exhibitor moving forward and it has paid off through it’s ease of configuration.
- Steps on how to build the Exhibitor artifact can be found on Github: https://github.com/soabase/exhibitor/wiki/Building-Exhibitor

Cloud Formation

On a higher level of Infrastructure as code lies the ability to orchestrate resource allocation, bringup, and system integration. AWS CloudFormation is a means to perform these deployment actions internal to AWS.

Generic Templates

By generalizing CloudFormation template files for your specific use cases and parameterizing them, you can use a homegrown application or Integration processes to regularly phoenix or A-B deploy your resources with ease.

Here at Careerbuilder we have developed an in-house JAVA application known as Nimbus that pulls our generic CF Stack templates from Github, populates their parameters with values from parameter files and triggers a CloudFormation Stack creation in AWS. This abstracts a lot of CloudFormation’s unused Stack template complexity.

Cluster Discovery through Instance Tagging

In many cases it is required that you define machine instances indexes to each machine in a clustered distributed system either due to dataset sharding or static clustering. It is possible to deploy Zookeeper with exhibitor in a static ensemble that does not utilize a shared S3 exhibitor.properties configuration file(more on this later) and this would require instance tagging. Each node could be tagged with an attribute, say {“application”=”zookeeper-development”} by CloudFormation to be used as a selector during bring up. The Chef/Ansible/Puppet processes running on each node could then acquire the IP’s of each of the nodes in the ensemble by going through a query-check-sleep cycle in the local chef script until the correct number of Zookeeper boxes are up and running. However, this deployment model is clumsy and error prone, therefore the shared configuration file in S3 for exhibitor is highly suggested, as server IPs are added and removed from the config file dynamically.

Starting Exhibitor and Zookeeper

Exhibitor will start and restart the Zookeeper processes periodically during clustering tasks and rolling configuration changes. Therefore, all you need to do is start Exhibitor on each node, each node will starte Zookeeper, then register itself with the shared config file specified by the “–s3config” parameter. Below is some example ruby/chef code that can be used to start Exhibitor:

execute “Start Exhibitor” do
command “nohup java -jar #{node[:search][:exhibitor_dir]}/exhibitor.jar –hostname #{node[‘ipaddress’]} –configtype s3 –s3config #{bucket} –s3backup true > /opt/search/zookeeper/exhibitorNohup.log 2> /opt/search/zookeeper/exhibitorNohup.error.log &”
action :run
end

Since Exhibitor manages the Zookeeper process it is important not to create any configuration files manually nor expect any manual configurations made to Zookeeper to persists after deployment or during runtime.

Exhibitor

Security

Securing Exhibitor/Zookeeper is an extensive topic left to the specific implementation of the reader. However, the Exhibitor Wiki lists command parameters that can be used to enable and configure security features within Exhibitor and suggest giving it a look.

https://github.com/soabase/exhibitor/wiki/Running-Exhibitor

Configuration

A typical exhibitor bucket root will look like the following, with an exhibitor.properties config file that you have uploaded containing a default level of configuration. I suggest also maintaining a ‘last known default’ config file in a “base-config” directory for backup of the exhibitor.properties file, in the event it gets corrupted.

An example exhibitor.properties file:

com.netflix.exhibitor-hostnames=
com.netflix.exhibitor-hostnames-index=0
com.netflix.exhibitor.auto-manage-instances-apply-all-at-once=1
com.netflix.exhibitor.auto-manage-instances-fixed-ensemble-size=5
com.netflix.exhibitor.auto-manage-instances-settling-period-ms=60000
com.netflix.exhibitor.auto-manage-instances=1com.netflix.exhibitor.backup-extra=
com.netflix.exhibitor.backup-max-store-ms=86400000
com.netflix.exhibitor.backup-period-ms=60000
com.netflix.exhibitor.check-ms=30000
com.netflix.exhibitor.cleanup-max-files=3
com.netflix.exhibitor.cleanup-period-ms=43200000
com.netflix.exhibitor.client-port=2181
com.netflix.exhibitor.connect-port=2888
com.netflix.exhibitor.election-port=3888
com.netflix.exhibitor.java-environment=
com.netflix.exhibitor.log-index-directory=/opt/search/zookeeper/logIndex/
com.netflix.exhibitor.log4j-properties=
com.netflix.exhibitor.observer-threshold=999
com.netflix.exhibitor.servers-spec=
com.netflix.exhibitor.zoo-cfg-extra=syncLimit\=5&tickTime\=2000&initLimit\=10
com.netflix.exhibitor.zookeeper-data-directory=/opt/search/zookeeper_data
com.netflix.exhibitor.zookeeper-install-directory=/opt/search/zookeeper
com.netflix.exhibitor.zookeeper-log-directory=/opt/search/zookeeper

Pro Tip: A great benefit we have found is the ability to modify Zookeeper Java Environment settings through Exhibitor for performance and Garbage Collection tuning.

Ensemble Registration

The following images outline the steps of the consensus process during Ensemble deployment.

Self Healing

The following images outline the event of a node failure in an auto-managed ensemble.

Recently, I was tasked with developing a load generation tool on top of Twitter’s open source Iago project. I initially validated the request rates of the app using a separate local Play! app as a victim server with restful endpoints summing the requests. But.. this setup wasn’t going to cut it within my acceptance test suite.

Solution: Embedded Jetty Server.

Goal

Programmatically stand up a web server with minimal restful endpoints/routes as concise as possible.

Overview

For this example our Jetty server will act as a “Counter” and have the following characteristics:

Listen on a random port
Two restful endpoints
- /increment
  - Will increment the counter by one and return the new count in the response body
- /reset
  - Will reset the counter back to zero

embeddedjettyserver — Embedded Jetty Server Component Diagram

Prerequisites

An SBT/Maven based Scala Project
- Scala version 2.11.8

Note: Setting up a project is out of the scope of this article

Steps:

1. Add Project Dependencies

Jetty has changed ownership a few times(currently Eclipse) and you can get Jetty working in a number of ways, but this example assumes the latest Jetty release from Eclipse.

Maven

	<dependency>
	<groupId>org.eclipse.jetty</groupId>
	<artifactId>jetty-server</artifactId>
	<version>9.3.12.v20160915</version>
	</dependency>
	<dependency>
	<groupId>org.eclipse.jetty</groupId>
	<artifactId>jetty-servlet</artifactId>
	<version>9.3.12.v20160915</version>
	</dependency>

view raw pom.xml hosted with ❤ by GitHub

build.sbt

	libraryDependencies ++= Seq(
	"org.eclipse.jetty" % "jetty-servlet" % "9.3.12.v20160915",
	"org.eclipse.jetty" % "jetty-server" % "9.3.12.v20160915"
	)

view raw build.sbt hosted with ❤ by GitHub

2. Create a Jetty manager Object

First thing we’ll do is create an Object that will contain the Jetty Server, configuration variables, and the Servlets (hold the route definitions). For this example I named it JettyExample.

Since the server will have two functions:

Incrementing the counter
Resetting the counter

we should define the string literals for those routes.

	object JettyExample {
	val incrementRoute = "/increment"
	val resetRoute = "/reset"

	def main(args: Array[String]) = {
	}
	}

view raw

JettyExample.scala

hosted with ❤ by GitHub

3. Add Helper Functions

Add a createServer() function and assign an internally global server variable to a createServer() call. The Server object will need an import reference as well.

Tip: You could add action methods similar to createServer() that Start or Stop the server, or execute handler functionality programmatically rather than through web requests.

	import org.eclipse.jetty.server.Server

	object JettyExample {
	val server = createServer()
	{…}

	def createServer() = new Server(0) // 0 for random port
	}

view raw

JettyExample.scala

hosted with ❤ by GitHub

Since the server was told to start on a random port, we need a function that grabs this port from the server instance. We will use this to print the port number later. Also, add the import reference to NetworkConnector.

	import org.eclipse.jetty.server.{NetworkConnector, Server}

	object JettyExample{
	{…}
	def port() = {
	val conn = server.getConnectors()(0).asInstanceOf[NetworkConnector]
	conn.getLocalPort()
	}
	}

view raw

JettyExample.scala

hosted with ❤ by GitHub

4. Create the Servlets

Servlet Container

CREATE THE SERVLETS! Wait, wait, wait…. first lets create a Servlet container object inside our JettyExample that will encapsulate our private counter variable as well as our endpoint Servlets. Don’t forget we need a Thread-Safe variable to eliminate race conditions.

	object JettyExample{
	{…}

	object CounterServlets{
	private var requestCount: Int = AtomicInteger(0) // encapsulate the state in a Thread safe way
	// TODO: Servlet Classes to go here
	}
	}

view raw

JettyExample.scala

hosted with ❤ by GitHub

IncrementServlet

Add the local requestCount variable, and the Servlet that handles the increment logic then returns html containing the new counter value. This Servlet will serve GET request types. Also, you must include the three imports required for defining an HttpServlet.

	import javax.servlet.http.{HttpServlet, HttpServletRequest, HttpServletResponse}

	object JettyExample{
	{…}
	object CounterServlets{
	private var requestCount: Int = AtomicInteger(0)

	class IncrementServlet extends HttpServlet {
	override protected def doGet(request: HttpServletRequest, response: HttpServletResponse):Unit = {
	requestCount.getAndIncrement()

	response.setContentType("text/html")
	response.setStatus(HttpServletResponse.SC_OK)
	response.getWriter().println(s"<h2>Increment performed. Count is now $requestCount.</h2>")
	}
	}
	}
	}

view raw

JettyExample.scala

hosted with ❤ by GitHub

ResetServlet

Add another HttpServlet titled “ResetServlet”. This Servlet will reset the counter, and return an OK message and HTML stating the counter has been reset.

At this point the Servlet Classes are defined, but they are not bound to the server with a ServletHandler as a route definition. We’ll do that next.

	{…}
	object JettyExample{
	{…}
	object CounterServlets{
	{…}
	class ResetServlet extends HttpServlet {
	override protected def doGet(request: HttpServletRequest, response: HttpServletResponse):Unit = {
	requestCount.getAndIncrement()

	response.setContentType("text/html")
	response.setStatus(HttpServletResponse.SC_OK)
	response.getWriter().println(s"<h2>Counter reset to 0.</h2>")
	}
	}
	}
	}

view raw

JettyExample.scala

hosted with ❤ by GitHub

5. Bind the Servlets with a ServletHandler

Before adding the handler bindings add an import declaration for the CountingServlets internal to the JettyExample so we can have access to the Servlet Classes.

Now add a ServletHandler variable to the JettyExample scope. Within the main method assign this handler to the current server instance, and add the increment Servlet to the server after being mapped to the route “/increment”. Do the same with the reset Servlet. ServletHandler requires one import.

Both endpoints are now configured, the last thing to do is Start the server, and have the server wait to terminate.

	import org.eclipse.jetty.servlet.ServletHandler

	object JettyExample {
	{…}
	import CounterServlets._

	val incrementRoute = "/increment"
	val resetRoute = "/reset"
	val handler = new ServletHandler()

	def main(args: Array[String]) = {
	server.setHandler(handler)
	handler.addServletWithMapping(classOf[IncrementServlet], incrementRoute)
	handler.addServletWithMapping(classOf[ResetServlet], resetRoute)
	}
	}

view raw

JettyExample.scala

hosted with ❤ by GitHub

6. Start the Server

Fire it up! Start the server and have it wait for termination. We also should print out some diagnostic info about the port so we can hit the endpoint from a browser or other HTTP client.

Note: server.join() causes the main thread to wait to continue util the server is fully up, and will cause the main thread to wait to terminate until the Jetty thread terminates.

	{…}
	object JettyExample {
	{…}
	val server = createServer()

	def main(args: Array[String]) = {
	{…}
	server.start()
	println(s"Server started on ${port()} with routes: '$incrementRoute'")
	server.join()
	}
	}

view raw

JettyExample.scala

hosted with ❤ by GitHub

When you start the server you’ll see a terminal printout similar to:

In a browser, go to http://localhost:{your port}/increment. Then refresh the page a few times. You should see something similar to the following:

To reset the count go to http://localhost:{your port}/reset . You should see

Complete Code

	import org.eclipse.jetty.servlet.ServletHandler
	import org.eclipse.jetty.server.{NetworkConnector, Server}
	import javax.servlet.http.{HttpServlet, HttpServletRequest, HttpServletResponse}

	object JettyExample {
	import CounterServlets._

	val incrementRoute = "/increment"
	val resetRoute = "/reset"
	val server = createServer()
	val handler = new ServletHandler()

	def main(args: Array[String]) ={
	server.setHandler(handler)
	handler.addServletWithMapping(classOf[IncrementServlet], incrementRoute)
	handler.addServletWithMapping(classOf[ResetServlet], resetRoute)

	server.start()
	println(s"Server started on ${port()} with endpoints: '$incrementRoute' and '$resetRoute'")
	server.join()
	}

	def port() = {
	val conn = server.getConnectors()(0).asInstanceOf[NetworkConnector]
	conn.getLocalPort()
	}

	def createServer() = new Server(0)

	object CounterServlets{
	private var requestCount: Int = AtomicInteger(0)

	class IncrementServlet extends HttpServlet {
	override protected def doGet(request: HttpServletRequest, response: HttpServletResponse):Unit = {
	requestCount.getAndIncrement() // Thread-Safe Increment

	response.setContentType("text/html")
	response.setStatus(HttpServletResponse.SC_OK)
	response.getWriter().println(s"<h2>Increment received. Count is now $requestCount.</h2>")
	}
	}

	class ResetServlet extends HttpServlet {
	override protected def doGet(request: HttpServletRequest, response: HttpServletResponse):Unit = {
	requestCount.getAndIncrement() // Thread-Safe Increment

	response.setContentType("text/html")
	response.setStatus(HttpServletResponse.SC_OK)
	response.getWriter().println(s"<h2>Counter reset to 0.</h2>")
	}
	}
	}
	}

view raw

JettyExample.scala

hosted with ❤ by GitHub

Resources:

Maven Jetty Server Repo

Twitter’s Iago | Load Generation for Engineers

Overview

How we use Zookeeper

Difficulties with Zookeeper

Assumptions

Deployment

Infrastructure as Code: Frameworks

Artifacts

JAVA

Zookeeper

Exhibitor

Cloud Formation

Generic Templates

Cluster Discovery through Instance Tagging

Starting Exhibitor and Zookeeper

Exhibitor

Security

Configuration

Ensemble Registration

Self Healing

Share this post:

Like this:

Goal

Overview

Prerequisites

Steps:

1. Add Project Dependencies

Maven

build.sbt

2. Create a Jetty manager Object

3. Add Helper Functions

4. Create the Servlets

Servlet Container

IncrementServlet

ResetServlet

5. Bind the Servlets with a ServletHandler

6. Start the Server

Complete Code

Resources:

Share this post:

Like this: