All About CRAM: the CREATE Real-time Application Manager

The CREATE Real-time Applications Manager (CRAM) is a framework for developing, deploying, and managing distributed realtime software. It has evolved through three implementations over the space of five years. The background of CRAM is the work done since the early 1990s on distributed processing environments (DPEs), which started in the telecommunications industry (see the section on background below).  

This document presents an overview of distributed processing and software  environments to support it, and describes the CRAM architecture and implementation. The intended audience is application developers who wish to use CRAM to manage their systems.

Introductory Scenario

Distributed applications are software programs where the components (services) run on different computers. Often, this corresponds to the client/server design pattern. As an initial scenario to motivate the need for distributed real-time systems, imagine a multi-computer music performance application where one computer is reading data from one or more input devices (MIDI devices, head trackers, computer vision systems, etc.) and mapping this controller data onto concrete parameters for a given set of synthesis programs. We can call this program the input server. In our scenario, another computer is running a synthesis server, which takes commands from the input server, and performs software sound synthesis, sending its output over the network to some an output server. The output server is a program that reads sound sample blocks (coming in via the network) from one or more synthesis servers and mixes and spatializes them.

The problem now arises of how we are to start and stop this application in a controlled manner. One could do it manually; logging in to the various machines to start the output server, then the synthesis server, then the input server. To better manage distributed applications, one needs software tools that can remotely start, stop, and monitor software on several computers connected by a network. This is the task of distributed processing environments (DPEs).

The Parts of a DPE

A DPE generally consist of at least three components: a node manager, a service interface, and a system manager. The node manager is a simple daemon (a stand-alone program) that is assumed to be running on each computer that the DPE intends to manage. Node managers accept commands from the network application manager to start/stop/monitor remote services. Their simplest role is as a "remote execution" server. A DPE service interface is a simple set of functions that applications need to implement in order to be managed by a DPE. This functionality is normally packaged as a software class that a developer includes in an application. The service interface functions generally include basic start/stop methods, and some sort of status query; they are used by the node manager to control the service. The third component is the system manager; it uses node managers to start the components of a distributed application. DPE systems often use databases to describe network hardware facilities and applications.

In CRAM, the node manager is a small program written in C++ that uses a simple socket-based protocol to talk to its services. The system manager uses the same protocol to communicate withy node managers. The service interface component that is incorporated into application programs includes code that implements this protocol, and starts a "listener" thread when the application starts. This thread waits for commands from the node manager to control the application.

There are two features that are often considered optional in DPEs, but are central to CRAM: fault-tolerance and load-balancing. For applications that require robust software (e.g., musical performance), the system must be able to identify and recover from a hardware or software fault within a small number of seconds. For large-scale systems that are to handle dynamic processing and I/O loads (e.g., musical performance), some manner of planning-time as well as run-time load-balancing is also necessary. We will discuss these features more below.

Using CRAM

To start and manage a distributed application using CRAM, we first assume that the network nodes (computers) are known, and that we have access to a database with information about them. We also assume that  node manager daemon programs are running on the nodes of the network. Lastly, we assume that the software we want to use is installed on the computers, or on a file server to which the network nodes have access.

We describe applications as collections of services running on nodes. A service is just an application program, written in any arbitrary language, that implements the CRAM service interface functions (including the command listener thread). The three services (input, synthesis, output) that we introduced in the scenario above are candidate CRAM services.

To manage the example music performance application using CRAM, we need to store a description of it in the CRAM database. This simply means that we define which service is to run on which node. If we have computers named waltz, jerk, and belly (all of our computers are named after dances), then we would send the following SQL (structured query language) command to the database:

insert into applications (name, services) values ('SirenCSL',  
	'{ "waltz.input_server", "courante.synthesis_server", "fire.output_server" }' );

Once this is done, we can start up the CRAM system manager. When it starts, it loads the database tables that describe nodes on the network, types of services and their options, and applications. We can use the system manager to make sure that the application will run (i.e., that the nodes are all on and have node managers running on them), and then tell it to start the application. When we do this, the system manager sends messages to each of the node managers requesting that then create and initialize the services described in the database. Once these are all ready, the application is started by sending "start" messages to the services.

While the application is running, we can use the system manager to monitor its status.

-------------------------------------------------------------------------------------------------------

The LifeCycle of a CRAM Service
	Creation -- constructor of the service object
	Initialization-- opptional message
	Start
	Stop
	Suspend (optional)
	Resume (optional)
	

The LifeCycle of a CRAM Node Manager

The LifeCycle of a CRAM System Manager

The LifeCycle of a CRAM Application

Writing CRAM Applications -- The server interface

Example: the clock service

CRAM database tables
	Networks
	Nodes
	Servicetypes
	Applications

-------------------------------------------------------------------------------------------------------

CRAM System Manager Reference

The CRAM system manager is a Smalltalk application; to start it, use the shell script startCRAM that's usually installed in /usr/local/CRAM/bin. It takes a few seconds to start up and load its database tables. Once it's running, you will see the standard two-pane system manager window as shown in the figure below.

Figure: The CRAM System Manager Window

The upper pane of the window is the CRAM tree view; the top-level items in the tree are the lists of known networks and  applications. Below this pane is the log/control pane.

The tree view shows several kinds of objects; the figure below shows a CRAM tree with several items expanded. In it, you can see the full hierarchy of CRAM.

Figure: Expanded CRAM System Tree

In the full system tree, one sees the dual hierarchies of (1) networks, nodes and services, and (2) applications and services.

Tree pop-up menus

Log view

Control view

Hierarchy
	Service on a Node
	Node manager service
	System manager
	CRAM databases

Example services

-------------------------------------------------------------------------------------------------------

Installing CRAM on a network

Database
NodeManagers

-------------------------------------------------------------------------------------------------------

INA/TINA-C DPE Background

The first set of INA specifications, known as the Cycle 1 Specifications For Information Network Architecture was released by Bellcore in June 1992, with a revised version being released in April 1993. An international research initiative set up in 1993 and known as the telecommunications information network architecture consortium (TINA-C) has developed and extended the INA framework. TINA-C defines a DPE with many of the features of CRAM. 

http://www.tinac.com
http://hegel.ittc.ukans.edu/projects/tina-c
http://www.nm.informatik.uni-muenchen.de/Vorlesungen/ws9900/tks/teil15.pdf
http://www.hpl.hp.com/hpjournal/96oct/oct96a2.pdf
http://www.item.ntnu.no/fag/SIE5050/HTML/pensum/Distributed%20Processing%20Platform.doc
http://www.comsoc.org/livepubs/surveys/public/1q00issue/berndt.html

ION DPE -- S. Bannerman, B. Stockdell, and M. Stutz, "Network Topology Configuration Management Experience Report," Proc. TINA'99, Turtle Bay, Hawaii, Apr. 1999, pp. 137­39.

How is CRAM different from traditional INA-based DPEs?

-------------------------------------------------------------------------------------------------------

Fault-tolerance in CRAM

Service hang-ups
NodeManager death/restart/service discovery
System Manager death/restart/node discovery

-------------------------------------------------------------------------------------------------------

Implementation Notes

A very simple packet protocol spoken between the system manager and a node manager, based on connection-less UDP sockets

Safety and exception handling

Platform dependencies
	POSIX pthreads package (pthread_kill())
	select system call
	Unix-style file I/O
	Log and PID files and names
	setsockopt arguments (Solaris)
	var_args and stdio

