Pond HOWTO

The OceanStore Project
HOWTO run Pond, the OceanStore prototype

This page contains information to help you configure and run OceanStore using the tools in the run directory. I've probably left out something very important; email me if you have any trouble.

Contents:

Using run-experiment
Other Tools

Using run-experiment
The main script used to start up OceanStore nodes is run-experiment, found in the run directory. Most of our benchmark scripts rely on run-experiment, so the information on this page applies more or less directly to them as well. This script takes an experiment file as input. That file describes what sort of OceanStore nodes to create and where to run them; we'll describe that file below.

run-experiment can configure and run hundreds of virtual OceanStore nodes on dozens of remote machines. It coordinates sharing information across config files, pushing data to remote sites, monitoring remote processes, and cleaning up all sites afterward.

Most of the real work is handled by a shared Perl module, Tools.pm. This library is somewhat efficient and rather sophisticated Perl, so please ask me before making changes to it yourself.

You usually must set a few environment variables before calling run-experiment:

Set OSTORE_SERVERS to the space-separated list of hosts on which you want to run. You can skip this step if your experiment file explicitly lists hosts, which most do not.
Set REMOTE_OSTORE_RUN_DIR to the full path to the run directory on the remote machines. You may skip this step if it's in the same place as the local machine.
Set REMOTE_OSTORE_USER to your login on the remote machines if it is different from your local username.

There are several other useful variables you might want to try:

Set OSTORE_DISABLE_COPYBACK to prevent run-experiment from copying logs over from remote machines on shutdown. That step can be a little slow, and is unnecessary if you don't want to look at the logs. Simply defining the variable will disable the copyback only on failure. Setting it to a positive number will disable copyback in all cases.
Set OSTORE_TMP and/or REMOTE_OSTORE_TMP if you want to change the directory into which run-experiment writes its files. The default is usually /tmp or /stripe. You might want to change it if the default directory is too small.
Set KEYSHARE_PREFIX to change the set of threshold keys used by the inner ring. The default is "ir". The only other option currently is "ir-1024-4-2".
Set OSTORE_SMART_TOOLS if you want to run an experiment across both the local cluster and planetlab. It's a pretty humble hack, so only use it if you understand what it does. Basically, it disables the REMOTE_OSTORE_RUN_DIR and REMOTE_OSTORE_USER for cluster machines, so that you can set those variables to their planetlab equivalents. If the defaults on the cluster work for you, then everything just works.
Set OSTORE_CFG_DIR and/or OSTORE_GRAPH_DIR if you understand what they mean (mostly just Patrick).

Experiment File
The files cluttering up the run directory that end in .exp are all experiment files. There are four main sections to an experiment file. Each is described below. Any text on a line following a '#' character is considered a comment, and is ignored.

Host specification
The first line of the experiment file lists the hosts on which to run the OceanStore nodes. It may be omitted. You can use hostnames, IP addresses, or the keyword "ENV", which tells run-experiment to use the contents of the OSTORE_SERVERS environment variable.
Format: hosts < host list >
Default: hosts ENV
Example: hosts localhost
Example: hosts x2 localhost foo.cs.berkeley.edu

Topology specification
Specify a topology file if you want the OceanStore network layer to emulate an artificial topology. This line may be omitted. The keyword "NONE" disables this feature. You may use the variable

Virtual machine specification
The meat of the experiment file--these lines describe the OceanStore nodes that run-experiment will run. ${graph_dir} in the filename. It generally evaluates to run/graphs.
Format: < tag > < config > < tapestry mode > < exit type > [ host index [ topology index ] ]

Tag: The unique name for this node. Usually something like "RP", or "Replica3". A few tags ("Federation", "RP", "Ring0", "Ring1", "Ring2", "Ring3") are somewhat magical because they control Tapestry startup and inner ring keys. They should be used verbatim for the appropriate OceanStore nodes. The rest of the tags are arbitrary.
Config: The configuration file template to use for the node. We will describe it more below . You may use the ${run_dir} variable in its filename.
Tapestry Mode: One of "fed", "static", or "dynamic". Only the "Federation"-tagged machine can be "fed". The rest are either "static" or "dynamic", depending on whether they should be started with static tapestry or whether they should wait and integrate themselves into tapestry dynamically. Configuration files that include a StaticTClient stage should not be used on dynamic nodes.
Exit type: One of "daemon" or "mortal". run-experiment expects the former to run until it is killed. It will wait on mortal nodes until all have exited, at which point it will shut down the daemons, too.
Host index (optional): an index into the zero-based list specified on the hosts line. If this field is omitted, run-experiment will use the next host on the list, wrapping around if necessary.
Topology index (optional): a line index (zero-based) into the topology file. This field assigns the OceanStore node to a node in the artificial topology. It can be omitted if and only if the topology specifictation above is "NONE". If not omitted, the host index must also be present.

The following example starts up the first inner ring machine on the 3rd (index 2) host, and does not specify a topology index. Example: Ring0 ${run_dir}/inner.cfg static daemon 2
Parameter definitions
You can override configuration parameters on a per-experiment basis. The parameters are set before interpolating the template variables, so you can insert a literal value or a variable name.
Format: set < tag.stage.parameter > < value >
- Tag: The tag can be one of the tags specified above, some prefix followed by a splat ('*'), like "Ring*", or just a splat. The splats allow a very limited globbing.
- Stage: One of the stage names in the configuration file, or "global", if the parameter should be set in the global initargs.
- Parameter: the name of the parameter to set
- Value: the value to assign, verbatim. Remember variables will be interpolated later.
Example: set Ring*.Inner.block_size 256
Example: set Client-23.DTree.nodes_to_avoid ${Ring3ID}

Node Configuration File
Each OceanStore node needs a SandStorm configuration file. We use template files that allow us to fill in (interpolate) useful variables at run-time. These variables make it easy to share templates across similar nodes, share information among nodes that are supposed to communicate (e.g. the RP and client nodes), move experiments easily among environments (e.g. from localhost to the cluster to planetlab).

The format is pretty simple XML, but we don't have a DTD written up. Each file includes a header with global information, like thread management, queue thresholds, and local node ID. The rest of the file is a list of stage entries. Each names a stage, the Java class file to run for that stage, and possibly some initialization parameters.

Most configuration templates are copied from an existing file and modified slightly. To minimize the horrible damage caused by copy and paste, I've implemented hierarchical components for configuration file. The new generation of configuration files are mostly a list of other files to include; these included files describe commonly used stages, and are shared by several SandStorm configs. Take a look at run/*inner*cfg to see what I mean.

It is not required that you use the new style. You can still inline all your header and stage definitions if you wish; however, I will refuse to help you if you have problems and I think they might be caused by slightly-inconsistent stage parameters. We already have enough copy-and-paste in our code. It's been hurting our configuration as well.

Variable Interpolation
There are a large number of variables you can use when writing configuration templates. Just put ${Variable} where you want the value of < Variable >. The braces are required.

Most variables are already in use somewhere, and you can learn their use by example. I'll give a short list here; it will probably be out of date by the time you read this, though. Look at examples, and ask me if you think we need to add something.

ID variables: There is a < Tag > ID for every tag in the experiment file. Also, NodeID evaluates to the node's own ID, and GatewayID evaluates to the ID for the first "static" tapestry node.
Examples: ${FederationID}, ${Client-23ID}.
Key variables: Like the ID variables above, but with the suffixes PublicKey and PrivateKey instead of ID. Important: run-experiment assigns keys to nodes in the experiment file from the set of all keys found in run/keys, on a first-come, first-serve basis. The specific key assigned to a node should be transparent, if it uses NodePublicKey to refer to its own key, and if all other nodes use < tag > PublicKey, and no one tries to use filenames directly.
Examples: ${RPPublicKey}, ${NodePrivateKey}.
Data Keys: variables for the keys used to sign updates and responses. ${IRPublicKey}, ${IRPrivateKeyShares} ${PrincipalPublicKey} ${PrincipalPrivateKey}
${Index}: If the node's tag ends in a number (e.g. Ring3), that number is available as the ${Index} variable. It might be useful for differentiating among otherwise-identical nodes.
${TapestryMode}: the value of the tapestry mode flag for this node in the experiment file.
Count variables: ${RingSize}, ${NumberStatic}: the number of nodes whose tag starts with "Ring", or whose tapestry mode is "static", respectively.
Temporary filenames: ${InternalLog} is the filename to use (at least as prefix) for any files that the node will create on the remote host and that should be copied back on shutdown. ${CacheDir} is the directory into which the cache should write its files. ${SSProfileFile} and ${SSGraphFile} are the output files for SandStorm's internal instrumentation, which we generally don't use.
Others can be added on request.

Other Tools
There are a few other scripts in run that help working on remote machines. They are particularly useful for running experiments on planetlab.

fetch-planetlab-list.pl reads two files from the planet-lab.org website and prints out a list of available planetlab hosts. This script is very simple and does not perform any resource allocation. It reads the list of production machines and blackballs any machines for which InfoSpect has detected errors. You can also manually blackball machines by adding their IPs to the OSTORE_BLACKLIST environment variable. You can disable the InfoSpect check using the --noblacklist option.
This script prints out a definition of the OSTORE_SERVERS environment variable. To use it, just paste that line into your shell.
Example: OSTORE_BLACKLIST="206.117.37.4 206.117.37.5" ./fetch-planetlab-list.pl.
call_iterator.pl scripts.
Several other scripts (fetch_host_keys, call_remote.pl, copy_remote.pl, remote_pull.pl, killall_java_remote, rm_remote_experiment) use OSTORE_SERVERS and REMOTE_OSTORE_USER to perform some function on several remote machines.
These scripts are pretty self-explanatory, and each has a nice usage printout. Be sure to try the --parallel option, which causes the scripts to operate on several remote hosts simultaneously, in the background. It really helps.

Last modified on Sat Jan 11 17:31:53 PST 2003 by geels.