GeoServer Replication

ScaleOut Software NamedCache API

This topic contains the following sections:

With version 3.0, ScaleOut StateServer® introduced the ScaleOut GeoServer® option to extend distributed caching across multiple, geographically distributed data centers. This option replicates stored objects between ScaleOut StateServer stores running on server farms at different sites. The original GeoServer approach accomplishes replication by asynchronously pushing changes across WAN links as soon as objects are updated. For example, a datacenter in New York City can continuously push its updates to Los Angeles, allowing the L.A. site to remain fully synchronized with NYC at every moment. Push replication is primarily targeted at assisting sites with disaster recovery by ensuring that stored objects are held at multiple geographic sites in case one of the sites becomes unavailable.

Version 5.0 of ScaleOut GeoServer introduced a second replication model that allows a remote site to access ("pull") objects from the local site as they are needed. Pull-based replication allows objects to be shared by a geographically diverse network of ScaleOut StateServer stores, with policies on each object dictating how frequently remote datacenters should update their local copies of the object. Furthermore, the authoritative "master" copy of an object can migrate from datacenter to datacenter as needed. This new replication model enables multiple sites to be combined into a single, "virtual" distributed data grid spanning multiple sites, providing transparent, global access to stored objects.

Version 5.10 of ScaleOut GeoServer® Pro introduces an extension to the "notify" coherency policy for pull replication across two ScaleOut StateServer stores that combines synchronized access to objects with push replication. This extension ensures that two copies of each object are always maintained, one in each store, while giving applications a single, coherent view of the object's data. ScaleOut GeoServer Pro automatically handles WAN outages to ensure uninterrupted operations. (See the User Guide for more information.)

The following sections explain the use of push replication and pull replication in detail.

Push Replication

GeoServer "push" replication quickly and efficiently replicates changes to stored data at a remote ScaleOut StateServer (SOSS) store in order to allow for uninterrupted availability after a site-wide failure. This option is configured to replicate the following changes from a local store to one or more remote stores:

  • new objects that are created on the local SOSS store,

  • objects or metadata that are updated on the store, and

  • objects that are removed from the store,

  • refreshes to sliding object timeouts due to read accesses.

The GeoServer option is configured to replicate objects from a local SOSS store to a remote SOSS store in one direction only. If the GeoServer option is also licensed on a remote SOSS store, it can be separately configured to replicate objects to the local store. For example, to configure GeoServer to replicate objects bi-directionally between sites A and B, it must be configured on site A to replicate objects to site B and then configured on site B to replicate objects to site A.

Note Note

To avoid circular replication, GeoServer option does not allow cascading data replication (for example, replicating objects from site A to B and then forwarding site A's objects to site C). To replicate objects from site A to sites B and C, configure site A to replicate objects directly to both sites B and C.

Configuration

To configure push replication:

  • Ensure that a secure, reliable, and fast network exists to the remote site. The network must be fast enough to handle the expected replication traffic so that excess memory is not consumed on the local store to buffer outbound requests. The available bandwidth should match or exceed the local store's aggregate access rate.

  • On the remote SOSS store, configure a unique gateway IP address and gateway server port on all SOSS hosts. The SOSS Console's Host Configuration tab can be used to set the gateway configuration. The IP address/port pair must be reachable from the local store using a TCP connection, and this pair must map to the SOSS server port on the remote host.

    Note Note

    The gateway IP address may be on a different subnet from that used for the remote store's network interface. Also, if a router is used at the remote store and port mapping is employed, the gateway port may not match the SOSS server port's value and should be configured as the value that the router expects external TCP connections to use.

  • On the local SOSS store, add a remote store to the GeoServer configuration using either the SOSS Console or the geos.exe command line program. To add a remote store, enter one of the remote store's gateway addresses in order to establish communications with the remote store. You optionally can add additional gateway addresses for other SOSS hosts in the remote store to ensure that the remote store initially can be reached in case one or more hosts are offline. Be sure to select the access mode Push.

    Note Note

    The GeoServer configuration for remote stores is recorded in the client parameters file, soss_client_params.txt, which is stored in ScaleOut StateServer's installation directory on every host of the local store. Versioning information is kept in the file to ensure that all hosts always have the latest updates to the file. You should not directly edit this file; please use the management tools instead.

  • Ensure that the remote store has been made active by joining all hosts.

  • Test communications with the remote store by using the Test command in SOSS Console or geos.exe. (The remote store must be active to test communications. However, the local store may be inactive.)

  • If communication with the remote store is successful, you can use the Populate command to download the gateway addresses from all active hosts on the remote store and save them in the local store's client parameters file. This ensures that you can establish communications with the remote store if any of these gateway addresses respond. (Gateway addresses for offline hosts are ignored during replication.)

    Note Note

    Gateway addresses are permanently stored in the client parameters file. If you know that a gateway address is no longer used at the remote store, you should remove it from the local store's client parameters file using the management tools.

  • Ensure that the local store has been made active by joining all hosts.

  • Start replication of object updates to the remote store using either the Start or Sync command. The Start command begins ongoing replication of object updates, and the Sync command first replicates all locally stored objects before beginning replication of object updates.

  • Verify that replication is proceeding normally by examining the replication status using the SOSS Console or geos.exe.

Caching API Usage

Once push replication has been configured, all objects in the local SOSS store will be subject to replication by default; every create, update, and delete operation performed on an object will be sent across the WAN connection to the remote datacenter(s) in order to keep objects the stores synchronized. In addition, refreshes to sliding object timeouts are replicated when objects are retrieved on the local store; these refreshes are grouped to minimize WAN traffic.

If desired, developers can disable replication for individual objects using the APIs. Use the CreatePolicyAllowReplication property to specify whether an object should be subject to replication as the object is created.

Disabling Push Replication for an Object
NamedCache nc = CacheFactory.GetCache();

CreatePolicy policy = new CreatePolicy();
policy.AllowReplication = false;

nc.Insert("key1", "Object value 1", policy, false, false);

The NamedCache's DefaultCreatePolicy property can be used to prevent replication for all objects in a named cache.

Disabling Push Replication for a Named Cache
NamedCache nc = CacheFactory.GetCache();
nc.DefaultCreatePolicy.AllowReplication = false;

nc.Add("key2", "Object value 2");
Pull Replication

Introduced in version 5.0 of ScaleOut StateServer, GeoServer pull replication differs from push replication in that objects are replicated across a WAN from a remote SOSS store to a local SOSS store as the objects are retrieved by the application. The local copy of an object which has been replicated from a remote SOSS store is called a proxy, and the original object on the remote store is called a master. The proxy is invisible to an application running on the local store in the sense that it is accessed using the same SOSS key as the master and is automatically refreshed by SOSS; a proxy object serves only as a locally cached copy of its corresponding master object.

To maintain efficient use of the WAN, a proxy object is allowed to become out of date with changes made to its associated master object based on a coherency policy specified by the application for the object. The coherency policy determines how often the proxy is automatically refreshed by SOSS. A loose coherency policy can result in reduced bandwidth usage between sites since fewer updates are sent across the link between datacenters. The developer can adjust the coherency policy based on the type of data stored in the object. Data that is seldom updated or allowed to become out-of-date for a short period of time, such as product descriptions on a Web site, can benefit from using a loose coherency policy. (The use of full coherency is described below.)

Unlocked updates performed on a local proxy of a replicated object automatically are not pushed across the WAN back to the remote SOSS store containing the master copy of the object. Note that unlocked updates should seldom be performed because the order of updates from multiple threads running at different sites generally cannot be predicted. Hence, updates could be lost if the application does not synchronize updates across multiple sites. Data for which loose coherency policies are appropriate usually are updated only at the master.

Some types of data, such as transactional data (for example, shopping carts for a Web site), must be kept fully coherent across all SOSS stores. Otherwise, updates to these objects could be lost. To efficiently handle this situation, SOSS pull replication can automatically migrate the master copy of an object from a remote SOSS store to the local store which is requesting exclusive access by locking the object. The remote SOSS store relinquishes ownership if an application on the local SOSS store acquires a lock on the object using NamedCacheAcquireLock or performs a locking NamedCacheRetrieve call.

Because the mastership of an object can migrate from store to store, it is common for GeoServer pull replication to be configured in both directions (for example, Los Angeles would be configured to pull from New York, and New York would be configured to pull from Los Angeles). This allows the master copy of an object to migrate from datacenter to datacenter as needed. GeoServer can also manage pull replication across more than two sites—for example, if three datacenters are involved, then each site will need to be separately configured to pull from the other two. If a local site does not lock any of its replicated (proxy) objects, then the corresponding master objects always remain at their current remote site(s), and bi-directional replication is not necessary.

Bi-directional replication also needs to be configured in situations in which the NotifyCoherencyPolicy is used to keep proxy objects synchronized. See the section on Creating the Original Object to learn more about coherency policies.

Configuration

Unlike push replication, pull replication is enabled on the store that will be accessing objects from a remote store. (Push replication is enabled on the store that holds the original copy of the object.) To configure pull replication for a local store so that it can access objects from a remote store:

  • Ensure that a secure, reliable, and fast network exists between stores. The network must be fast enough to handle the expected request rate. Otherwise, remote access to objects could be delayed and local proxies could become stale.

  • On the remote SOSS store that contains the master objects to be accessed, give the store a unique name from either the SOSS Console’s "Store Info" tab or the geos.exe command line program:

  • On the remote SOSS store that contains the master objects to be accessed, configure a unique gateway IP address and gateway server port on all SOSS hosts. The SOSS Console’s Host Configuration tab can be used to set the gateway configuration. The IP address/port pair must be reachable from the local store using a TCP connection, and this pair must map to the SOSS server port on the remote host.

    Note Note

    The gateway IP address may be on a different subnet from that used for the remote store's network interface. Also, if a router is used at the local datacenter and port mapping is employed, the gateway port may not match the SOSS server port's value and should be configured as the value that the router expects external TCP connections to use.

  • On the local SOSS store, add a store to the GeoServer configuration using either the SOSS Console or the geos.exe command line program. To add a store, enter one of the remote store's gateway addresses in order to establish communications. The name of the store must match the name that was configured earlier back on the remote store. You can also optionally add additional gateway addresses for other SOSS hosts in the remote store to enable connectivity in case one or more hosts are offline. Be sure to select the access mode Pull.

    Note Note

    The GeoServer configuration for remote stores is recorded in the client parameters file, soss_client_params.txt, which is stored in ScaleOut StateServer's installation directory on every host of the local store. Versioning information is kept in the file to ensure that all hosts always have the latest updates to the file. You should not directly edit this file; please use the management tools instead.

  • Ensure that the remote store has been made active by joining all hosts.

  • Test communications with the remote store by using the Test command in SOSS Console or geos.exe. (The remote store must be active to test communications.)

  • If communication with the remote store is successful, you can use the Populate command to download the gateway addresses from all active hosts on the remote store and save them in the client parameters file. This ensures that you can establish communications with the remote store if any of these gateway addresses respond. (Gateway addresses for offline hosts are ignored during replication.)

    Note Note

    Gateway addresses are permanently stored in the client parameters file. If you know that a gateway address is no longer used at the remote store, you should remove it from the local store's client parameters file using the management tools.

  • Ensure that both stores have been made active by joining all hosts at both sites.

  • Begin allowing remote access to the master objects in the remote store using the Start command.

  • Repeat the above steps to establish replication in the opposite direction, if necessary. bi-directional pull replication should be configured in the following circumstances:

Verify that replication is proceeding normally by examining the replication status using the SOSS Console or geos.exe.

Caching API Usage

GeoServer pull replication is designed to be as transparent to the application as possible. Applications access objects as if they were all held in the local SOSS store. Should SOSS not find an object in the local store, it searches remote stores as necessary to locate the missing object. If it finds the object in a remote store, SOSS reads the object across the WAN and stores a copy of the object on the local store as a proxy. All local accesses to the object are automatically handled by SOSS using the proxy object, which is kept coherent with the remote store's master copy using the coherency policy specified by the application for that object.

Local reads to a proxy object optionally can bypass the local proxy and directly read the remote master copy. In this case, the local proxy is automatically refreshed with the latest data when the read operation completes.

If the master copy of an object migrates to another remote store, SOSS automatically tracks this change so that proxy objects can be kept coherent with the master. When the master copy is removed by the application, SOSS removes remote proxy objects when coherency checks are made. If an application accessing a local proxy object removes the object, the master copy is also removed; this helps keep the use of proxy objects transparent to the application.

The local store automatically takes over ownership of an object when a locking retrieve or lock request is made on the object. In this case, the local proxy is converted to the master copy of the object (and the remote master copy is demoted to a proxy). SOSS implements this transfer of ownership in anticipation that the application will update (and unlock) the object and it helps to make synchronized updates efficient by avoiding WAN usage. Should an application repeatedly ping pong locking reads and updates between stores, high WAN usage cannot be avoided.

In addition to using the standard named cache APIs to access objects, two additional steps are needed to prepare for pull replication:

  • When an object that may be subject to pull replication is created, its coherency policy must be configured when it is initially inserted into the store.

  • Prior to accessing an object from a remote store, the list of remote stores to be searched (and the search order) must be specified. This list is provided separately for each named cache using the NamedCacheSetRemoteStores method; it tells the named cache which remote stores to contact whenever a retrieve request is made for an object that isn't present locally.

The following sections describe in more detail how the named cache APIs are used to implement pull replication.

Creating the Original Object

The CreatePolicy class is used to specify how SOSS should treat an object that is being inserted in the distributed store; it allows a client application to specify and object’s timeout and dependencies, among other properties. The CreatePolicy class contains two new properties to support GeoServer pull replication. When creating an object that is to be made accessible to remote stores, the CreatePolicyAllowRemoteAccess property must be set to true, and the CreatePolicyDefaultCoherencyPolicy property must be set to an implementation of ICoherencyPolicy.

Creating an Object Eligible for Pull Replication
NamedCache nc = CacheFactory.GetCache("GeoServer pull sample");

CreatePolicy policy = new CreatePolicy();
policy.AllowRemoteAccess = true;

// remotes sites should poll the master object every 30 seconds for updates:
policy.DefaultCoherencyPolicy = new PollingCoherencyPolicy(TimeSpan.FromSeconds(30));

nc.Insert("key1", "Object value 1", policy, true, false);

There are currently two supported implementations of ICoherencyPolicy available for use that may be assigned to the CreatePolicyDefaultCoherencyPolicy property:

  • PollingCoherencyPolicy: Indicates that proxy objects in remote stores should poll the master object for updates every time the specified coherency interval expires. This policy optimizes network traffic. Note that SOSS transparently provides remote polling based on this policy. If remote reads experience delays or access failures, SOSS marks the local proxy as stale, and local reads may return stale data (and optionally throw an exception) until the WAN access issue is resolved. Bi-directional replication is not required when using the PollingCoherencyPolicy.

  • NotifyCoherencyPolicy: Indicates that the proxy object in the remote store should be notified every time the master object is updated. If a WAN access error occurs when reading a remote master object, the local proxy object will automatically be promoted to a master to enable uninterrupted access. (See the User Guide for details.) This is effectively per-object "push" replication between two stores combined with synchronized access using object locking. Note that bi-directional replication must be configured to use the NotifyCoherencyPolicy.

  • Null: Indicates that the proxy object should not be automatically refreshed by SOSS. The application should directly read the master copy as necessary to implement an application-specific coherency policy.

Note Note

The ICoherencyPolicy interface exists to support the ScaleOut API infrastructure, so application developers should not create their own custom implementations of ICoherencyPolicy.

To configure replication for all objects created in a NamedCache, the AllowRemoteAccess and DefaultCoherencyPolicy properties can be configured on NamedCacheDefaultCreatePolicy.

Configuring Pull Replication for an Entire Named Cache
NamedCache nc = CacheFactory.GetCache("GeoServer pull sample");

nc.DefaultCreatePolicy.AllowRemoteAccess = true;
nc.DefaultCreatePolicy.DefaultCoherencyPolicy = new PollingCoherencyPolicy(TimeSpan.FromSeconds(30));

nc.Add("key2", "Object value 2");

Reading an Object from a Remote Store

Preparing the NamedCache

Once an object has been created with a coherency policy and is marked to allow remote access, the object becomes available for reading and updating by remote stores. Client applications that need to remotely access an object must first call the NamedCacheSetRemoteStores method and pass in an ordered collection of remotes store names. This list of names specifies the set of stores and search order that will be used to find an object when a read operation is performed and the requested key is not found in the local store.

Preparing the NamedCache for RemoteReads
NamedCache nc = CacheFactory.GetCache("GeoServer pull sample");

RemoteStoreCollection remStores = new RemoteStoreCollection();
remStores.Add(new RemoteStore("LosAngeles"));
remStores.Add(new RemoteStore("Paris"));

nc.SetRemoteStores(remStores);
Note Note

The store names specified in the collection must match the names used in the GeoServer configuration. Stores must be named consistently across datacenters when specifying local store names and remote store names in the SOSS Console and in API calls. For example, a store should not be called "LosAngeles" in one datacenter and referred to as "LA" in another.

Passing null or an empty collection into a SetRemoteStores call disables pull replication for that instance of the NamedCache.

Performing the Retrieval

Once the search order has been set using NamedCacheSetRemoteStores, the SOSS service checks the specified stores for objects when a call to the NamedCacheRetrieve method (or a call to NamedCacheGetMetadata) is performed on a key that is not present in the local store. The local store goes through the specified RemoteStoreCollection and contacts each store in turn until it is able to locate the desired object. If none of the stores contain the object, the retrieve call returns null, indicating that the object cannot be found. If a WAN access error occurs when attempting to access one of the remote stores in the list, an exception is thrown.

Once the local store retrieves a remote object, it creates a local proxy copy of the object and then returns the value to the calling application. Subsequent retrieve calls return the proxy copy's data, which is refreshed according to the coherency policy that was set up on the master object when it was originally created (see the Creating the Original Object section).

The following diagram illustrates how objects created at the "LosAngeles" store are retrieved by the "NewYork" store:

The behavior of the client APIs can be adjusted on a call-by-call basis when reading an object that might be remotely accessed. The RetrieveOptions parameter that is passed into a retrieve call has a GeoServerReadMode property that controls API behavior when reading these objects. This enumeration presents four options:

None

Indicates that GeoServer replication is not used with this cache. (This is the default value.) Note that the effect of this option is identical to LocalDoNotThrowOnStaleData.

Remote

Indicates that the caller has configured GeoServer for "pull" replication and wants to force the local store to read the object from the remote store (rather than reading from a local proxy of the object if it should exist). This option guarantees that the caller will always retrieve the latest value for the object even if a WAN request is required.

LocalDoNotThrowOnStaleData

Indicates that the retrieve operation can return the object value from the proxy held in the local ScaleOut store if it should exist and no exception will be thrown if the local proxy value is stale.

LocalThrowOnStaleData

Indicates that the retrieve operation can return the object's value from the local proxy if it should exist. However, if the local proxy value is stale (which can occur if one of the remotes store has a WAN access error), the Retrieve/GetMetadata call will throw a StaleDataException.

Using RetrieveOptions
// Configure access to any remote stores. This step is typically
// performed once at application startup:
NamedCache nc = CacheFactory.GetCache("GeoServer pull sample");
RemoteStoreCollection remStores = new RemoteStoreCollection();
remStores.Add(new RemoteStore("LosAngeles"));
nc.SetRemoteStores(remStores);

// Set up options for retrieval. We want to get the value back from the
// proxy of the object, if there's one stored locally:
RetrieveOptions options = new RetrieveOptions();
options.GeoServerReadMode = GeoServerReadMode.LocalDoNotThrowOnStaleData;
options.LockingMode = ReadLockingMode.NoLockOnRead;

// Perform the retrieval:
string myObj = nc.Retrieve("key1", options) as string;

The NamedCacheGetMetadata call also has an overload that takes a GetMetadataOptions argument, which is the metadata corollary to the RetrieveOptions struct. A NamedCache.GetMetadata call behaves in exactly the same way as normal Retrieve call and will result in the local store contacting remote stores for missing objects and the creation of local proxy objects.

Using GetMetadataOptions
GetMetadataOptions options = new GetMetadataOptions();
options.GeoServerReadMode = GeoServerReadMode.LocalDoNotThrowOnStaleData;
options.LockingMode = ReadLockingMode.NoLockOnRead;

// Perform the GetMetadata call:
nc.GetMetadata("key1", options);

The GeoServerReadMode option can also be configured for all read calls that are performed by an instance of the NamedCache. Read operations that do not take a RetrieveOptions/GetMetadataOptions parameter will default to the behavior specified by the NamedCacheDefaultGeoServerReadMode property. The NameCache.DefaultGeoServerReadMode property defaults to LocalDoNotThrowOnStaleData.

Setting the GeoServerReadMode for an Entire Named Cache
NamedCache nc = CacheFactory.GetCache("GeoServer pull sample");
RemoteStoreCollection remStores = new RemoteStoreCollection();
remStores.Add(new RemoteStore("LosAngeles"));
nc.SetRemoteStores(remStores);

// Specify that all read calls should default to forcing remote reads,
// unless specified otherwise on a call-by-call basis:
nc.DefaultGeoServerReadMode = GeoServerReadMode.Remote;

// Using indexer syntax or a Get() call will honor the DefaultGeoServerReadMode
// property set above, so we force a remote read with this call instead of hitting
// the local proxy:
string myObj1 = nc["key1"] as string;

// We can still override the DefaultGeoServerReadMode on a call-by-call basis
// by passing explicit options into a Retrieve or GetMetadata call:
RetrieveOptions options = new RetrieveOptions();
options.GeoServerReadMode = GeoServerReadMode.LocalDoNotThrowOnStaleData;
string myObj2 = nc.Retrieve("key2", options) as string;

Locking and Updating an Object from a Remote Store

Locking an object in ScaleOut StateServer is a common operation that can be performed when a thread in a client application needs to acquire exclusive access to an object. ScaleOut StateServer supports distributed locks, so when an application acquires a lock on an object, no other application code--regardless of whether it’s running in a different thread, process, or machine--will be allowed to acquire a lock on that same object.

ScaleOut GeoServer's pull replication architecture extends the distributed locking model to span datacenters, allowing a single thread on a single machine to acquire a lock so that no other client code across a globally distributed network of caches will be allowed to acquire a lock on that same object.

Pull replication achieves this global locking behavior by allowing the master copy of an object to migrate from store to store. Locking an object using the NamedCacheAcquireLock method (or performing a Retrieve call with the LockOnRead parameter set) from a local store causes the remote store holding the master copy of the object to relinquish control, and the requesting local store then assumes mastership, at which point the lock is acquired for the caller.

The following diagram illustrates the sequence of events that occur when an object created on the "LosAngeles" store is locked by an application running on the "NewYork" store:

Applications that use locking in conjunction with pull replication must take care to establish bi-directional replication when configuring GeoServer so that all stores involved in replication have the ability to access an object and take over mastership when acquiring a lock on the object.