com.continuent.tungsten.replicator.thl
Class THLParallelQueue

java.lang.Object
  com.continuent.tungsten.replicator.thl.THLParallelQueue

All Implemented Interfaces:

ReplicatorPlugin, ParallelStore, Store

public class THLParallelQueue
extends java.lang.Object
implements ParallelStore

Implements a parallel event store based on on-disk queues. On-disk queues work by managing a set of threads that read from the THL in parallel on behalf of the next stage and populate queues for each applier task. This class is responsible for setting up the task threads as well as the queues they feed, ensuring that queues do not get too far apart when executing, and to handle control events including getting apply tasks to commit their restart position both at regular intervals as well as prior to clean shutdown.

Applier tasks are known as "channels" in replicator end-user documentation.

This class makes a very strong assumption that shard IDs are correctly assigned in prior stages. If not, parallelization may fail due to conflicts when transactions are assigned to incorrect channels. This in turn can lead to apply failing on slaves in such a way that the slave must be resynchronized with its master to continue.

Version:

1.0

Author:

Robert Hodges

Constructor Summary

THLParallelQueue()

Method Summary

void	configure(PluginContext context) Complete plug-in configuration.
ReplEvent	get(int taskId) Removes and returns next event from the queue, blocking if empty.
long	getHeadSeqno() Returns the current head seqno to which read tasks may advance.
AtomicIntervalGuard<?>	getIntervalGuard() Returns the interval guard structure that tracks positions of channels.
int	getMaxDelayInterval()
int	getMaxOfflineInterval() Returns the maximum number of seconds to do a clean shutdown.
int	getMaxSize() Maximum size of individual queues.
long	getMaxStoredSeqno() Returns the maximum persistently stored sequence number.
ReplDBMSHeader	getMinLastHeader() Returns the minimum last header processed in order to handle restart correctly.
long	getMinStoredSeqno() Returns the minimum persistently stored sequence number.
java.lang.String	getName() Gets the storage name.
Partitioner	getPartitioner()
java.lang.String	getPartitionerClass() Returns the class used for partitioning transactions across queues.
int	getPartitions() Returns the number of partitions for events.
int	getSyncInterval() Returns the number of events between sync intervals.
void	insertStopEvent() Inserts stop control event after next complete transaction.
void	insertWatchSyncEvent(WatchPredicate<ReplDBMSHeader> predicate) Inserts watch synchronization event after next complete transaction that matches the provided predicate.
ReplEvent	peek(int taskId) Returns next event from the queue without removing it, returning null if queue is empty.
void	prepare(PluginContext context) Prepare plug-in for use.
void	put(int taskId, ReplDBMSEvent event) Puts an event in the queue, blocking if it is full.
void	release(PluginContext context) Release queue.
void	setLastHeader(int taskId, ReplDBMSHeader header) Sets the last header processed.
void	setMaxDelayInterval(int maxDelayInterval) Sets the maximum number of seconds to delay before allowing transactions to continue even when they would cause maxOfflineInterval to be exceeded.
void	setMaxOfflineInterval(int maxOfflineInterval) Sets the maximum number of seconds for a clean shutdown.
void	setMaxSize(int size) Returns the maximum size of individual queues.
void	setName(java.lang.String name) Sets the storage name.
void	setPartitioner(Partitioner partitioner) Sets the instance used to assign events to partitions (channels).
void	setPartitionerClass(java.lang.String partitionerClass) Sets the name of the class used to assign events to partitions (channels).
void	setPartitions(int partitions) Sets the number of queue partitions.
void	setSyncInterval(int syncInterval) Sets the number of events to process before generating an automatic control event if sync is enabled.
int	size(int taskId) Returns the current queue size.
void	start(int taskId) Start the reader for a particular task.
TungstenProperties	status() Returns status information as a set of named properties.
void	stop(int taskId) Stop the reader for a particular task.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

THLParallelQueue

public THLParallelQueue()

Method Detail

getName

public java.lang.String getName()

Description copied from interface: Store

Gets the storage name.

Specified by:

getName in interface Store

setName

public void setName(java.lang.String name)

Description copied from interface: Store

Sets the storage name.

Specified by:

setName in interface Store

getMaxSize

public int getMaxSize()

Maximum size of individual queues.

setMaxSize

public void setMaxSize(int size)

Description copied from interface: ParallelStore

Returns the maximum size of individual queues.

Specified by:

setMaxSize in interface ParallelStore

setPartitions

public void setPartitions(int partitions)

Sets the number of queue partitions.

Specified by:

setPartitions in interface ParallelStore

getPartitions

public int getPartitions()

Returns the number of partitions for events.

Specified by:

getPartitions in interface ParallelStore

getPartitioner

public Partitioner getPartitioner()

setPartitioner

public void setPartitioner(Partitioner partitioner)

Sets the instance used to assign events to partitions (channels).

getPartitionerClass

public java.lang.String getPartitionerClass()

Description copied from interface: ParallelStore

Returns the class used for partitioning transactions across queues.

Specified by:

getPartitionerClass in interface ParallelStore

setPartitionerClass

public void setPartitionerClass(java.lang.String partitionerClass)

Sets the name of the class used to assign events to partitions (channels).

Specified by:

setPartitionerClass in interface ParallelStore

getSyncInterval

public int getSyncInterval()

Returns the number of events between sync intervals.

Specified by:

getSyncInterval in interface ParallelStore

setSyncInterval

public void setSyncInterval(int syncInterval)

Sets the number of events to process before generating an automatic control event if sync is enabled.

Specified by:

setSyncInterval in interface ParallelStore

getMaxOfflineInterval

public int getMaxOfflineInterval()

Returns the maximum number of seconds to do a clean shutdown.

Specified by:

getMaxOfflineInterval in interface ParallelStore

setMaxOfflineInterval

public void setMaxOfflineInterval(int maxOfflineInterval)

Sets the maximum number of seconds for a clean shutdown. This is maintained by keeping the THL read tasks from getting too far apart from each other.

Specified by:

setMaxOfflineInterval in interface ParallelStore

getMaxDelayInterval

public int getMaxDelayInterval()

setMaxDelayInterval

public void setMaxDelayInterval(int maxDelayInterval)

Sets the maximum number of seconds to delay before allowing transactions to continue even when they would cause maxOfflineInterval to be exceeded.

getHeadSeqno

public long getHeadSeqno()

Returns the current head seqno to which read tasks may advance.

getIntervalGuard

public AtomicIntervalGuard<?> getIntervalGuard()

Returns the interval guard structure that tracks positions of channels.

setLastHeader

public void setLastHeader(int taskId,
                          ReplDBMSHeader header)
                   throws ReplicatorException

Sets the last header processed. This is required for restart.

Throws:

ReplicatorException

getMinLastHeader

public ReplDBMSHeader getMinLastHeader()
                                throws ReplicatorException

Returns the minimum last header processed in order to handle restart correctly.

Throws:

ReplicatorException

getMaxStoredSeqno

public long getMaxStoredSeqno()

Returns the maximum persistently stored sequence number.

Specified by:

getMaxStoredSeqno in interface Store

See Also:

Store.getMaxStoredSeqno()

getMinStoredSeqno

public long getMinStoredSeqno()

Returns the minimum persistently stored sequence number.

Specified by:

getMinStoredSeqno in interface Store

See Also:

Store.getMinStoredSeqno()

put

public void put(int taskId,
                ReplDBMSEvent event)
         throws java.lang.InterruptedException,
                ReplicatorException

Puts an event in the queue, blocking if it is full.

Throws:

java.lang.InterruptedException

ReplicatorException

get

public ReplEvent get(int taskId)
              throws java.lang.InterruptedException,
                     ReplicatorException

Removes and returns next event from the queue, blocking if empty.

Throws:

java.lang.InterruptedException

ReplicatorException

peek

public ReplEvent peek(int taskId)
               throws java.lang.InterruptedException,
                      ReplicatorException

Returns next event from the queue without removing it, returning null if queue is empty.

Throws:

java.lang.InterruptedException

ReplicatorException

size

public int size(int taskId)

Returns the current queue size.

configure

public void configure(PluginContext context)
               throws ReplicatorException

Complete plug-in configuration. This is called after setters are invoked at the time that the replicator goes through configuration.

Specified by:

configure in interface ReplicatorPlugin

Throws:

ReplicatorException - Thrown if configuration is incomplete or fails

See Also:

ReplicatorPlugin.configure(com.continuent.tungsten.replicator.plugin.PluginContext)

prepare

public void prepare(PluginContext context)
             throws ReplicatorException,
                    java.lang.InterruptedException

Prepare plug-in for use. This method is assumed to allocate all required resources. It is called before the plug-in performs any operations.

Specified by:

prepare in interface ReplicatorPlugin

Throws:

ReplicatorException - Thrown if resource allocation fails

java.lang.InterruptedException

See Also:

ReplicatorPlugin.prepare(com.continuent.tungsten.replicator.plugin.PluginContext)

release

public void release(PluginContext context)
             throws ReplicatorException

Release queue. Release all resources used by plug-in. This is called before the plug-in is deallocated.

Specified by:

release in interface ReplicatorPlugin

Throws:

ReplicatorException - Thrown if resource deallocation fails

See Also:

ReplicatorPlugin.release(com.continuent.tungsten.replicator.plugin.PluginContext)

start

public void start(int taskId)

Start the reader for a particular task.

stop

public void stop(int taskId)

Stop the reader for a particular task.

insertStopEvent

public void insertStopEvent()
                     throws java.lang.InterruptedException

Inserts stop control event after next complete transaction.

Specified by:

insertStopEvent in interface ParallelStore

Throws:

java.lang.InterruptedException

See Also:

ParallelStore.insertStopEvent()

insertWatchSyncEvent

public void insertWatchSyncEvent(WatchPredicate<ReplDBMSHeader> predicate)
                          throws java.lang.InterruptedException

Inserts watch synchronization event after next complete transaction that matches the provided predicate.

Specified by:

insertWatchSyncEvent in interface ParallelStore

Throws:

java.lang.InterruptedException

See Also:

ParallelStore.insertWatchSyncEvent(com.continuent.tungsten.replicator.util.WatchPredicate)

status

public TungstenProperties status()

Returns status information as a set of named properties.

Specified by:

status in interface Store

See Also:

Store.status()