com.continuent.tungsten.replicator.pipeline
Class Pipeline

java.lang.Object
  com.continuent.tungsten.replicator.pipeline.Pipeline

All Implemented Interfaces:

ReplicatorPlugin

public class Pipeline
extends java.lang.Object
implements ReplicatorPlugin

Stores the information related to a replication pipeline, which is a set of independent processing stages.

The pipeline life cycle requires that pipelines do not release underlying stages or progress trackers within stages. This is necessary to avoid race conditions for monitoring and status calls, which may call pipelines at various stages of preparation and also following release. To release pipeline resources, clients must drop references to the pipeline itself.

Author:

Robert Hodges

Constructor Summary

Pipeline()

Method Summary

void	addService(java.lang.String name, PipelineService service)
void	addStage(Stage stage)
void	addStore(java.lang.String name, Store store)
void	configure(PluginContext context) Configures pipeline data structures including stages and stores.
java.util.concurrent.Future<ReplDBMSHeader>	flush() Find the current native event ID in the DBMS and wait until it reaches the log.
double	getApplyLatency() Returns the latency of applying the last committed event in seconds.
int	getChannels() Returns the number of parallel apply channels in final stage.
PluginContext	getContext()
Stage	getFirstStage()
Extractor	getHeadExtractor() Returns extractor at head of pipeline.
ReplDBMSHeader	getLastAppliedEvent() Returns the last event applied in the last stage.
long	getLastAppliedSeqno() Returns the last sequence number applied in the last stage.
long	getLastExtractedSeqno() Returns the last value processed in the first stage.
Stage	getLastStage()
long	getMaxCommittedSeqno() getMaxCommittedSeqno returns the max committed sequence number from all the stores.
long	getMaxStoredSeqno() Returns the current maximum stored sequence number.
long	getMinStoredSeqno() Returns the current minimum stored sequence number.
java.lang.String	getName() Sets the name of this pipeline, which must be unique across all defined pipelines.
java.lang.String	getOfflineRequests() Returns a formatted list of current offline requests.
java.lang.String	getPipelineSource() Returns the source of the current pipeline, if known.
PipelineService	getService(java.lang.String name)
java.util.List<java.lang.String>	getServiceNames()
java.util.List<ShardProgress>	getShardProgress() Returns shard progress instances ordered by shard ID.
Stage	getStage(java.lang.String name)
java.util.List<Stage>	getStages()
Store	getStore(java.lang.String name)
java.util.List<java.lang.String>	getStoreNames()
Applier	getTailApplier() Returns applier at tail of pipeline.
java.util.List<TaskProgress>	getTaskProgress() Returns task progress instances ordered by task ID.
java.util.List<Watch<?>>	getWatches(boolean committed) Returns the currently scheduled watches.
boolean	isAutoSync()
boolean	isShutdown() Returns true if the pipeline has stopped.
void	prepare(PluginContext context) Prepare plug-in for use.
void	release(PluginContext context) Release all resources used by plug-in.
void	setApplySkipCount(long skipCount) Sets the number of apply transactions to skip, which allows the pipeline to skip over errors.
void	setApplySkipEvents(java.util.SortedSet<java.lang.Long> seqnos)
void	setAutoSync(boolean autoSync)
void	setInitialEventId(java.lang.String eventId) Sets the native event ID from which to start extracting.
void	setName(java.lang.String name)
void	setSyncTHLWithExtractor(boolean syncTHLWithExtractor)
void	shutdown(boolean immediate) Stop pipeline operation.
java.util.concurrent.Future<Pipeline>	shutdownAfterEventId(java.lang.String eventId) Shuts down after a particular event ID is applied.
java.util.concurrent.Future<Pipeline>	shutdownAfterHeartbeat(java.lang.String name) Shuts down after a heartbeat event is seen.
java.util.concurrent.Future<Pipeline>	shutdownAfterSequenceNumber(long seqno) Shuts down after a particular sequence number is applied.
java.util.concurrent.Future<Pipeline>	shutdownAfterTimestamp(java.sql.Timestamp timestamp) Shuts down after the replication event timestamp meets or exceeds the argument.
void	start(com.continuent.tungsten.fsm.event.EventDispatcher eventDispatcher) Start pipeline operation.
boolean	syncTHLWithExtractor()
java.util.concurrent.Future<ReplDBMSHeader>	watchForCommittedSequenceNumber(long seqno, boolean terminate) Sets a watch for a particular sequence number to be safely committed on all channels at the end of the pipeline.
java.util.concurrent.Future<ReplDBMSHeader>	watchForExtractedEventId(java.lang.String eventId) Sets a watch for a particular event ID to be extracted.
java.util.concurrent.Future<ReplDBMSHeader>	watchForExtractedSequenceNumber(long seqno) Sets a watch for a particular sequence number to be extracted.
java.util.concurrent.Future<ReplDBMSHeader>	watchForProcessedEventId(java.lang.String eventId) Sets a watch for a particular event ID to be processed.
java.util.concurrent.Future<ReplDBMSHeader>	watchForProcessedSequenceNumber(long seqno) Sets a watch for a particular sequence number to be processed.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Pipeline

public Pipeline()

Method Detail

getName

public java.lang.String getName()

Sets the name of this pipeline, which must be unique across all defined pipelines.

setName

public void setName(java.lang.String name)

isAutoSync

public boolean isAutoSync()

setAutoSync

public void setAutoSync(boolean autoSync)

addStage

public void addStage(Stage stage)

addStore

public void addStore(java.lang.String name,
                     Store store)

addService

public void addService(java.lang.String name,
                       PipelineService service)

getStage

public Stage getStage(java.lang.String name)

getStages

public java.util.List<Stage> getStages()

getFirstStage

public Stage getFirstStage()

getLastStage

public Stage getLastStage()

getContext

public PluginContext getContext()

getHeadExtractor

public Extractor getHeadExtractor()

Returns extractor at head of pipeline.

getTailApplier

public Applier getTailApplier()

Returns applier at tail of pipeline.

getStore

public Store getStore(java.lang.String name)

getStoreNames

public java.util.List<java.lang.String> getStoreNames()

getService

public PipelineService getService(java.lang.String name)

getServiceNames

public java.util.List<java.lang.String> getServiceNames()

getPipelineSource

public java.lang.String getPipelineSource()

Returns the source of the current pipeline, if known.

getChannels

public int getChannels()

Returns the number of parallel apply channels in final stage.

configure

public void configure(PluginContext context)
               throws ReplicatorException,
                      java.lang.InterruptedException

Configures pipeline data structures including stages and stores. All pipeline information is accessible after this call.

Specified by:

configure in interface ReplicatorPlugin

Throws:

ReplicatorException - Thrown if configuration is incomplete or fails

java.lang.InterruptedException

See Also:

ReplicatorPlugin.configure(com.continuent.tungsten.replicator.plugin.PluginContext)

prepare

public void prepare(PluginContext context)
             throws ReplicatorException,
                    java.lang.InterruptedException

Prepare plug-in for use. This method is assumed to allocate all required resources. It is called before the plug-in performs any operations.

Specified by:

prepare in interface ReplicatorPlugin

Throws:

ReplicatorException - Thrown if resource allocation fails

java.lang.InterruptedException

See Also:

ReplicatorPlugin.prepare(com.continuent.tungsten.replicator.plugin.PluginContext)

release

public void release(PluginContext context)

Release all resources used by plug-in. This is called before the plug-in is deallocated.

Specified by:

release in interface ReplicatorPlugin

See Also:

ReplicatorPlugin.release(com.continuent.tungsten.replicator.plugin.PluginContext)

start

public void start(com.continuent.tungsten.fsm.event.EventDispatcher eventDispatcher)
           throws ReplicatorException

Start pipeline operation. This is called when replication goes online.

Throws:

ReplicatorException

shutdown

public void shutdown(boolean immediate)

Stop pipeline operation. This is called when replication goes offline.

shutdownAfterSequenceNumber

public java.util.concurrent.Future<Pipeline> shutdownAfterSequenceNumber(long seqno)
                                                                  throws java.lang.InterruptedException,
                                                                         ReplicatorException

Shuts down after a particular sequence number is applied.

Parameters:

seqno - Sequence number to watch for

Returns:

Returns future to wait for pipeline shutdown

Throws:

java.lang.InterruptedException

ReplicatorException

shutdownAfterEventId

public java.util.concurrent.Future<Pipeline> shutdownAfterEventId(java.lang.String eventId)
                                                           throws java.lang.InterruptedException,
                                                                  ReplicatorException

Shuts down after a particular event ID is applied.

Parameters:

eventId - Event ID to watch for

Returns:

Returns future to wait for pipeline shutdown

Throws:

java.lang.InterruptedException

ReplicatorException

shutdownAfterHeartbeat

public java.util.concurrent.Future<Pipeline> shutdownAfterHeartbeat(java.lang.String name)
                                                             throws java.lang.InterruptedException,
                                                                    ReplicatorException

Shuts down after a heartbeat event is seen.

Returns:

Returns future to wait for pipeline shutdown

Throws:

java.lang.InterruptedException

ReplicatorException

shutdownAfterTimestamp

public java.util.concurrent.Future<Pipeline> shutdownAfterTimestamp(java.sql.Timestamp timestamp)
                                                             throws java.lang.InterruptedException,
                                                                    ReplicatorException

Shuts down after the replication event timestamp meets or exceeds the argument.

Parameters:

timestamp - Timestamp value to wait for

Returns:

Returns future to wait for pipeline shutdown

Throws:

java.lang.InterruptedException

ReplicatorException

isShutdown

public boolean isShutdown()

Returns true if the pipeline has stopped.

getLastExtractedSeqno

public long getLastExtractedSeqno()

Returns the last value processed in the first stage.

getLastAppliedSeqno

public long getLastAppliedSeqno()

Returns the last sequence number applied in the last stage.

getLastAppliedEvent

public ReplDBMSHeader getLastAppliedEvent()

Returns the last event applied in the last stage.

getApplyLatency

public double getApplyLatency()

Returns the latency of applying the last committed event in seconds.

getMinStoredSeqno

public long getMinStoredSeqno()

Returns the current minimum stored sequence number.

getMaxStoredSeqno

public long getMaxStoredSeqno()

Returns the current maximum stored sequence number.

getOfflineRequests

public java.lang.String getOfflineRequests()

Returns a formatted list of current offline requests.

getTaskProgress

public java.util.List<TaskProgress> getTaskProgress()

Returns task progress instances ordered by task ID.

getShardProgress

public java.util.List<ShardProgress> getShardProgress()

Returns shard progress instances ordered by shard ID. Shard progress is measured from the end of the pipeline, so we fetch it from the final task only.

setInitialEventId

public void setInitialEventId(java.lang.String eventId)

Sets the native event ID from which to start extracting. This overrides the default value obtained from the applier at the end of the pipeline. Must be called before start() to have an effect.

Parameters:

eventId - Event ID from which to start replication

setApplySkipCount

public void setApplySkipCount(long skipCount)

Sets the number of apply transactions to skip, which allows the pipeline to skip over errors.

watchForExtractedSequenceNumber

public java.util.concurrent.Future<ReplDBMSHeader> watchForExtractedSequenceNumber(long seqno)
                                                                            throws java.lang.InterruptedException

Sets a watch for a particular sequence number to be extracted.

Parameters:

seqno - Sequence number to watch for

Returns:

Returns a watch on a corresponding event

Throws:

java.lang.InterruptedException - if cancelled

watchForExtractedEventId

public java.util.concurrent.Future<ReplDBMSHeader> watchForExtractedEventId(java.lang.String eventId)
                                                                     throws java.lang.InterruptedException

Sets a watch for a particular event ID to be extracted.

Parameters:

eventId - Native event ID to watch for

Returns:

Returns a watch on a corresponding event

Throws:

java.lang.InterruptedException - if cancelled

watchForCommittedSequenceNumber

public java.util.concurrent.Future<ReplDBMSHeader> watchForCommittedSequenceNumber(long seqno,
                                                                                   boolean terminate)
                                                                            throws java.lang.InterruptedException

Sets a watch for a particular sequence number to be safely committed on all channels at the end of the pipeline.

Parameters:

seqno - Sequence number to watch for

terminate - If true, terminate task when watch is successful

Returns:

Returns a watch on matching event

Throws:

java.lang.InterruptedException

watchForProcessedSequenceNumber

public java.util.concurrent.Future<ReplDBMSHeader> watchForProcessedSequenceNumber(long seqno)
                                                                            throws java.lang.InterruptedException

Sets a watch for a particular sequence number to be processed.

Parameters:

seqno - Sequence number to watch for

Returns:

Returns a future on the event that meets criterion

Throws:

java.lang.InterruptedException - if cancelled

watchForProcessedEventId

public java.util.concurrent.Future<ReplDBMSHeader> watchForProcessedEventId(java.lang.String eventId)
                                                                     throws java.lang.InterruptedException

Sets a watch for a particular event ID to be processed.

Parameters:

eventId - Native event ID to watch for

Returns:

Returns a watch on a corresponding event

Throws:

java.lang.InterruptedException - if canceled

getWatches

public java.util.List<Watch<?>> getWatches(boolean committed)

Returns the currently scheduled watches.

flush

public java.util.concurrent.Future<ReplDBMSHeader> flush()
                                                  throws java.lang.InterruptedException,
                                                         ReplicatorException

Find the current native event ID in the DBMS and wait until it reaches the log.

Returns:

A Future on the ReplDBMSEvent that has this eventId or a greater one.

Throws:

java.lang.InterruptedException

ReplicatorException

setSyncTHLWithExtractor

public void setSyncTHLWithExtractor(boolean syncTHLWithExtractor)

syncTHLWithExtractor

public boolean syncTHLWithExtractor()

setApplySkipEvents

public void setApplySkipEvents(java.util.SortedSet<java.lang.Long> seqnos)

getMaxCommittedSeqno

public long getMaxCommittedSeqno()
                          throws ReplicatorException

getMaxCommittedSeqno returns the max committed sequence number from all the stores.

Returns:

the max committed seqno

Throws:

ReplicatorException - in case an error occurs