Class NonBlockingCoordinator
- java.lang.Object
- 
- org.apache.catalina.tribes.group.ChannelInterceptorBase
- 
- org.apache.catalina.tribes.group.interceptors.NonBlockingCoordinator
 
 
- 
- All Implemented Interfaces:
- ChannelInterceptor,- Heartbeat,- MembershipListener
 
 public class NonBlockingCoordinator extends ChannelInterceptorBase Title: Auto merging leader election algorithm Description: Implementation of a simple coordinator algorithm that not only selects a coordinator, it also merges groups automatically when members are discovered that weren't part of the This algorithm is non blocking meaning it allows for transactions while the coordination phase is going on This implementation is based on a home brewed algorithm that uses the AbsoluteOrder of a membership to pass a token ring of the current membership. 
 This is not the same as just using AbsoluteOrder! Consider the following scenario:
 Nodes, A,B,C,D,E on a network, in that priority. AbsoluteOrder will only work if all nodes are receiving pings from all the other nodes. meaning, that node{i} receives pings from node{all}-node{i}
 but the following could happen if a multicast problem occurs. A has members {B,C,D}
 B has members {A,C}
 C has members {D,E}
 D has members {A,B,C,E}
 E has members {A,C,D}
 Because the default Tribes membership implementation, relies on the multicast packets to arrive at all nodes correctly, there is nothing guaranteeing that it will.
 
 To best explain how this algorithm works, lets take the above example: For simplicity we assume that a send operation is O(1) for all nodes, although this algorithm will work where messages overlap, as they all depend on absolute order
 Scenario 1: A,B,C,D,E all come online at the same time Eval phase, A thinks of itself as leader, B thinks of A as leader, C thinks of itself as leader, D,E think of A as leader
 Token phase:
 (1) A sends out a message X{A-ldr, A-src, mbrs-A,B,C,D} to B where X is the id for the message(and the view)
 (1) C sends out a message Y{C-ldr, C-src, mbrs-C,D,E} to D where Y is the id for the message(and the view)
 (2) B receives X{A-ldr, A-src, mbrs-A,B,C,D}, sends X{A-ldr, A-src, mbrs-A,B,C,D} to C
 (2) D receives Y{C-ldr, C-src, mbrs-C,D,E} D is aware of A,B, sends Y{A-ldr, C-src, mbrs-A,B,C,D,E} to E
 (3) C receives X{A-ldr, A-src, mbrs-A,B,C,D}, sends X{A-ldr, A-src, mbrs-A,B,C,D,E} to D
 (3) E receives Y{A-ldr, C-src, mbrs-A,B,C,D,E} sends Y{A-ldr, C-src, mbrs-A,B,C,D,E} to A
 (4) D receives X{A-ldr, A-src, mbrs-A,B,C,D,E} sends sends X{A-ldr, A-src, mbrs-A,B,C,D,E} to A
 (4) A receives Y{A-ldr, C-src, mbrs-A,B,C,D,E}, holds the message, add E to its list of members
 (5) A receives X{A-ldr, A-src, mbrs-A,B,C,D,E}
 At this point, the state looks like
 A - {A-ldr, mbrs-A,B,C,D,E, id=X}
 B - {A-ldr, mbrs-A,B,C,D, id=X}
 C - {A-ldr, mbrs-A,B,C,D,E, id=X}
 D - {A-ldr, mbrs-A,B,C,D,E, id=X}
 E - {A-ldr, mbrs-A,B,C,D,E, id=Y}
 
 A message doesn't stop until it reaches its original sender, unless its dropped by a higher leader. As you can see, E still thinks the viewId=Y, which is not correct. But at this point we have arrived at the same membership and all nodes are informed of each other.
 To synchronize the rest we simply perform the following check at A when A receives X:
 Original X{A-ldr, A-src, mbrs-A,B,C,D} == Arrived X{A-ldr, A-src, mbrs-A,B,C,D,E}
 Since the condition is false, A, will resend the token, and A sends X{A-ldr, A-src, mbrs-A,B,C,D,E} to B When A receives X again, the token is complete.
 Optionally, A can send a message X{A-ldr, A-src, mbrs-A,B,C,D,E confirmed} to A,B,C,D,E who then install and accept the view.Lets assume that C1 arrives, C1 has lower priority than C, but higher priority than D. 
 Lets also assume that C1 sees the following view {B,D,E}
 C1 waits for a token to arrive. When the token arrives, the same scenario as above will happen.
 In the scenario where C1 sees {D,E} and A,B,C cannot see C1, no token will ever arrive.
 In this case, C1 sends a Z{C1-ldr, C1-src, mbrs-C1,D,E} to D
 D receives Z{C1-ldr, C1-src, mbrs-C1,D,E} and sends Z{A-ldr, C1-src, mbrs-A,B,C,C1,D,E} to E
 E receives Z{A-ldr, C1-src, mbrs-A,B,C,C1,D,E} and sends it to A
 A sends Z{A-ldr, A-src, mbrs-A,B,C,C1,D,E} to B and the chain continues until A receives the token again. At that time A optionally sends out Z{A-ldr, A-src, mbrs-A,B,C,C1,D,E, confirmed} to A,B,C,C1,D,ETo ensure that the view gets implemented at all nodes at the same time, A will send out a VIEW_CONF message, this is the 'confirmed' message that is optional above. Ideally, the interceptor below this one would be the TcpFailureDetector to ensure correct memberships The example above, of course can be simplified with a finite statemachine: 
 But I suck at writing state machines, my head gets all confused. One day I will document this algorithm though.
 Maybe I'll do a state diagram :)State DiagramsInitiate an election
 Receive an election message
- 
- 
Nested Class SummaryNested Classes Modifier and Type Class Description static classNonBlockingCoordinator.CoordinationEventstatic classNonBlockingCoordinator.CoordinationMessage- 
Nested classes/interfaces inherited from interface org.apache.catalina.tribes.ChannelInterceptorChannelInterceptor.InterceptorEvent
 
- 
 - 
Field SummaryFields Modifier and Type Field Description protected static byte[]COORD_ALIVEAlive messageprotected static byte[]COORD_CONFCoordination confirmation, for blocking installationsprotected static byte[]COORD_HEADERheader for a coordination messageprotected static byte[]COORD_REQUESTCoordination requestprotected java.util.concurrent.atomic.AtomicBooleancoordMsgReceivedprotected java.lang.ObjectelectionMutexprotected MembershipmembershipOur nonblocking membershipprotected static StringManagersmprotected booleanstartedprotected intstartsvcprotected MembershipsuggestedViewprotected UniqueIdsuggestedviewIdindicates that we are running an election and this is the one we are runningprotected MembershipviewOur current viewprotected UniqueIdviewIdOut current viewIdprotected longwaitForCoordMsgTimeoutTime to wait for coordination timeout- 
Fields inherited from class org.apache.catalina.tribes.group.ChannelInterceptorBaseoptionFlag
 
- 
 - 
Constructor SummaryConstructors Constructor Description NonBlockingCoordinator()
 - 
Method SummaryAll Methods Instance Methods Concrete Methods Modifier and Type Method Description protected booleanalive(Member mbr)ChannelDatacreateData(NonBlockingCoordinator.CoordinationMessage msg, Member local)voidfireInterceptorEvent(ChannelInterceptor.InterceptorEvent event)MembergetCoordinator()Returns coordinator if one is availableMembergetLocalMember(boolean incAlive)Return the member that represents this node.MembergetMember(Member mbr)Intercepts theChannel.getMember(Member)methodMember[]getMembers()Get all current cluster membersMember[]getView()UniqueIdgetViewId()protected voidhalt()Block in/out messages while a election is going onprotected voidhandleMyToken(Member local, NonBlockingCoordinator.CoordinationMessage msg, Membership merged)protected voidhandleOtherToken(Member local, NonBlockingCoordinator.CoordinationMessage msg, Membership merged)protected voidhandleToken(NonBlockingCoordinator.CoordinationMessage msg, Membership merged)protected voidhandleViewConf(NonBlockingCoordinator.CoordinationMessage msg, Membership merged)protected booleanhasHigherPriority(Member[] complete, Member[] local)booleanhasMembers()has membersvoidheartbeat()Theheartbeat()method gets invoked periodically to allow interceptors to clean up resources, time out object and perform actions that are unrelated to sending/receiving data.booleanisCoordinator()booleanisHighest()protected booleanisViewConf(NonBlockingCoordinator.CoordinationMessage msg)voidmemberAdded(Member member)A member was added to the groupvoidmemberAdded(Member member, boolean elect)protected booleanmemberAlive(Member mbr, long conTimeout)voidmemberDisappeared(Member member)A member was removed from the group
 If the member left voluntarily, the Member.getCommand will contain the Member.SHUTDOWN_PAYLOAD dataprotected MembershipmergeOnArrive(NonBlockingCoordinator.CoordinationMessage msg)voidmessageReceived(ChannelMessage msg)themessageReceivedis invoked when a message is received.protected voidprocessCoordMessage(NonBlockingCoordinator.CoordinationMessage msg)protected voidrelease()Release lock for in/out messages election is completedprotected voidsendElectionMsg(Member local, Member next, NonBlockingCoordinator.CoordinationMessage msg)protected voidsendElectionMsgToNextInline(Member local, NonBlockingCoordinator.CoordinationMessage msg)voidsendMessage(Member[] destination, ChannelMessage msg, InterceptorPayload payload)ThesendMessagemethod is called when a message is being sent to one more destinations.protected voidsetupMembership()voidstart(int svc)Starts up the channel.voidstartElection(boolean force)voidstop(int svc)Shuts down the channel.protected voidwaitForRelease()Wait for an election to end- 
Methods inherited from class org.apache.catalina.tribes.group.ChannelInterceptorBasegetChannel, getNext, getOptionFlag, getPrevious, okToProcess, setChannel, setNext, setOptionFlag, setPrevious
 
- 
 
- 
- 
- 
Field Detail- 
smprotected static final StringManager sm 
 - 
COORD_HEADERprotected static final byte[] COORD_HEADER header for a coordination message
 - 
COORD_REQUESTprotected static final byte[] COORD_REQUEST Coordination request
 - 
COORD_CONFprotected static final byte[] COORD_CONF Coordination confirmation, for blocking installations
 - 
COORD_ALIVEprotected static final byte[] COORD_ALIVE Alive message
 - 
waitForCoordMsgTimeoutprotected final long waitForCoordMsgTimeout Time to wait for coordination timeout- See Also:
- Constant Field Values
 
 - 
viewprotected volatile Membership view Our current view
 - 
viewIdprotected UniqueId viewId Out current viewId
 - 
membershipprotected Membership membership Our nonblocking membership
 - 
suggestedviewIdprotected UniqueId suggestedviewId indicates that we are running an election and this is the one we are running
 - 
suggestedViewprotected volatile Membership suggestedView 
 - 
startedprotected volatile boolean started 
 - 
startsvcprotected final int startsvc - See Also:
- Constant Field Values
 
 - 
electionMutexprotected final java.lang.Object electionMutex 
 - 
coordMsgReceivedprotected final java.util.concurrent.atomic.AtomicBoolean coordMsgReceived 
 
- 
 - 
Method Detail- 
startElectionpublic void startElection(boolean force) throws ChannelException- Throws:
- ChannelException
 
 - 
sendElectionMsgprotected void sendElectionMsg(Member local, Member next, NonBlockingCoordinator.CoordinationMessage msg) throws ChannelException - Throws:
- ChannelException
 
 - 
sendElectionMsgToNextInlineprotected void sendElectionMsgToNextInline(Member local, NonBlockingCoordinator.CoordinationMessage msg) throws ChannelException - Throws:
- ChannelException
 
 - 
createDatapublic ChannelData createData(NonBlockingCoordinator.CoordinationMessage msg, Member local) 
 - 
aliveprotected boolean alive(Member mbr) 
 - 
memberAliveprotected boolean memberAlive(Member mbr, long conTimeout) 
 - 
mergeOnArriveprotected Membership mergeOnArrive(NonBlockingCoordinator.CoordinationMessage msg) 
 - 
processCoordMessageprotected void processCoordMessage(NonBlockingCoordinator.CoordinationMessage msg) throws ChannelException - Throws:
- ChannelException
 
 - 
handleTokenprotected void handleToken(NonBlockingCoordinator.CoordinationMessage msg, Membership merged) throws ChannelException - Throws:
- ChannelException
 
 - 
handleMyTokenprotected void handleMyToken(Member local, NonBlockingCoordinator.CoordinationMessage msg, Membership merged) throws ChannelException - Throws:
- ChannelException
 
 - 
handleOtherTokenprotected void handleOtherToken(Member local, NonBlockingCoordinator.CoordinationMessage msg, Membership merged) throws ChannelException - Throws:
- ChannelException
 
 - 
handleViewConfprotected void handleViewConf(NonBlockingCoordinator.CoordinationMessage msg, Membership merged) throws ChannelException - Throws:
- ChannelException
 
 - 
isViewConfprotected boolean isViewConf(NonBlockingCoordinator.CoordinationMessage msg) 
 - 
getCoordinatorpublic Member getCoordinator() Returns coordinator if one is available- Returns:
- Member
 
 - 
getViewpublic Member[] getView() 
 - 
getViewIdpublic UniqueId getViewId() 
 - 
haltprotected void halt() Block in/out messages while a election is going on
 - 
releaseprotected void release() Release lock for in/out messages election is completed
 - 
waitForReleaseprotected void waitForRelease() Wait for an election to end
 - 
startpublic void start(int svc) throws ChannelExceptionDescription copied from class:ChannelInterceptorBaseStarts up the channel. This can be called multiple times for individual services to start The svc parameter can be the logical or value of any constants- Specified by:
- startin interface- ChannelInterceptor
- Overrides:
- startin class- ChannelInterceptorBase
- Parameters:
- svc- int value of
 DEFAULT - will start all services
 MBR_RX_SEQ - starts the membership receiver
 MBR_TX_SEQ - starts the membership broadcaster
 SND_TX_SEQ - starts the replication transmitter
 SND_RX_SEQ - starts the replication receiver
- Throws:
- ChannelException- if a startup error occurs or the service is already started.
- See Also:
- Channel
 
 - 
stoppublic void stop(int svc) throws ChannelExceptionDescription copied from class:ChannelInterceptorBaseShuts down the channel. This can be called multiple times for individual services to shutdown The svc parameter can be the logical or value of any constants- Specified by:
- stopin interface- ChannelInterceptor
- Overrides:
- stopin class- ChannelInterceptorBase
- Parameters:
- svc- int value of
 DEFAULT - will shutdown all services
 MBR_RX_SEQ - stops the membership receiver
 MBR_TX_SEQ - stops the membership broadcaster
 SND_TX_SEQ - stops the replication transmitter
 SND_RX_SEQ - stops the replication receiver
- Throws:
- ChannelException- if a startup error occurs or the service is already started.
- See Also:
- Channel
 
 - 
sendMessagepublic void sendMessage(Member[] destination, ChannelMessage msg, InterceptorPayload payload) throws ChannelException Description copied from interface:ChannelInterceptorThesendMessagemethod is called when a message is being sent to one more destinations. The interceptor can modify any of the parameters and then pass on the message down the stack by invokinggetNext().sendMessage(destination,msg,payload)
 Alternatively the interceptor can stop the message from being sent by not invokinggetNext().sendMessage(destination,msg,payload)
 If the message is to be sent asynchronous the application can be notified of completion and errors by passing in an error handler attached to a payload object.
 The ChannelMessage.getAddress contains Channel.getLocalMember, and can be overwritten to simulate a message sent from another node.- Specified by:
- sendMessagein interface- ChannelInterceptor
- Overrides:
- sendMessagein class- ChannelInterceptorBase
- Parameters:
- destination- Member[] - the destination for this message
- msg- ChannelMessage - the message to be sent
- payload- InterceptorPayload - the payload, carrying an error handler and future useful data, can be null
- Throws:
- ChannelException- if a serialization error happens.
- See Also:
- ErrorHandler,- InterceptorPayload
 
 - 
messageReceivedpublic void messageReceived(ChannelMessage msg) Description copied from interface:ChannelInterceptorthemessageReceivedis invoked when a message is received.ChannelMessage.getAddress()is the sender, or the reply-to address if it has been overwritten.- Specified by:
- messageReceivedin interface- ChannelInterceptor
- Overrides:
- messageReceivedin class- ChannelInterceptorBase
- Parameters:
- msg- ChannelMessage
 
 - 
memberAddedpublic void memberAdded(Member member) Description copied from interface:MembershipListenerA member was added to the group- Specified by:
- memberAddedin interface- MembershipListener
- Overrides:
- memberAddedin class- ChannelInterceptorBase
- Parameters:
- member- Member - the member that was added
 
 - 
memberAddedpublic void memberAdded(Member member, boolean elect) 
 - 
memberDisappearedpublic void memberDisappeared(Member member) Description copied from interface:MembershipListenerA member was removed from the group
 If the member left voluntarily, the Member.getCommand will contain the Member.SHUTDOWN_PAYLOAD data- Specified by:
- memberDisappearedin interface- MembershipListener
- Overrides:
- memberDisappearedin class- ChannelInterceptorBase
- Parameters:
- member- Member
- See Also:
- Member.SHUTDOWN_PAYLOAD
 
 - 
isHighestpublic boolean isHighest() 
 - 
isCoordinatorpublic boolean isCoordinator() 
 - 
heartbeatpublic void heartbeat() Description copied from interface:ChannelInterceptorTheheartbeat()method gets invoked periodically to allow interceptors to clean up resources, time out object and perform actions that are unrelated to sending/receiving data.- Specified by:
- heartbeatin interface- ChannelInterceptor
- Specified by:
- heartbeatin interface- Heartbeat
- Overrides:
- heartbeatin class- ChannelInterceptorBase
 
 - 
hasMemberspublic boolean hasMembers() has members- Specified by:
- hasMembersin interface- ChannelInterceptor
- Overrides:
- hasMembersin class- ChannelInterceptorBase
- Returns:
- boolean - if the channel has members in its membership group
- See Also:
- Channel.hasMembers()
 
 - 
getMemberspublic Member[] getMembers() Get all current cluster members- Specified by:
- getMembersin interface- ChannelInterceptor
- Overrides:
- getMembersin class- ChannelInterceptorBase
- Returns:
- all members or empty array
- See Also:
- Channel.getMembers()
 
 - 
getMemberpublic Member getMember(Member mbr) Description copied from interface:ChannelInterceptorIntercepts theChannel.getMember(Member)method- Specified by:
- getMemberin interface- ChannelInterceptor
- Overrides:
- getMemberin class- ChannelInterceptorBase
- Parameters:
- mbr- Member
- Returns:
- Member
- See Also:
- Channel.getMember(Member)
 
 - 
getLocalMemberpublic Member getLocalMember(boolean incAlive) Return the member that represents this node.- Specified by:
- getLocalMemberin interface- ChannelInterceptor
- Overrides:
- getLocalMemberin class- ChannelInterceptorBase
- Parameters:
- incAlive- boolean
- Returns:
- Member
- See Also:
- Channel.getLocalMember(boolean)
 
 - 
setupMembershipprotected void setupMembership() 
 - 
fireInterceptorEventpublic void fireInterceptorEvent(ChannelInterceptor.InterceptorEvent event) - Specified by:
- fireInterceptorEventin interface- ChannelInterceptor
- Overrides:
- fireInterceptorEventin class- ChannelInterceptorBase
 
 
- 
 
-