DISTRIBUTED SYSTEMS Principles and Paradigms Second Edition ANDREW S. TANENBAUM MAARTEN VAN STEEN Chapter 6 Synchronization (2)

Chapter 8Fault Tolerance (2)DISTRIBUTED SYSTEMS (dDist)2014

Distributed Commit (1/3)

•Given a process group and an operation

–The operation might or might not becommittable at all processes

•Goal:

–If committable at all processes, commit at allprocesses

–Either everybody eventually commits oreverybody eventually aborts

•Even servers which crash and come back to life

•Consistency, validity, termination

Distributed Commit (2/3)

•Can we not just do this with Virtual Synchrony?

–Coordinator multicasts vote request

–All processes respond to request

–Coordinator multicasts vote result

•COMMIT iff all vote COMMIT

•This handles some error cases

•But, what if a participant B crashes between abackup votes COMMIT and the COMMIT resultis broadcast and then comes back to live?

•We have to bring him up to a consistent state,even if others crash as he wakes up and so on…

Distributed Commit (3/3)

•We want to tolerate these errors:

–Transient Crash-silent errors

•We have timeouts to detect crashes

•Transient: Crashed servers come back to life

–And must then make the right decision

–Messages can be dropped

•Even if we secure communication against omissionerrors and crash errors, say by implementingvirtual synchrony, a server might be out of theview when a message is sent and then come backto life

Two-Phase Commit

1) Commit →

2) Vote-request →

3) Vote-commit ←

3) Vote-commit←

4) Global-commit →

Two-Phase Commit

•Figure 8-18. (a) The finite state machine for the coordinator in2PC. (b) The finite state machine for a participant.

Input event

Output event

COORDINATOR

PARTICIPANT

Two-Phase Commit

•2PC detects crashes via timeouts

•2PC handles crashes by logging state topermanent storage, turning crash errorsinto ommision errors

–It is possible to execute code on an arrow andthen fall back to the state before the arrow

–Then the code can get executed again, andagain, and again, …

–More on this later…

Coordinator Perspective

•Blocks in WAIT

–Participant may havefailed

–That participant mightvote ABORT, in whichcase a GLOBALCOMMIT would bewrong and irreversible

–So, must do a GLOBALABORT

TIMEOUT

COORDINATOR

Coordinator Perspective

•Figure 8-20. Outline of the steps taken by thecoordinator in a two-phase commit protocol.

. . .

COORDINATOR

Coordinator Perspective

•Figure 8-20. Outline of the steps taken by thecoordinator in a two-phase commit protocol.

. . .

COORDINATOR

Participant Perspective

•Blocks in READY

–Coordinator may havefailed

•What to do?

–Some participants mayalready havecommitted…

–Perhaps anotherparticipant knows whatto do…?

PARTICIPANT

Participant Perspective

•Figure 8-19. Actions taken by a participantP when residing in state READY andhaving contacted another participant Q.

We know that coordinator

managed to start commit

At least one participant

aborted and coordinatornoticed

Q did not even receivevote-request, so no onecommitted yet

What if all in READY?

After timeout allowing all messagesin transit to arrive:

Two-PhaseCommit

•Figure 8-21. (a) The steps taken by a participant process in 2PC.

PARTICIPANT

All READY (1/2)

Why do we block when all liveparticipants are in the READY state?

PARTICIPANT

COORDINATOR

All READY (2/2)

Same view, but different decisions, soYellow needs to wait for Blue or Green tocome up again and inspect their log files!

PARTICIPANT

COORDINATOR

Two-Phase Commit

•Two-Phase Commit has the problem thatif the coordinator and one participantcrashes at a bad time the entire systemfreezes until one of them is up again

•Getting a server up and running againtypically involves humans (a.k.a. veryslow) intervention

Three-Phase Commit

•Three-Phase Commit enhances Two-Phase Commit in that it is non-blockingin many more cases

•As long as the live participants can makea majority decision they can continue ontheir own

–Majority among all, not only the live ones

•If there are many participants, this makesit very unlikely that 3PC blocks

•Figure 8-22. (a) The finite state machine for thecoordinator in 3PC. (b) The finite state machine for aparticipant.

TIMEOUT

PARTICIPANT

COORDINATOR

Three-Phase Commit

•Figure 8-22. (a) The finite state machine for thecoordinator in 3PC. (b) The finite state machine for aparticipant.

TIMEOUT

PARTICIPANT

COORDINATOR

Three-Phase Commit

On timeout (maybe after coming alive again):

•IF anyone else in ABORT  ABORT

•ELIF anyone else in COMMIT  COMMIT

•ELIF anyone else in INIT  ABORT

•ELSE everyone else in READY or PRECOMMIT:

•If a majority of participants is in READY gotoABORT

•If a majority is in PRECOMMIT goto PRECOMMIT

•If no majority, then block until more come back to life

PARTICIPANT

COORDINATOR

PARTICIPANT

COORDINATOR

•If anyone is inPRECOMMIT, thenoriginal coordinators voteis set to be PRECOMMIT,as the original coordinatormust be in PRECOMMIT

PARTICIPANT

COORDINATOR

•If anyone is inPRECOMMIT, thenoriginal coordinators voteis set to be PRECOMMIT,as the original coordinatormust be in PRECOMMIT

More Non-Blocking

•Follows from the decision rules that thelive agents always can make decisionson their own unless no true majority forREADY or PRECOMMIT can be foundamong the live participants

•True majority: Majority among allprocesses, both dead and live

Termination

•PROPERTYIf all servers eventually are alive at the sametime, then all servers eventually end up inABORT or COMMITProof sketch:

1.If any live server is in ABORT all the remainingunresolved go to ABORT eventually

2.If any live server is in COMMIT all the remainingunresolved go to COMMIT eventually

3.Otherwise, there will be a true majority in READY orPRECOMMIT and when all are alive it can be seenwhich, and then someone goes to ABORT or COMMIT

Correctness: An Easy Case

•If the coordinator goes to ABORT withoutusing the red box, i.e., via a normal protocolflow or via the timeout, then no participant willever reach PRECOMMIT an therefore noparticipant can ever reach COMMIT

•(So since they all eventually reach COMMIT orABORT, they will all eventually reach ABORT)

PARTICIPANT

COORDINATOR

•If anyone is inPRECOMMIT, thenoriginal coordinators voteis set to be PRECOMMIT,as the original coordinatormust be in PRECOMMIT

On timeout (maybe after coming alive again):

•IF anyone else in ABORT  ABORT

•ELIF anyone else in COMMIT  COMMIT

•ELIF anyone else in INIT  ABORT

•ELSE everyone else in READY or PRECOMMIT:

•If a majority of participants is in READY gotoABORT

•If a majority is in PRECOMMIT goto COMMIT

•If no majority, then block unil more come back to life

Correctness: Another Easy Case

•If the coordinator goes to COMMIT withoutusing the red box, i.e., via a normal protocolflow, then all participant are in PRECOMMIT orCOMMIT and therefore no participants willever reach ABORT

•(So since they all eventually reach COMMIT orABORT, they will all eventually reach ABORT)