Versions Compared

Old Version 2

changes.mady.by.user 정은진

Saved on

compared with

New Version 3

changes.mady.by.user 정은진

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Section


Column

After configuring redundancy environment using MCCS, some failures might occur.
This chapter will explain how MCCS detects the failure and administrates after failure or failover is done. 
(In the following example, the operating server as 'Active', standby server name as 'Standby' is registered on MCCS.)


Column
width350px


Panel

Table of Contents

Table of Contents
maxLevel4



...

  1. MCCS will show the failure when failure occurs in standby server.
  2. Data replication will be paused until standby server is back to normal.


    [Figure] Failure in Standby Server

  3. Data synchronization cannot be achieved. Mirror disk becomes the 'Network Connection Failure' state (   ).
  4. Even if the standby server failed, it does not affect operation. But as there is no server to perform failover to, the server operator must check the trouble in the MCCS web console and make sure that the standby server is normalized in time.
  5. When the standby server is normalized, the  icon disappears.
  6. Mirror disk's DiskState value is changed from 'Inconsistent' to 'UpToDate' and for this purpose, data synchronization () is  is performed. 
  7. When the synchronization is finished, the current normal data is synchronized in real time. The icon is changed to (). 

Application Failure

Active application resources are operated by 4 elements below.

...

Network failure happens when network connection has problem, such as network switch or network interface card is broken or disconnection in network cable, or ping timeout of some network and so on.

Warning

※ Since MCCS license referenced to MAC address, license should be reissued if there is a change in network interface card.

...

    • Replication (Mirroring) Network Failure

      When the copying network failed, data copying cannot be done. The mirror disk resource of MCCS web console displays the 'Disconnect' ( ).

      [Figure]Failure in Replicated Network


...

    • Single Network Switch Fault

      When failure occurs in network switch connected to Public Network where it is configured by single network switch, all the resources in active and standby server will be taken offline, resources where failure occurs will show as 'fault'.

      [Figure] Failure in Network Switch    

  1. Network switch failure can be checked on MCCS log, OS System log. If failure occurs in service network connection, server operator should check on the TCP/IP of server, physical connection check on the service network through ping test.
  2. If you want the sign of failures to be removed automatically, enter a positive number in AutoFaultClearTime of the group attribute.
  3. Please get the supports regarding the recovery of Network switch failure through manufacturer.



...

  • Target Disk Failure
    If the disk at the standby server failed, the disk resource icon of MCCS web console is not changed. But the attribute values of Diskstate is changed from UptoDate to Diskless. However, the service running in the source server is not affected by it. 

    [Figure] Failure in Target Disk

  1. When MCCS detects failures of the target disk, only the DiskState value of the disk is displayed. 
  2. Disk failure can be caused by the following. After resolving the above issues, the OS will detect the newly changed disk again. After that, DRBD will proceed with synchronization.
    • Disk controller problems or H/W problems should be fixed by the manufacturers.
    • Physical disk problems or H/W problems should be fixed by the manufacturers.
  3. If the mirror disk does not perform synchronization, delete the mirror disk resource and try to create it again. But when you delete the resource, you must also delete the created mirror and create them again. 

...

  1. Due to the failure of source server (A), a failover occurs.
  2. The role of target server (B) is changed to Primary. (Mirror disk role changed)
  3. Reboot the initial source server (A).
  4. After the initial source server (A) boots, check the role of the target server (B).
  5. Check the GI value for the both nodes.
  6.  Check if the GI data is matched, and if it is, proceed with data synchronization automatically. (5th/ 6th processes checking fails.)
  7. GI data is not matched. So, synchronization is required on one node. No automatic synchronization is taking place.  (Split brain occurred.)
     
    When this state is reached, the icon of mirror disk resource is overlapped in the MCCS web console. Both the  and the 'SplitBrainStatus' attribute values will be set to true.
    In this case, you need to manually change the mirror disk role. After that, you need to resynchronize it.
    If you want to manually change the mirror disk role, you need to use the MCCS web console. 

...

  1. Check the resource attribute view. 

    [Figure] Verify SplitBrain of MirrorDisk


  2. Check the mirror management view.


    [Figure] Checking Mirror Disk Split Brains

    Warning

    1) The ConnectState of both node is StandAlone and SplitBrainStatus values are set to True.
    2) Check LastMirrorOnlineTime on the mirror disk. (LastMirrorOnlineTime is the system time. So, it is not the absolute value used to determine whether it is the latest data.)
    3) When a split brain occurs, the log will be displayed.
    (DRBD volume (r0) has a split brain.)
    4) In the mirror management window, the mirror condition is set to 'SPLIT'.


  3. Select the mirror disk and right click with your mouse button and click on the 'Resolve Split Brains' button.

    [Figure] Split Brain Resolving Selected

  4. Display the window to explain split brains.
     
    [Figure] Checking the Source Node Selection

  5. Select the source node.

    [Figure] Source Roll Node Selection

  6. Recheck Check again the selected source node.

    [Figure] Rechecking the Source Node Selection

  7. Split brains problems being resolved.

    [Figure] Split Brain Resolved

  8. Resolving split brains problems is finished.

    [Figure] Resolving Split Brain Finished

  9. The selected node becomes the source node and the mirror disk condition is changed from "DiskState" to "UpToData". 

    [Figure] Split Brain Resolve

    Warning

    The changed information of node B will be all overwritten.


...

  1.  In the MCCS web console, click 'File' on the menu bar to collect support files.

    [Figure] Collecting Support Files from Menu Bar  

  2. Support files can be collected by clicking the toolbar shown in the figure below.

    [Figure] Collecting Support Files from Toolbar

  3. You can select a node to collect support files from and get the previous support file again.

    [Figure] Support File Node Selection and Previous Support File Selection

  4. Click 'the "OK' " button and support file is collected.

    [Figure] Support Files Being Collected


    Info

    It may take several minutes depending on the log file capacity and the network condition.


  5. The collected support files can be checked in the designated location.

    [Figure] Support Files

...