...

This is the case of system being rebooting or shut down because of conflicts of each device (NIC, Raid Controller), kernel driver problem of other application.

Active Server Failure

1. 서버의 정상 혹은 비정상 종료에 따른 MCCS의 역할에는 차이점이 없습니다. MCCS는 운영 서버에서 장애가 발생하면 대기 서버로 페일오버를 진행합니다.
   화면의 오른쪽에 있는 노드 관리에서 해당 서버를 선택하면 '리소스 상태' 및 '리소스 의존성' 화면을 통하여 장애를 확인할 수 있습니다 There is no difference in the MCCS role resulting from abnormal or normal termination of the server. MCCS will perform a failover to the standby server when the operation server fails.
   In the node management menu on the right side of the screen, select the server. You can check the details of failures in the 'Resource Status' & 'Resource Dependency' screens.
   • Normal Termination of a system
     This is a case where user selected 'system shutdown' in operating systems.
   • Abnormal Termination of a system 
     This is a case where system is terminated or rebooted due to an unexpected situation or blue screen.
   
   [ Figure] Failure in Active Server
   
   
2. Since data cannot be replicated due to the server failure, will be shown in the mirror disk resource.
3. Server operators check on the failure and put the server back to normal.
4. After checking on the mirror role of two servers when server with the failure is rebooted, switch the server with the failure as replication target and proceed partial resync.

...

1. MCCS will show the failure when failure occurs in standby server.
   
2. Data replication will be paused until standby server is back to normal.
   
   [ Figure] Failure in Standby Server
   
   
3. If I/O keep happens, data is impossible to replicate and mirror disk will be shown as 'Paused'(change of icon)
4. If there is no I/O, icon of mirror disk has no change but failure messages related mirror disk exists in MCCS log.
5. 대기 서버에서 장애가 발생하면 운영상에는 문제가 없지만 페일오버할 대상이 없으므로 서버 운영자는 반드시 MCCS 웹 콘솔을 통하여 장애를 확인하고 대기 서버를 정상화 시켜야 합니다.
6. When standby server is back Even if the standby server failed, it does not affect operation. But as there is no server to perform failover to, the server operator must check the trouble in the MCCS web console and make sure that the standby server is normalized in time.
7.  When standby server is back to normal, it will recover from 'Paused' to 'Normal' and  icon will be disappeared.

...

• MonitorInterval
  Monitors the resource with interval set value. (Default Value=10sec)
• MonitorTimeout
  If there is no reply as much as the set value, it is considered as a failure. (Default Value=10sec)
• RestartLimit
  It will restart the application resource as the set value. (Default Value=0)
• OnlieTrustTime
  It re-sets the time of number of resource restarting number.It is the time to reset the frequency of the resource to restart. (Default Value=600sec)
  Attributes above are the set value of the registered being added the resource, and users can check or change the values through Resource Attribute view of MCCS console. 
  
  [Figure] Resource attribute value Edit
  
  

1. MCCS periodically monitors the resources referring  'MonitorInterval'.
2. If there is no response as the time set in 'MonitorTimeout', it is considered as a failure.
   
3. If there are no response after sending the command as the number set in 'RestartLimit', MCCS will failover the group which resource belongs to.
4. If the resource stays in normal state within the time limit set by 'OnlineTrustTime'. MCCS will initialize the attribute value of 'RestartLimit'. This is to ensure restart number when failure occurs in a resource.
   
5. If there is a failover due to a failure in the resource, server operator checks on the problem and put it back to normal.
   
6. 장애가 발생한 부분은 MCCS 웹 콘솔에서 확인할 수 있으며, 장애가 발생한 부분을 사용자가 확인한 후에 장애 표시를 제거해 주어야 다시 페일오버 기능이 활성화됩니다. 
   자동으로 장애 표시를 제거하고자 할 경우에는 그룹 속성의 AutoFaultClearTime에 0보다 큰 값을 설정하면 됩니다In the MCCS web console, a user can see where the trouble occurs. After a user checks the trouble area, they must remove the Trouble sign, so that the failover function can be activated again. 
7. After checking on the mirror role of two servers when server with the failure is rebooted, switch the server with the failure as replication target and proceed partial resync.
   
   
   [Figure] Failure in Resource Clear
   
   

...

• Service Network Failure

  If failure occurs in service network of active server, the fault mark will be shown on the network interface card resource or IP address of the node in MCCS UI, and will failover to the standby server.

  
  [Figure] Failure in Network Interface Card

  
  

1. 서비스 네트워크 장애는 장애가 발생한 부분을 MCCS 웹 콘솔에서 확인할 수 있습니다In the MCCS web console, you can check in which part of service network, trouble has occurred.
2. MCCS checks network cable disconnection of server where network failure occurred, and whether ping timeout occurs from network.
   
3. If IP address resource is the cause of the failure, user should check on the network switch or network interface card.
   When physical parts related to network is back to normal, select 'Clear Fault' from the MCCS console and remove fault mark in order to re-enable the failover function. 
   
4. 자동으로 장애 표시를 제거하려면 그룹 속성의 AutoFaultClearTime에 0보다 큰 값을 설정하면 됩니다If you want the sign of failures to be removed automatically, enter a positive number in AutoFaultClearTime of the group attribute. 
   
   

• Heartbeat Network Fault

  핫빗은 노드 상호간의 상태를 동기화하고 장애 상태를 결정하는 중요한 역할을 하기 때문에 반드시 이중화되어 있어야 합니다. 이중화된 핫빗 네트워크 중에서 어느 하나라도 장애가 발생하면 장애 내용은 로그창에 표시 됩니다. 
  하지만 MCCS 웹 콘솔 에는 아무런 변화가 나타나지 않습니다. 이것은 운영 서버와 대기 서버에는 아무런 문제가 없다는 것을 뜻합니다Heartbeat should be dualized because it plays a very important role of synchronizing the inter node status and determining the condition of failure. If any one of the dualized heartbeat network fails, the details of failure is displayed in the log window.
  However, the MCCS web console has no changes. It means that the operation server or the standby server has no problems. 

  At this point, when failure occurs in active server and needs to failover to the standby server, MCCS will use redundant normal heartbeat network to failover.
  If all the redundant heartbeat is disconnected, MCCS will use the service network as heartbeat line.

...

1. Heartbeat failure can be checked on MCCS log, Window System log. If failure occurs in heartbeat line, server operator should check on the TCP/IP of server, physical connection check on the heartbeat through ping test.
2. If it is an abnormal situation, check on card, cable connection or cable disconnection and clear the cause of the failure.
   
   

• Replication (Mirroring) Network Failure

  When failure occurs in replication network, data cannot be replicated and it will be shown as 'Paused' in mirror disk resource of MCCS console.

  
  [Figure] Failure in Replicated Network

  
  

...

1. Network switch failure can be checked on MCCS log, Window System log. If failure occurs in service network connection, server operator should check on the TCP/IP of server, physical connection check on the service network through ping test.
   
2. 자동으로 장애 표시를 제거하려면 그룹 속성의 AutoFaultClearTime에 0보다 큰 값을 설정하면 됩니다If you want the sign of failures to be removed automatically, enter a positive number in AutoFaultClearTime of the group attribute.
3. Please get the supports regarding the recovery of Network switch failure through manufacturer.
   
   

Disk Failure

Source Disk Failure

• 소스 디스크 장애 Source Disk Failure

  If failure occurs in disk resource of active server, MCCS GUI will show the failure. MCCS will failover to the standby server since it is impossible to Read/Write in the disk.

   

  [Figure] Failure in Mirror Disk

  
  

1. Availability of disk monitoring of MCCS are as below.
2. Periodic read/write test on the disk.
Determines whether drive letter exists in the disk.
디스크 장애 발생 요인은 다음과 같은 경우가 있을 수 있습니다. 위의 문제가 해결 된 후에 운영 체제는 변경된 디스크를 다시 인식합니다.이후 DataKeeper에서 동기화를 진행합니다.
3. 디스크 컨트롤러 문제 하드웨어 자체의 문제는 해당 업체에서 해결해야 합니다.
물리적인 디스크 문제 하드웨어 자체의 문제는 해당 업체에서 해결해야 합니다.
미러 리소스에서 에서 동기화가 진행되지 않으면 미러디스크 리소스를 삭제한 후에 다시 생성해야 합니다. 단, 삭제시 리소스만 삭제가 아니라 생성된 미러까지 삭제 하고 다시 생성해야 합니다.
• Periodic read/write test on the disk.
• Determines whether drive letter exists in the disk.
  
  
4. Disk failure can be caused by the following. After resolving the above issues, the OS will detect the newly changed disk again. After that, Datakeeper will proceed with resynchronization.
   • Disk controller problems or H/W problems should be fixed by the manufacturers.
   • Physical disk problems or H/W problems should be fixed by the manufacturers.
     
     
5. If the mirror disk does not perform synchronization, delete the mirror disk resource and try to create it again. But when you delete the resource, you must also delete the created mirror and create them again. 
   
   

• Target Disk Failure

  If failure occurs in disk resource of active server, MCCS UI will show as 'Paused'. It does not affect the operating service of active server.

  
  [Figure] Failure in Target Disk

  
  

1. MCCS에서 타깃 디스크에 대한 장애 감지는 해당 디스크의 드라이브 문자가 있는지 없는지 만을 판단합니다.
2. 디스크 장애 발생 요인은 다음과 같은 경우가 있을 수 있습니다.위의 문제가 해결 된 후에 운영 체제는 변경된 디스크를 다시 인식합니다.이후 DataKeeper에서 동기화를 진행합니다.
   • 디스크 컨트롤러 문제 하드웨어 자체 문제는 해당 업체에서 해결해야 합니다.
   • 물리적인 디스크 문제 하드웨어 자체 문제는 해당 업체에서 해결해야 합니다.
3. 미러 리소스에서 에서 동기화가 진행되지 않으면 미러디스크 리소스를 삭제한 후에 다시 생성시도합니다. 단, 삭제시 리소스만 삭제가 아니라 생성된 미러까지 삭제 하고 다시 생성해야 합니다When MCCS detects failures of the target disk, it will only determine whether the disk has drive characters.
4. Disk failure can be caused by the following. After resolving the above issues, the OS will detect the newly changed disk again. After that, Datakeeper will proceed with resynchronization.
   • Disk controller problems or H/W problems should be fixed by the manufacturers.
   • Physical disk problems or H/W problems should be fixed by the manufacturers.
5. If the mirror disk does not perform synchronization, delete the mirror disk resource and try to create it again. But when you delete the resource, you must also delete the created mirror and create them again.  
   
   

• Split Brain of Mirror Disk Resource 

  This happens rarely but mirror disks identifies as source on both servers. This happens in the process of changing from existing source cannot be changed to target source.
  Both servers will try to synchronize the data and that cause the split brain. Split brain occurs in the situation as shown below.

1. Failover happens due to a failure in source server (A).
   
2. MCCS sends a 'SWITCHOVERVOLUME' command.
   
3. Without deleting the existing source volume, target source (B) has switched to source.
4. Reboot the original source server(A).
5. After booting the original source server(A), check in the role of target server(B).
6. If target server(B) is source, it determines itself that failover happened normally and MCCS tries to switch the original server (Node A) to target.
7. Role cannot be checked due to the problem in mirror network or other errors. (Process of 5 and 6 get failed)
8. Mirror role of both servers become source.
   
   

   In this case, icon of mirror disk resource is shown as put on one another in MCCS Console and attribute value of 'MirrorRole' on both servers are 'Source'.
   When this happens, role of mirror disk should be changed manually, and after the change, re-synchronization process happens.
   There is a way to change the role of mirror disk, using MCCS Console.

•  MCCS 웹 콘솔을 사용해서 스플릿브레인을 해결하는 방법

...

• How to resolve the split brain issues by using the MCCS web console

1. Check the resource attribute view.
   
   [Figure] Verify SplitBrain of MirrorDisk

   
   

2. 미러관리 창을 확인합니다Check the mirror management view.

   
   [그림] 미러디스크 스플릿 브레인 확인Figure] Checking Mirror Disk Split Brains

    

   Warning

   1) 양노드의 MirrorRole은 The both nodes' MirrorRole is Source, MirrorState은 and their MirrorState is MIRROR_PAUSED상태가 됩니다. 2) 미러디스크의 TimeAquiredSourceRole을 확인합니다. (TimeAquiredSourceRole은 시스템의 시간이므로 최신 데이터의 유무를 결정할 수 있는 절대값은 아닙니다) 3) 스플릿 브레인이 발생했을 때 발생하는 로그가 출력됩니다.  (윈도우즈 이벤트 오류Check the mirror disk's TimeAquiredSourceRole. (TimeAquiredSourceRole is the system time. So, it is not the absolute value used to determine whether it is the latest data.) 3) When a split brain occurs, the log will be displayed.  (Windows event error: An invalid attempt to establish a mirror occurred. Both systems were found to be Source.  Local Volume: F Remote system: 200.200.124.49 Remote Volume: F The mirror has been paused, or left in its current non-mirroring state.  Use the DataKeeper User Inteface to resolve this Split Brain condition.) 4) 미러관리 창에서 미러 상태가 In the mirror management window, the mirror condition is set to 'SPLIT' 상태 입니다.

3. In the Group tab of the configuration tree, right click mirrordisk resource and you can select the source node when you place the cursor on the "Resolve Split Brain".
   
   [그림] 스플릿 브레인 해결  선택
   스플릿브레인에 대한 설명창이 출력됩니다Figure] Split Brain Resolving Selected
   
   
4. Display the window to explain split brains.
    
   [그림] 소스 노드 선택에 대한 내용 확인
   소스노드를 선택합니다Figure] Checking the Source Node Selection
   
   
5. Select the source node.
   
   [그림] 소스 롤 노드 선택
   선택한 소스노드에 대해 다시 한번 확인합니다Figure] Source Roll Node Selection
   
   
6. Recheck the selected source node.
   
   [그림] 소스 노드 선택 다시 확인
   스플릿해결 중인 화면입니다Figure] Rechecking the Source Node Selection
   
   
7. Split brains problems being resolved.
   
   [그림] 스플릿 브레인 해결 화면
   스플릿해결 완료 화면입니다Figure] Split Brain Resolved
   
   
8. Resolving split brains problems is finished.
   
   [그림] 스플릿 브레인 해결 완료 화면
   선택한 노드가 소스노드가 되고 미러디스크의 상태는 MIRRORING 상태로 바뀌게 됩니다Figure] Resolving Split Brain Finished
   
   

9. The selected node becomes the source node and the mirror disk condition is changed to MIRRORING. 
   
   [그림] 스플릿 브레인 해결Figure] Split Brain Resolved
   
   

   Warning
   노드 B 의 변경된 정보는 모두 덮어써지게 됩니다The changed information of node B will be all overwritten.

    

External Storage Failure

외장 디스크의 연결 경로 및 디스크에 장애가 발생하면 해당 디스크의 Read/Write가 불가능하므로 MCCS는 장애를 표시하고 페일오버를 진행합니다When the external disk fails or has a bad connection path, you cannot read/write the disk. So, MCCS will display the sign of failure and proceed with a failover.

[Figure] Failure in Shared Disk

...

1. External storage failure can be checked through MCCS log, System log.

2. If there is a problem in external storage, service is stopped until the storage recovers. Therefore, storage should be recovered in a short period of time or it should be replaced to other one (back up storage).

3. Problems related to the external storage should be dealt with the vendor.

4. When the server of external storage connection and disk where failure occurs is back to normal, Server should be rebooted so that MCCS Kernel Driver can identify the recovered environment.

5. Also, redundancy measures should be solved from storage vendor.
   

NetBIOS

...

Failures

Use Direct-Hosted SMB

SMB which is supported on Windows 2000 or later uses Direct-Hosted. This feature is support directly file sharing service without NetBIOS interface.
To resolve name resolution for an IP address, DNS lookup occurs and not used NetBIOS name resolution.

...

Windows Server service supports sharing the file, print, and named pipe over the network for this computer. If this service is stopped, cannot use these features. If this is disabled, the following services have a dependency on this service will not be able to start.

When the server service is stopped, you must be careful.

...

When maintaining the state where DUID is not solved after registering agent

레터를 정의하고 활성화를 요청해야 main.json에 해당 레터에 연결된 DUID 정보가 기록됩니다You must first define the letter and request activation before the information of DUID connected to the letter is recorded in main.json.

When delete agent

SCSI Lock 에이전트가 삭제될때 예약을 해제합니다. 따라서 예약대상 공유디스크가 상대 노드에서 사용될 가능성을 염두에 두고 삭제를 해야 합니다. 즉 삭제할 경우에는 상대 노드를 다운시킨 후 작업하십시요.Reservation is canceled when an SCSI Lock agent is deleted. When you delete it, you must consider the fact that the shared disk to be reserved can be used at the other node. In other words, when you delete it, you must make sure the other node is down.

Ways to collect support files

When problems occur in MCCS, support file must be collected to collect log and preference information.
There are 2 ways to collect support file.

웹 콘솔 통해서 수집하는 방법

...

How to collect by using the web console

1. In the MCCS web console, click 'File' on the menu bar to collect support files.
   
   [Figure] Support file Collect Icon 1
   
   
2. Support files can be collected by clicking the toolbar shown in the figure below.
   
   [Figure] Support File collect icon 2
   
   
3. 서포트 파일을 수집할 노드의 선택과 이전에 받은 서포트파일을 다시 받을 수 있습니다You can select a node to collect support files from and get the previous support file again.
   
   [그림] 서포트 파일 노드 선택 및 이전 서포트 파일 선택 여부Figure] Support File Node Selection and Previous Support File Selection
   
   

4. Click 'OK' button and support file is collected.
   Image Modified
   [그림] 서포트 파일 수집 중 화면Figure] Support Files Being Collected

    

   Info
   로그파일의 용량과 네트워크의 상태에 따라서 몇 분이 걸릴수도 있습니다.

   아래와 같이 다운로드 창이 열리게 되고 다운받으시면 됩니다

   It may take several minutes depending on the log file capacity and the network condition.

5. As shown below, you can download it from the download window.
   
   
   
   [그림] 서포트 파일 수집 확인Figure] Support Files Collection Checked
   
   
   
   

Collecting file using script files

...