How to use EMS(Emergency Message Service)

MCCS has a bundled product called EMS(Emergency Message service) that automatically sends SMS to the defined admin members in charge of critical events.
In addition, since console is web-based management, whenever an error or fault occurs, it can be managed anywhere that has internet service. Plus, Failures records in the past, management, reporting are all very easy to use.
Please refer to "EMS User Guide and EMS Agent Installation Guide" for more details about EMS system.

EMS Component

EMS Agent

It is a program installed in the server to connect with EMS server.

EMS Server

It is an installed server program from the product provider company of MCCS.

EMS Workflow

Save Log

EMS Agent saves logs.
EMS server can specify logs by its type using 'LogType' attribute as shown below.

H

It saves the logs related to HA (MCCS).
(It can only specify file monitor.)

A

It saves logs related to application.
(It can only specify file monitor.)

S

It saves event log of Windows system.
(It can only specify Windows event monitor.)

P

It saves log related to process.
(It can only monitor specified file.)

Log Analysis

EMS Server users can set failure level of the system that wants to receive EMS service.
EMS server uses failure level that is set to filter EMS Agent system of operating server and analysis log to determine if it is a failure

SMS Notification

After failure monitoring for given filter is checked, EMS will send the SMS to the system operator and MCCS server operator so that it can be dealt quickly.

After connecting to EMS server, analysis cause of failure

System operator and MCCS service operator can access to the EMS server where anywhere with internet connection to check on the log and analyze the cause.
In addition, in case of manufacturing customer, it provides a centered monitoring system for all the servers in the factory and also provides a statistic of periodical failure type and trouble-shooting solutions.
The following graph is the workflow if EMS system.

[Figure] Workflow of EMS System

Control Monitoring of EMS Server Consolidated Web-based dashboard of EMS Server

Following is a part of consolidated web-based dashboard of EMS Server.
Servers with failures are shown in red, servers that had failure and had notified to the server operators are shown in yellow, and servers that operate normally are shown in blue.
Users registered in EMS server are the only ones that can monitor the dashboard. 

[Figure] Redundant server monitoring view of EMS system

[Figure] Statistic view of EMS system

Server Failure

This is the case of system being rebooting or shut down because of conflicts of each device (NIC, Raid Controller), kernel driver problem of other application.

Active Server Failure

1.  There is no difference in the MCCS role resulting from abnormal or normal termination of the server. MCCS will perform a failover to the standby server when the operation server fails.
   In the node management menu on the right side of the screen, select the server. You can check the details of failures in the 'Resource Status' & 'Resource Dependency' screens.
   Since data cannot be replicated due to the server failure, will be shown in the mirror disk resource.
   • Normal Termination of a system
     This is a case where user selected 'system shutdown' in operating systems.
   • Abnormal Termination of a system 
     This is a case where system is terminated or rebooted due to an unexpected situation or blue screen.
   
   [ Figure] Failure in Active Server
   
   
2. Server operators check on the failure and put the server back to normal.
3. After checking on the mirror role of two servers when server with the failure is rebooted, switch the server with the failure as replication target and proceed partial resync.

Standby Server Failure

1. MCCS will show the failure when failure occurs in standby server.
2. Data replication will be paused until standby server is back to normal.
   
   [ Figure] Failure in Standby Server
   
   
3. If I/O keep happens, data is impossible to replicate and mirror disk will be shown as 'Paused'()
4. If there is no I/O, icon of mirror disk has no change but failure messages related mirror disk exists in MCCS log.
5. Even if the standby server failed, it does not affect operation. But as there is no server to perform failover to, the server operator must check the trouble in the MCCS web console and make sure that the standby server is normalized in time.
6. When standby server is back to normal, it will recover from 'Paused' to 'Normal' and  icon will be disappeared.

Application, Process and Window Service Failure

Active application, process and window service resources are operated by 4 elements below.

• MonitorInterval
  Monitors the resource with interval set value. (Default Value=10sec)
• MonitorTimeout
  If there is no reply as much as the set value, it is considered as a failure. (Default Value=10sec)
• RestartLimit
  It will restart the application resource as the set value. (Default Value=0)
• OnlieTrustTime
  It re-sets the time of number of resource restarting number.It is the time to reset the frequency of the resource to restart. (Default Value=600sec)
  Attributes above are the set value of the registered being added the resource, and users can check or change the values through Resource Attribute view of MCCS console. 
  
  [Figure] Resource attribute value Edit
  
  

1. MCCS periodically monitors the resources referring  'MonitorInterval'.
2. If there is no response as the time set in 'MonitorTimeout', it is considered as a failure.
   
3. If there are no response after sending the command as the number set in 'RestartLimit', MCCS will failover the group which resource belongs to.
4. If the resource stays in normal state within the time limit set by 'OnlineTrustTime'. MCCS will initialize the attribute value of 'RestartLimit'. This is to ensure restart number when failure occurs in a resource.
   
5. If there is a failover due to a failure in the resource, server operator checks on the problem and put it back to normal.
   
6. In the MCCS web console, a user can see where the trouble occurs. After a user checks the trouble area, they must remove the Trouble sign, so that the failover function can be activated again. 
7. After checking on the mirror role of two servers when server with the failure is rebooted, switch the server with the failure as replication target and proceed partial resync.
   
   
   [Figure] Failure in Resource Clear
   
   

Network Failure

Network failure happens when network connection has problem, such as network switch or network interface card is broken or disconnection in network cable, or ping timeout of some network and so on.

• Service Network Failure

  If failure occurs in service network of active server, the fault mark will be shown on the network interface card resource or IP address of the node in MCCS web console, and will failover to the standby server.

  
  [Figure] Failure in Network Interface Card

  
  

1. In the MCCS web console, you can check in which part of service network, trouble has occurred.
2. MCCS checks network cable disconnection of server where network failure occurred, and whether ping timeout occurs from network.
3. If IP address resource is the cause of the failure, user should check on the network switch or network interface card.
   When physical parts related to network is back to normal, select 'Clear Fault' from the MCCS console and remove fault mark in order to re-enable the failover function. 
4. If you want the sign of failures to be removed automatically, enter a positive number in AutoFaultClearTime of the group attribute. 
   
   

• Heartbeat Network Fault

  Heartbeat should be dualized because it plays a very important role of synchronizing the inter node status and determining the condition of failure. If any one of the dualized heartbeat network fails, the details of failure is displayed in the log window.
  However, the MCCS web console has no changes. It means that the operation server or the standby server has no problems. 

  At this point, when failure occurs in active server and needs to failover to the standby server, MCCS will use redundant normal heartbeat network to failover.
  If all the redundant heartbeat is disconnected, MCCS will use the service network as heartbeat line.

          [Figure] Failure in Heartbeat

1. Heartbeat failure can be checked on MCCS log, Window System log. If failure occurs in heartbeat line, server operator should check on the TCP/IP of server, physical connection check on the heartbeat through ping test.
2. If it is an abnormal situation, check on card, cable connection or cable disconnection and clear the cause of the failure.
   
   

• Replication (Mirroring) Network Failure

  When failure occurs in replication network, data cannot be replicated and it will be shown as 'Paused()' in mirror disk resource of MCCS web console.

  
  [Figure] Failure in Replicated Network

  
  

1. Replication network failure history can be checked on MCCS log, Window System log. If failure occurs in replication network, server operator should check on the TCP/IP of server, physical connection check on the replication network through ping test.
2. If it is an abnormal situation, check on card, cable connection or cable disconnection and clear the cause of the failure.
   
   

• Single Network Switch Fault

  When failure occurs in network switch connected to Public Network where it is configured by single network switch, all the resources in active and standby server will be taken offline, resources where failure occurs will show as 'fault'.

  
  [Figure] Failure in Network Switch

  
  

1. Network switch failure can be checked on MCCS log, Window System log. If failure occurs in service network connection, server operator should check on the TCP/IP of server, physical connection check on the service network through ping test.
2. If you want the sign of failures to be removed automatically, enter a positive number in AutoFaultClearTime of the group attribute.
3. Please get the supports regarding the recovery of Network switch failure through manufacturer.
   
   

Disk Failure

Source Disk Failure

• Source Disk Failure

  If failure occurs in disk resource of active server, MCCS web console will show the failure. MCCS will failover to the standby server since it is impossible to Read/Write in the disk.

   

  [Figure] Failure in Mirror Disk

  
  

1. Availability of disk monitoring of MCCS are as below.
   • Periodic read/write test on the disk.
   • Determines whether drive letter exists in the disk.
     
     
2. Disk failure can be caused by the following. After resolving the above issues, the OS will detect the newly changed disk again. After that, Datakeeper will proceed with resynchronization.
   • Disk controller problems or H/W problems should be fixed by the manufacturers.
   • Physical disk problems or H/W problems should be fixed by the manufacturers.
     
     
3. If the mirror disk does not perform synchronization, delete the mirror disk resource and try to create it again. But when you delete the resource, you must also delete the created mirror and create them again. 
   
   

• Target Disk Failure

  If failure occurs in disk resource of active server, MCCS web console will show as 'Paused'. It does not affect the operating service of active server.

  
  [Figure] Failure in Target Disk

  
  

1. When MCCS detects failures of the target disk, it will only determine whether the disk has drive characters.
2. Disk failure can be caused by the following. After resolving the above issues, the OS will detect the newly changed disk again.
   After that, Datakeeper will proceed with resynchronization.
   • Disk controller problems or H/W problems should be fixed by the manufacturers.
   • Physical disk problems or H/W problems should be fixed by the manufacturers.
3. If the mirror disk does not perform synchronization, delete the mirror disk resource and try to create it again. But when you delete the resource, you must also delete the created mirror and create them again.  
   
   

• Split Brain of Mirror Disk Resource 

  This happens rarely but mirror disks identifies as source on both servers. This happens in the process of changing from existing source cannot be changed to target source.
  Both servers will try to synchronize the data and that cause the split brain. Split brain occurs in the situation as shown below.

1. Failover happens due to a failure in source server (A).
2. MCCS sends a 'SWITCHOVERVOLUME' command.
3. Without deleting the existing source volume, target source (B) has switched to source.
4. Reboot the original source server(A).
5. After booting the original source server(A), check in the role of target server(B).
6. If target server(B) is source, it determines itself that failover happened normally and MCCS tries to switch the original server (Node A) to target.
7. Role cannot be checked due to the problem in mirror network or other errors. (Process of 5 and 6 get failed)
8. Mirror role of both servers become source.
   
   

   In this case, icon of mirror disk resource() is shown as put on one another in MCCS web console and attribute value of 'MirrorRole' on both servers are 'Source'.
   When this happens, role of mirror disk should be changed manually, and after the change, re-synchronization process happens.
   There is a way to change the role of mirror disk, using MCCS web console.

•  How to resolve the split brain issues by using the MCCS web console

1. Check the resource attribute view.
   
   [Figure] Verify SplitBrain of MirrorDisk

   
   

2. Check the mirror management view.

   
   [Figure] Checking Mirror Disk Split Brains

    

   1) The both nodes' MirrorRole is Source, and their MirrorState is MIRROR_PAUSED. 2) Check the mirror disk's TimeAquiredSourceRole. (TimeAquiredSourceRole is the system time. So, it is not the absolute value used to determine whether it is the latest data.) 3) When a split brain occurs, the log will be displayed.  (Windows event error: An invalid attempt to establish a mirror occurred. Both systems were found to be Source.  Local Volume: F Remote system: 200.200.124.49 Remote Volume: F The mirror has been paused, or left in its current non-mirroring state.  Use the DataKeeper User Inteface to resolve this Split Brain condition.) 4) In the mirror management window, the mirror condition is set to 'SPLIT'.

   
   
   
   

3. In the Group tab of the configuration tree, right click mirrordisk resource and you can select the source node when you place the cursor on the "Resolve Split Brain".
   
   [Figure] Split Brain Resolving Selected
   
   
4. Display the window to explain split brains.
    
   [Figure] Checking the Source Node Selection
   
   
5. Select the source node.
   
   [Figure] Source Roll Node Selection
   
   
6. Recheck the selected source node.
   
   [Figure] Rechecking the Source Node Selection
   
   
7. Split brains problems being resolved.
   
   [Figure] Split Brain Resolved
   
   
8. Resolving split brains problems is finished.
   
   [Figure] Resolving Split Brain Finished
   
   

9. The selected node becomes the source node and the mirror disk condition is changed to MIRRORING. 
   
   [Figure] Split Brain Resolved
   
   

   The changed information of node B will be all overwritten.

    

External Storage Failure

When the external disk fails or has a bad connection path, you cannot read/write the disk. So, MCCS will display the sign of failure and proceed with a failover.

[Figure] Failure in Shared Disk

1. External storage failure can be checked through MCCS log, System log.

2. If there is a problem in external storage, service is stopped until the storage recovers. Therefore, storage should be recovered in a short period of time or it should be replaced to other one (back up storage).

3. Problems related to the external storage should be dealt with the vendor.

4. When the server of external storage connection and disk where failure occurs is back to normal, Server should be rebooted so that MCCS Kernel Driver can identify the recovered environment.

5. Also, redundancy measures should be solved from storage vendor.

NetBIOS Failures

Use Direct-Hosted SMB

SMB which is supported on Windows 2000 or later uses Direct-Hosted. This feature is support directly file sharing service without NetBIOS interface.
To resolve name resolution for an IP address, DNS lookup occurs and not used NetBIOS name resolution.

Transmit and receive operation

• Basically redirector(WorkStation Service) is transmit call to NetBIOS devices and SMB devices, file server receive call about NetBIOS devices and SMB devices.
• File server SMB device is waiting to receive call TCP port 445 preferentially not in TCP over NetBIOS 139.
• If SMB session fails to listen TCP port 445, use TCP over 139(NetBT). If both of TCP port 445 and TCP over 139(NetBT) are failed, session will be failed.

Namely, if you want to work with the DNS server while using NetBIOS agent, most of clients are connected by Direct-Hosted SMB.

Related Cache Flush

When verify agent action, related cache will be flushed.

Flush a NetBIOS table cache

netbtstat -R

Flush A DNS cache

ipconfig /flushdns

Flush an ARP cache

arp -d

Turn off firewall settings

Turn off NetBIOS communication related are : the destination port number.

Turn off DNS Server update, WINS Server update communication related are : the destination port number.

Considerations when Workstation Service is stopped

The Workstation Service on Windows Services creates and maintains client connection for the remote server using SMB protocol.
If this service is stopped, cannot keep the connections. if this service is disabled, then this connection explicitly using the service cannot be started.
When the workstation service is stopped, you must be careful.

Considerations of Server Service stop

Windows Server service supports sharing the file, print, and named pipe over the network for this computer. If this service is stopped, cannot use these features. If this is disabled, the following services have a dependency on this service will not be able to start.

When the server service is stopped, you must be careful.

When file sharing is not working

At first, please verify file sharing is working as original NetBIOS computer name except virtual name.
On the client, please verify that access to node file on a regular basis is working using command like dir, start, explorer or net view.
Please verify as DIR command.
DIR command is run by following syntax.

dir \\virtual_name\shared_folder 

Please verify as START command.
START command is run by following syntax.

start \\virtual_name

Please verify as EXPLORER command.
EXPLORER command is run by following syntax.

explorer \\virtual_name

Please verify as NET VIEW command.
NET VIEW command is run by following syntax.

net view virtual_name

Your computer's file and print sharing lists are created. On the specified computer, there are no file or print shares available, "there are no entries in the list" message.
When the client isn't refreshed the mapping information between virtual name and real IP address after failover occurs, the client's NetBIOS cache is not communication for a few minutes until flushed.
This case will be happened when you use WINS server. Therefore the clients program is needed to be cluster aware in this case.

SCSI Lock Failure

When interlock with volume manager using SCSI3-PR

Volume Manager (Ex: something like SFW of Symantec that has SCSI3-PR reservation function) can be used with SCSI Lock agent.

When check if SCSI-PR is supported

To check of the disk supports SCSI-3PR function, PR type can be checked using scsicmd.cmd command.

When cannot find sg_scan.exe or sg_persist.exe pass

Check if the command exist in %MCCS_HOME%/bin.

When interlock with shared disk

When interlock shared disk agent and SCSI Lock agent, check if the shared disk agent works normally and then register SCSI Lock agent.
The purpose of disk of SCSI Lock agent is to use as a LOCK device in hardware perspective, not the contents of the disk. Therefore, size of disk can be small and it is not protected.

When registration key error occurs

Remove Reservation key and registration key using scsicmd.cmd -c or -cf command and re-set. Before registering resource, check if there is any registered key and if there is, remove the key first before registering.
Note that the current key is set automatically by its MAC address. It uses the first adapter among the network adapters. This key is automatically recorded in setting file. If key does not exist in setting file then new key is not created.

When various letters exist in one disk and when register one letter, other letters cannot access

SCSI Lock disk supports basic disk and single letter. Please do not use the disk that uses dynamic disk or multiple volume(use one LUN to configure various partition).

When maintaining the state where DUID is not solved after registering agent

You must first define the letter and request activation before the information of DUID connected to the letter is recorded in main.json.

When delete agent

Reservation is canceled when an SCSI Lock agent is deleted. When you delete it, you must consider the fact that the shared disk to be reserved can be used at the other node. In other words, when you delete it, you must make sure the other node is down.

Ways to collect support files

When problems occur in MCCS, support file must be collected to collect log and preference information.
There are 2 ways to collect support file.

How to collect by using the web console

1. In the MCCS web console, click 'File' on the menu bar to collect support files.
   
   [Figure] Support file Collect Icon 1
   
   
2. Support files can be collected by clicking the toolbar shown in the figure below.
   
   [Figure] Support File collect icon 2
   
   
3. You can select a node to collect support files from and get the previous support file again.
   
   [Figure] Support File Node Selection and Previous Support File Selection
   
   

4. Click 'OK' button and support file is collected.
   
   [Figure] Support Files Being Collected

    

   It may take several minutes depending on the log file capacity and the network condition.

5. As shown below, you can download it from the download window.
   
   
   
   [Figure] Support Files Collection Checked
   
   
   
   

Collecting file using script files

Script file is located as below:

%MCCS_HOME%\bin\Support\support.cmd

Collected support file is created in the following directory.

%MCCS_HOME%\support-%COMPUTERNAME%\%COMPUTERNAME.zip