| Page Tree | root | 5. Working
|---|
| Table of Contents |
Operatings
When the configuration file is ready, it moves to the step of operating the replication. Operation examples such as resource start, stop, synchronization/replication, and transfer are explained in sequence.
Replication operation is performed on a per resource basis.
Resource Up
Before starting the resource for the first time, you must perform an initialization process that creates metadata. Meta data initialization is performed only once when the resource is initially started.
fsradm metacreate[resource name] {–force | -f}
Load the resource for which metadata was created as a replication target.
fsradm attach [resource name]
The loaded resource is in a neutral state that has not yet tried to connect with the other node. Try to connect to the replication through the connect command.
fsradm connect [resource name] [peer node name]
You can perform attach and connect sequentially through the up command. Usually, you use the up command to start up resources.
fsradm up [resource name]
Resource Down
You can disconnect the resource with the disconnect command.
fsradm disconnect [] [resource namepeer node name]
Detach disconnected resources.
fsradm detach []resource name
You can disconnect and detach sequentially with the down command. Usually, the down command is used to stop the resource.
fsradm down []resource name
Synchronizations
Initial Sync
When a replication connection is established by starting up the resources of both the source and target nodes, it waits in the state before starting synchronization. It is an equilibrium state where the direction of initial synchronization is not determined. In this state, the initial synchronization starts by promoting the resource role of the node to be the source to Primary. As soon as synchronization starts, if there is a change in the source-side data, the change is also replicated in real time. FSR essentially performs synchronization and replication simultaneously.
The command to promote a resource is:
fsradm primary [resource name]
| Info |
|---|
| Switching the role of a resource to primary is called a promotion, and switching the role to secondary is called a demotion. |
Local files at the time of initial synchronization are defaulted to the Inconsistent state, which is not consistent with both nodes, so promotion is denied by default. During initial promotion, the user must explicitly inform the user that the resource will be sourced through forced (-f option) promotion.
| Code Block |
|---|
c:\>fsradm primary r0
declined
r0: not up to date
c:\>fsradm primary r0 -f
done |
When the forced promotion is successful, the source node changes its file status to UpToDate and starts initial synchronization with target nodes connected to it.
Initial synchronization is performed for the entire fileset, but when synchronization is performed again after synchronization is completed, partial synchronization is performed only for the changes on the source side. For example, if the replication connection is disconnected and then reconnected after the initial synchronization, it proceeds to partial synchronization.
While synchronization is in progress, the target file status is Inconsistent, and when synchronization is complete, the source and target are in the UptoDate status. Inconsistent status is not up-to-date, so it is desirable to keep the Inconsistent status as short as possible in terms of replication operations.
Manually Sync
If you need to synchronize manually during operation, this is done via the invalidate-remote command. This command synchronizes peer nodes with local as a source.
| Code Block |
|---|
c:\>fsradm invalidate-remote r0 |
The invalidate command is a command that synchronizes with the peer node as a source.
| Code Block |
|---|
c:\>fsradm invalidate r0 |
Automatic synchronization
Except for manual manipulation, synchronization is all automatic by default once a mirroring connection is established. FSR does not track changes between files or keep information about them in advance for file synchronization (this is not practical in the realm of file replication). It identifies and synchronizes differences between files on the fly at the time the synchronization is performed.
How file differences are compared
Sync has several customizable behaviors, depending on how aware it is of differences between files on both sides. The behavior makes a difference in how fast the sync performs, so be sure to configure it to a reasonable value.
| Info | |
|---|---|
|
The values you specify for comparison_level have the following meanings
- 0: Full sync, no comparison. It synchronizes the entire file unconditionally, so it takes longer, but it provides the best guarantee of matching.
- 1: Attribute comparison sync, compares the differences in the attributes of the files and synchronizes them. This is the preferred method because it synchronizes quickly.
- 2: Hash comparison synchronization, get the hash values of the files, compare them, and synchronize if there is a difference. The comparison unit of the hash works efficiently by dividing the file into certain blocks for comparison and synchronizing only the parts that have a difference. The hash algorithm can be specified by selecting CRC32, MD5, SHA1, SHA256, SHA512 as the HASH_TYPE value. When replicating DB files, hash comparison synchronization should be specified as the default.
However, hash comparison synchronization may take precedence over attribute comparison synchronization (1) in the following situations.
- When the target's file is unlocked
- When the source side fails or is forced to reboot (crashed primary) due to an operational failure.
In unintended operational and failure-like situations, comparison via file attribute values may not always be consistent, so hash comparison synchronization is used to ensure consistency between files.
Bandwidth settings
| Info |
|---|
{ ... "network": { "sync_ratio" : "7:3", "sync_min" : 100M, "sync_max" : 1G }} |
The replication and synchronization bandwidth of your replication network should be coordinated in advance and configured to the appropriate numbers or ratios. A typical ratio of 7:3 (7 replication, 3 synchronization) is a good starting point and can be adjusted to suit your network. Leaning toward replication as much as possible is good for local I/O performance.
You can set the synchronization band between sync_min and sync_max, with sync_min being the minimum guaranteed synchronization band. The unit is bps.
Replications
As the secondary node is promoted and synchronization starts, if a real-time change occurs in the data of the source node, the change is automatically reflected in parallel. Replication is defined as an action that reflects real-time changes in local data to a target in real time, and proceeds from the primary node to the secondary node.
Even during synchronization and replication, the role of each node can be manually changed by user command, and replication is stopped when the primary node is demoted.
The command to demote the promoted resource is as follows.
fsradm secondary [resource name]
Replication is sourced from the node promoted to the Primary role, but synchronization occurs when synchronization is required regardless of role. Even if there are no changes to be replicated or replication is interrupted by demoting, if synchronization was in progress, synchronization will continue until completion.
| Info | ||
|---|---|---|
| ||
During replication after synchronization is complete, files that did not exist on the replication destination may suddenly be included in the replication destination path. These files are called missing files and can occur in the following operating situations.
In the first case, the FSR cannot capture Filesystem I/O for that file, it only receives the rename of the file path, so it cannot be processed as a duplicate. In this case, the FSR maintains the replication status once and at the same time performs synchronization for the missing files individually and processes them. In the case of omission due to the second exclusion pattern change, it is basically treated as resynchronization because only the replication target is changed without file system I/O operation. |
| Info | ||
|---|---|---|
| ||
Unlike missing files, orphaned files are defined as files left without any connection to the target's replication path. This doesn't happen in normal duplication situations, but it happens when there is unintentional file manipulation in a situation where the target file is not protected. When an orphaned file occurs, it is processed according to the FSR's orphaned file response policy, and basically, it is processed as a backup to a specific path of the target. You can also specify the option to process the deletion immediately without the need for backup. |
FileLock
Files copied to the target must be protected from write I/O other than the mirroring data received from the source. Otherwise, data consistency to maintain a duplicate copy is not guaranteed. In particular, when operating HA, the secondary file lock must be activated to protect data.
File lock is generally activated in the secondary and deactivated in the primary depending on the role of the resource to operate as a target file protection function.
File lock can be set automatically according to the role of the resource through the auto_file_lock option in the nodes section of the resource, or can be manually activated or deactivated through the fsradm lock or unlock command.
Auto Lock
The auto_file_lock option is enabled by default. When a resource's role is demoted, the files are locked by default. To unlock locked files, you need to promote the role of the resource or unlock it via the unlock command.
Locking is automatic, but unlocking is not.
Manual Lock
You can also manually operate file locking by disabling the auto_file_lock option. To operate file lock manually, you must separately execute the lock command and the demote command as follows and follow the command sequence.
| Code Block |
|---|
c:\>fsradm lock r0
done
c:\>fsradm secondary r0
done |
If the -l option is specified, the above two commands can be processed as one demotion command. The order of commands is the same as above, locking first and then demoting.
| Code Block |
|---|
c:\>fsradm secondary -l r0
done |
Conversely, during the promotion process, the lock is released after the primary command.
| Code Block |
|---|
c:\>fsradm primary r0
done
c:\>fsradm unlock r0
done |
It can be processed in a single promote command using the -u option.
| Code Block |
|---|
c:\>fsradm primary -u r0
done |
| Info |
|---|
|
| Info |
|---|
|
Switchover
A switchover is the manual exchange of access to resources from one system to another within a replication cluster. It is the process of demoting a source node and promoting a target node to the source node role to enable data for a service. Also known as manual switchover, the opposite of this is failover, which is an automatic response to a failure.
Demote the resource on the source node.
| Code Block |
|---|
c:\>fsradm secondary r0
done |
Promote the target node's resource.
| Code Block |
|---|
c:\>fsradm primary r0
done |
If the promotion is successful, the transfer is considered complete.
| Info | ||
|---|---|---|
| ||
|
Persist Role
Resource roles can be changed based on operational circumstances, but sometimes you may want to keep a role persistent. (FSR 1.2.4 and later)
A resource with persist-role set will continue to have the resource role explicitly specified(with the fsradm command) at the time of restart. This works in any situation where the replication service or system reboots, causing the resource to restart.
| Code Block |
|---|
{
...
"options": {
"persist-role": true,
...
}
} |
One-way Replications
If you always want only one-way replication/backup from the primary to the secondary node, without switching over, consider the target-only attribute on the standby node side. (FSR 1.2.4 and later)
- Set the persist-role attribute described above in the resource options section to fix the roles of the primary and secondary nodes.
- Set the target-only attribute on the standby node side to force the replication/synchronization direction from the primary node to the standby node only.
A target-only node is prohibited from having the source role in all replication/sync operations, including explicit commands, and can only have the target role. Any manual synchronization or promotion commands that act as a source role are blocked (but promotion is allowed on replication disconnect).
| Code Block |
|---|
{
...
"options": {
"persist-role": true,
...
}
"nodes": [
{
"name": "active",
...
},
{
"name": "standby",
"target-only": true
...
}
} |
| Info | ||
|---|---|---|
| ||
After disconnecting replication
|
Inquiry
Status
The status of the FSR can be queried using the fsradm status command.
| Code Block |
|---|
λ fsradm status all
r0 role:primary file:up_to_date pending:0 locked:false
node2 state:repl_source peer-state:repl_target role:secondary file:up_to_date
last-synced:2019-10-24T15:30:12+09:00
node3 state:connecting peer-state:unknown role:secondary file:unknown
last-synced:none
r1 role:secondary file:inconsistent pending:0 locked:false
node2 state:connecting peer-state:unknown role:secondary file:unknown
last-synced:none |
More status information can be retrieved by using the verbose output option.
| Code Block |
|---|
λ fsradm status -v
r0:node1 role:primary file:up_to_date pending:0 locked:false
last-promoted:2020-06-10T09:40:32+09:00
node2 state:repl_source peer-state:repl_target role:secondary file:up_to_date
repl-started:2020-06-10T09:40:32+09:00 last-synced:2020-06-10T09:40:33+09:00
node3 state:connecting peer-state:unknown role:secondary file:unknown
repl-started:2020-04-09T09:50:38+09:00 last-synced:2020-04-09T09:50:53+09:00 |
If you want to keep the status lookup, you can use the --watch(-w) and --interval(-i) options to monitor the status.
| Code Block |
|---|
λ fsradm status all -w -i 1
r0 role:secondary file:inconsistent locked:false
node2 state:established peer-state:established role:secondary file:inconsistent
last-synced:none
node3 state:connecting peer-state:unknown role:secondary file:unknown
last-synced:none
r1 role:secondary file:inconsistent locked:false
node2 state:connecting peer-state:unknown role:secondary file:unknown
last-synced:none
update every 1.0s. current executions: 84
press 'q' or 'ctrl+c' to quit... |
File Status
Indicates the replication status of the file to be replicated.
unknown unknown state. Represents the unknown file state of the unconnected partner node.
fileless detached status of replication target. The attach command switches to the attaching state.
attaching attaching status. If it fails during loading, If it fails during attaching, it becomes failed, or when loading is complete it becomes consistent or inconsistent.
detaching detaching the replication target. When detaching is complete, it becomes fileless.
failed the status indicating failure in the event of a replication configuration failure or file I/O error.
inconsistent the status of the files in the synchronization target or where the data sequence cannot be guaranteed. Basically, promotion is not possible (forced promotion is possible).
consistent State that guarantees data sequence. It is in an intermediate state and has a final transition to outdated or up_to_date.
outdated the data is available, but the old data. The status when the latest data cannot be received due to disconnection or pause in the replication target situation. Basically, promotion is not possible. (Forced promotion possible)
up_to_date latest data status. This is the primary or replication target.
Connection/Replication Status
The state until both nodes are connected is defined as the connection state, and the state after connection establishment is defined as the replication state. The following states are defined.
standalone A neutral state in which no connection is attempted, which is the initial connection state of the resource. It switches to the connecting state by the connect command.
disconnecting The connection is disconnected and is being cleaned up. Switches to standalone or connecting state.
connecting Connection attempting state. If an error occurs while attempting to connect, it becomes standalone, and if the connection is successful, it becomes connected. In actual implementation, accept and connect are attempted at the same time in the socket layer.
connected The connection is successful and you are authenticating to the replica network. If authentication is successful, it is established, and if authentication fails, it becomes standalone.
established This is the default state when the secondary connection is completed. It does not go directly to synchronization or replication. When promoted in this state, it becomes sync_source or repl_source, and when the opponent promotes, it becomes sync_target or repl_taret.
sync_source Synchronization source status. When synchronization is paused, the status is sync_source_paused, and when synchronization is completed, the status is repl_source. When synchronization between the secondary is completed, it is established.
sync_source_paused Synchronization source paused state. When synchronization resumes, it enters the sync_source state.
sync_target Sync target status. When synchronization is paused, it becomes sync_target_paused, and when synchronization is complete, it becomes repl_target. Synchronization between secondarys is completed in the Established state.
sync_target_paused Sync target paused state. When synchronization resumes, it will be in the sync_target state.
repl_sourceReplication source status. From this state, it switches to the established state when demoted, repl_source_paused when paused, and sync_source when starting synchronization.
repl_source_paused Replication source paused state. When replication resumes, it will be in the repl_source state.
repl_target Replication target status. In this state, when Primary is demoted, it becomes established, when paused, repl_target_paused, and when synchronization starts, it becomes sync_target.
repl_target_paused Replication target paused state. When replication resumes, it enters the repl_target state.
Performance
Performance can be queried through the fsradm perfmon command.
| Code Block |
|---|
c:\>fsradm perfmon r0 |
For inquiries about performance, you can use several options, such as printing the results on the console screen and checking them directly, or saving the query results as a file.
- --json <filename> JSON file path
- --csv <filename> CSV file path
- --display output to console screen
- --watch monitoring mode
- --interval inquery interval
Performance Indicator
Event
FSR can be notified of events defined by FSR through event subscription command. Event subscriptions allow you to track changes in status, such as files or connections, in real time.
| Code Block |
|---|
λ fsradm events r0
2020-06-12T12:42:39.295379 type=rpc state=connected
2020-06-12T12:42:41.685784 type=state node=node2 peer=node1 resource=r0 value=standalone
2020-06-12T12:42:41.685784 type=added node=node2 resource=r0
2020-06-12T12:42:41.685784 type=role node=node2 resource=r0 role=secondary
2020-06-12T12:42:41.685784 type=file_state node=node2 resource=r0 value=fileless
2020-06-12T12:42:41.728821 type=file_state node=node2 resource=r0 value=attaching
2020-06-12T12:42:41.744835 type=file_state node=node2 resource=r0 value=outdated
2020-06-12T12:42:41.774378 type=state node=node2 peer=node1 resource=r0 value=connecting |
For ease of event interpretation, json format output is supported, and additional options for synchronization status (--sync) and performance statistics monitoring (–perf) are supported.
| Code Block |
|---|
λ fsradm events --json r0
{"type":"rpc","timestamp":"2020-06-12T03:43:56.152358300Z","datas":{"state":"connected"}}
{"type":"state","timestamp":"2020-06-12T03:43:58.396422300Z","datas":{"node":"node2","peer":"node1","resource":"r0","value":"standalone"}}
{"type":"added","timestamp":"2020-06-12T03:43:58.396422300Z","datas":{"node":"node2","resource":"r0"}}
{"type":"role","timestamp":"2020-06-12T03:43:58.396422300Z","datas":{"node":"node2","resource":"r0","role":"secondary"}}
{"type":"file_state","timestamp":"2020-06-12T03:43:58.396422300Z","datas":{"node":"node2","resource":"r0","value":"fileless"}}
{"type":"file_state","timestamp":"2020-06-12T03:43:58.437426600Z","datas":{"node":"node2","resource":"r0","value":"attaching"}}
{"type":"file_state","timestamp":"2020-06-12T03:43:58.452638800Z","datas":{"node":"node2","resource":"r0","value":"outdated"}}
{"type":"state","timestamp":"2020-06-12T03:43:58.479433800Z","datas":{"node":"node2","peer":"node1","resource":"r0","value":"connecting"}} |
For more information on the type of event, refer to the appendix's command.
Online Verify
Data consistency check between source and target can be performed with the following command. The consistency check is performed by requesting the verify check from the target, not the source, as follows.
| Code Block |
|---|
λ fsradm verify r0 |
Onine Verify differs in operation mode depending on whether or not to perform replication. If both source and target are Secondary, it operates in normal verify test mode. However, if there is a replication in which one is the primary, there is a difference between the data between the source and the target, so it operates in the advanced-verify mode, which waits for the data sequence for the replication change to respond. Normal verify mode and advanced-verify mode are automatically determined by the engine, so you don't have to worry about it, but be aware that there are differences between the two methods.
Basically, Online Verify is based on the premise of checking between data that is UpToDate, so if both sides are not up to date, or if there is a state change such as synchronization progresses or replication state changes during the consistency check, the consistency check is canceled.
The target to be checked is the file with differences through hash comparison, and after the consistency check is finished, the result of the test can be checked through the result command.
| Code Block |
|---|
λ fsradm result r0
{
"id": "r0",
"result": {
"summary": {
"start_time": "2019-09-09T06:22:26.6958913Z",
"end_time": "2019-09-09T06:22:27.4653424Z",
"peer_node": "node2"
},
"totals": {
"diff_dir": "3",
"diff_file": "1",
"diff_bytes": "14",
"orphaned_dir": "0",
"orphaned_file": "0",
"orphaned_bytes": "0",
"missing_dir": "0",
"missing_file": "0",
"missing_bytes": "0",
"synced_bytes": "0"
},
"files": [
{
"type": "different",
"name": "G:\\Temp\\test1\\conf\\drbd.d",
"is_dir": true,
"out_of_sync": "0",
"synced": "0",
"flags": 4,
"properties": {
"mod_time": {
"local": "2019-09-06T13:26:59.1427926+09:00",
"remote": "2019-09-02T07:24:39.161996Z"
}
}
},
{
"type": "different",
"name": "G:\\Temp\\test1\\conf\\drbd.d\\1",
"is_dir": true,
"out_of_sync": "0",
"synced": "0",
"flags": 4,
"properties": {
"mod_time": {
"local": "2019-09-06T13:26:54.0042751+09:00",
"remote": "2019-09-02T07:24:39.3341577Z"
}
}
},
{
"type": "different",
"name": "G:\\Temp\\test1\\conf",
"is_dir": true,
"out_of_sync": "0",
"synced": "0",
"flags": 4,
"properties": {
"mod_time": {
"local": "2019-09-06T13:26:59.0677748+09:00",
"remote": "2019-08-07T02:15:58.4057437Z"
}
}
},
{
"type": "different",
"name": "G:\\Temp\\test1\\contributors.txt",
"out_of_sync": "14",
"synced": "0",
"flags": 5,
"properties": {
"mod_time": {
"local": "2019-09-09T14:00:05.6379239+09:00",
"remote": "2018-12-12T04:42:50.6605579Z"
},
"size": {
"local": 9,
"remote": 15
}
}
}
],
"file_count": 4
}
} |
Reconfigurations
Describes the procedure for changing settings during operation and for recovering from an unexpected failure.
Change Settings
The FSR's cross-node configuration settings must all be the same at the configuration file level. If there is a difference in the settings, the operation method may be ambiguous, so the difference between these configurations is limited. So, to change the FSR settings, first disconnect the connection between the source and target nodes, change the settings of each node identically, adjust, and then re-establish the connection. If a difference is found in the configuration of both nodes at the time of connection, the connection is disconnected with an error and the status is StandAlone. Also, if a connection is already established and an attempt is made to adjust the settings on both sides to be different, an error is returned.
| Info |
|---|
The process of adjusting the settings entails state changes between nodes through protocols implemented internally by the FSR. The node that performed the adjust is in the adjusting state, and at this point the other node is in the need_to_adjust state, which induces the adjust to be performed. |
Reconfigurations after failure
In the event of an unexpected environmental problem such as physical disk damage during replication operation, a procedure is required to respond to it and recover the replication. Basically, if this happens you will have to replace the disk and reconfigure the replication.
You should perform the procedure to reconfigure and resynchronize replication using the following procedure.
- Stop running resources.
| Code Block |
|---|
c:\>fsradm down r0
done |
- Performs recovery operations such as disk replacement.
- Regenerate the meta. If there is any change in the configuration, you must create a new configuration file and regenerate the meta.
| Code Block |
|---|
c:\>fsradm meta create r0
done |
- Start the resource.
| Code Block |
|---|
c:\>fsradm up r0
done |
- Synchronization starts when a connection is established with the source node.
File deletion backup
FSR provides backup for file deletion. Backup for file deletion is a function that temporarily stores files that are accidentally deleted in a specific path of the target, and can be specified by the archive attribute. The archive attribute is disabled by default, and you can specify the path to be backed up and how long to keep.