Commands

Owned by SungEun Kim
Last updated: 2024/01/15
25 min read

bsradm

bsradm [options...] [-- [ backend-options...]] {command} {context...}

DESCRIPTION

The bsradm utility is used to manage bsr based on configuration files. This command overrides the "bsrsetup" and "bsrmeta" utilities by converting high-level commands that control kernel modules and manipulate metadata on disk into one or more low-level commands. Works on one or more resources, devices, connections, or peer devices, depending on the command. Specify resources by name, and the all keyword refers to all resources.

The following command context is defined.

device

Device. Specify as a minor number (minor-minornumber, e.g. minor-0) or as a resource volume number (resource/volume). If you specify only resources, the command repeats for all devices in that context.

connection

Connection. Specify as resource: connection-name. If you specify only a resource, the command repeats for all connections in that context.

peer_device

Peer device. Specify Resource:connection name/volume number (resource: connection-name/volume). If you specify only resources, devices, and connections, the command repeats for all peer devices in that context.

 

bsradm knows most of the options in bsrsetup and passes all double dash options or no double dash options to the specified low-level utility.

 

OPTIONS

-V, --version

Print the version information of the BSR.

-d, --dry-run

Displays commands to execute without actually executing, such as bsradm -d up <resource>. This can be a useful way to learn how to use bsrsetup and bsrmeta.

-c, --config-file file

Alternative configuration files can be used. By default, bsradm uses the first one of the following: /etc/bsr-90.conf, ./etc/bsr-84.conf, /etc/bsr-83.conf, /etc/bsr-82.conf, /etc/bsr-08.conf, /etc/bsr.conf.

-t, --config-to-test file

Additional configuration files are available. This option can only be used with the dump and sh-nop commands.

-s, --bsrsetup file

Specifies the full path to the bsrsetup. If this option is omitted, bsradm looks first in the same path and then in the PATH path.

-m, --bsrmeta file

Specifies the full path to the bsrmeta. If this option is omitted, bsradm looks first in the same path and then in the PATH path.

-S, --stacked

Deprecated.

 

COMMANDS

adjust { resource}

Adjust the kernel module's configuration to match the configuration file. The result should be the same when stopping and restarting all resources ('bsradm up all' followed by 'bsradm up all'). The Adjust command may interpret configuration changes incorrectly in some cases. For safety, see what the command will do (using the --dry-run option) before running the actual command.

adjust-with-progress { resource}

Same as adjust, but provides additional information on the progress of the command.

apply-al { device}

Apply the activity log of the specified device.

attach { device}

Attach the lower-level devices as replicated devices.

connect {connection}

Activate an existing connection to the peer. Connections must be made first with the new-peer command and one or more paths with the new-path command.

create-md {device}

Initialize the device's metadata. This is required before connecting the clone device for the first time.

cstate {connection}

Displays the current connection status.

detach {device}

Detach the lower device of the replicated device.

disconnect {connection}

Remove the connection to the peer host.

disk-options {device}

Change the disk options of the attached device.

down {resource}

This command stops all volumes, connections, and the resource itself.

dstate {device}

Prints the current disk status of the lower device.

dump {resource}

Parse the configuration file and print it to stdout.

dump-md {device}

Dumps the device's metadata in text format. Bitmaps and activity logs are also included.

get-gi {peer_device}

Outputs the generatation identifiers(GI) for a device on a specific connection. Use bsrsetup for attached devices and bsrmeta for detached devices.

hidden-commands

Displays all commands that are not explicitly documented.

invalidate {peer_device}

Synchronize the device's local data with the peer's data.

invalidate-remote {peer_device}

Synchronize data from peer devices to devices on the local node.

net-options {connection}

Change the net options of an existing connection.

new-current-uuid {device}

Create a new current UUID.

new-current-uuid --no-rotate-bitmap {device}

Creates a new current UUID but does not rotate the bitmap UUID. Gives the effect that I/O occurred locally.

outdate {device}

Set the data of the child device to outdated.

pause-sync {peer_device}

Set the local pause flag to stop resynchronization between local and peer devices.

primary {resource}

Change the role of the node in the resource to primary.

resize {device}

Resize the lower device of the replica on all nodes. This is done by combining check-resize and resize low-level commands.

resource-options {resource}

Change resource options for an existing resource.

resume-sync {peer_device}

Clear the local sync pause flag so that resynchronization resumes.

role {resource}

Print the current role of the resource.

secondary {resource}

Change the role of the resource to Secondary. This command will fail on Linux if the resource's replicated device is in use. On Windows, it is demoted whether or not the device is used.

show-gi {peer_device}

Displays the data generation identifier of the device on a specific connection. Also, the output is explained in detail.

up {resource}

Start the resource in the following procedure.

  • Apply activity log of all volumes: bsrmeta apply-al

  • Create resource: bsrsetup new-resource

  • Create a replicated device: bsrsetup new-device, bsrsetup new-minor

  • Attach the lower device: bsrsetup attach

  • Connect to all peers: bsrsetup connect

verify {peer_device}

Start, stop, or specify verification for specific parts of online verification.

wait-connect {[device] | [connection] | [resource]}

Wait for devices on the peer, all devices on the connection, all devices on all peers to be verified.

wait-sync {[device] | [connection] | [resource]}

Wait for the device to connect and the final resync operation to complete. Also available at the connection and resource level

wipe-md {device}

Clears the device's bsr metadata.

forget-peer {connection}

Removes references to unconnected peers completely from metadata.

 

bsrsetup

bsrsetup command {argument...} [option...]

DESCRIPTION

The bsrsetup utility configures the bsr kernel module and displays the current configuration. Users typically interact with the bsradm utility, which provides a higher level interface to bsr than bsrsetup. (See bsradm's --dry-run option to see how to use bsrsetup in bsradm.) Some option arguments have a default scale applied when specifying a regular number (eg, Kilo). These default scales can be overridden using a suffix (eg Mega for M). Common suffixes K = 2^10 = 1024, M = 1024K and G = 1024M are supported.

COMMANDS

bsrsetup attach minor lower_dev meta_data_dev meta_data_index,

bsrsetup disk-options minor

The attach command attaches a lower-level device to an existing replicated device. The disk-options command changes the disk options of the attached child device. All commands All Targets the clone device created by bsrsetup new-minor and specifies it as the minor number of the replicated device. lower_dev is the name of the child device. meta_data_dev is the name of the device that contains the metadata, it can be the same as lower_dev. meta_data_index is the number of index metadata or the internal keyword for internal metadata or the flexible keyword for variable-size external metadata. The available options are:

  • --al-extents extents

    • bsr manages active and recently rewritten areas based on recent disk write operations. When write I/O occurs, the active area can be written to disk immediately, but the inactive disk area must be activated first, so metadata write is required here. This active disk area is called activity log. If you save the metadata write to the activity log, but recover the failed node, you will need to resynchronize over the entire activity log. Therefore, the size of the activity log is a major factor in how long it will take to resynchronize after the primary crash and how quickly the consistency of the clone disk is achieved. Activity log consists of several 4 MiB unit segments. The al-extents parameter determines the number of segments that can be active simultaneously. The default for al-extents is 6001, with a minimum of 7 and a maximum of 65536. Depending on how you generated the device metadata, the maximum valid value may be smaller.

    • The maximum effective value is 919 * (available on-disk activity log ring buffer area / 4kB -1), up to 6433 (including 25 GiB or more data) in the default 32KB ring buffer. It's recommended to keep the size of the activity log within an amount where the backend storage and replication links can be resynchronized in about 5 minutes. Changing the size of al-extents requires a resource down.

  • --al-updates {yes | no}

    • Setting this parameter to no enables the activity log to be turned off completely. Writing speeds up because less metadata writes are required, but the entire device must be resynchronized upon recovery of the failed primary node. The default value of al-updates is yes.

--disk-barrier, --disk-flushes, --disk-drain bsr has three ways to handle the order of write requests.

  • disk-flushes

    • After writing I / O to disk, force flush to write all data to disk. Depending on the platform or drive vendor, the implementation of flush may be different. In the old method, it was used as a technique to bypass the disk cache called 'force unit access', but recently, it is basically implemented in a way that writes the contents of the disk cache to disk to ensure disk write. This option is enabled by default.

  • disk-barrier

    • Use this option to ensure that requests are written to disk in the correct order. The barrier ensures that all requests submitted before the barrier are all requested to disk prior to requests subsequently submitted. This is implemented using 'tagged command queuing' of SCSI devices and 'native command queuing' of SATA devices. Only some devices and device stacks support this method. The device mapper (LVM) only supports barriers in some configurations. Using this option on systems that do not support disk-barrier can result in data loss or corruption. This option was supported by older Linux kernels, but kernels after linux-2.6.36 (or 2.6.32 RHEL6) can no longer detect if disk-barrier is supported. This option is off by default and must be explicitly enabled.

  • disk-drain

    • Wait for the request queue to "drain" (that is, until the request is complete) before submitting a write request. To use this method, requests must be stable on disk until the request is completed. Previously, this option was enabled by default, but it is not the default option at this time.

Of these three methods, bsr uses the first method enabled and supported by the backup storage device. If you turn off all three options, bsr submits the request without worrying about write dependencies. On a cluster node in another environment, write requests can be reordered according to the I / O stack and submitted in a different order. This can result in data loss or corruption. Therefore, we recommend that you do not turn off all three methods of controlling the write order.

A general guideline for configuring write ordering is to use disk-barrier or disk-flush when using a regular disk (or a regular disk array) with a volatile write cache. Disk-drain is suitable for storage with no cache or with a battery-backed write cache.

  • --disk-timeout

    • If the I/O request fails to complete within the disk time defined for the lower device that stores the data, bsr treats it as a failure. In this case, the lower device is detached, and the disk status of the device will be diskless. If bsr is connected to one or more peers, the failed request is forwarded to one of them. This option is critical and can lead to a kernel panic. Aborting the request and forcing the disk to be removed is an action for a completely blocked and stopped local backup device that no longer completes the request and returns no errors. In this situation, usually a hard reset and failover is the only way. The default value of disk-timeout is 0, which indicates an infinite timeout. Timeouts are specified in 0.1 second increments.

  • --md-flushes

    • Enables disk flush and disk barrier on metadata devices. This option is enabled by default.

  • on-io-error handler

    • Configure how bsr responds to I/O errors on low-level devices. The following policies are defined.

      • passthrough If an error is returned from the lower device, the block layer is written as OOS and the error is passed to the upper layer. The error block is usually retried I / O by the upper layer, and if it succeeds at the time of retry, the OOS will be resolved naturally, otherwise the OOS will be recorded and left. This is the default for bsr.

      • call-local-io-error local-io-error Call the handler.

      • detach Detach the low-level device and switch to diskless state. In diskless state, I/O cannot be performed and failover is required immediately.

  • resync-after minor

    • Defines a device to resynchronize only after another specified device has been synchronized. By default, no synchronization order is defined between devices, and all devices are resynchronized in parallel. Depending on the sub-device configuration, available network and disk bandwidth, the entire re-synchronization process can be slow, so you can use this option to form a chain or tree of dependencies between devices.

bsrsetup peer-device-options resource peer_node_id volume

The following options affect the peer device.

  • --c-delay-target delay_target, 

  • --c-fill-target fill_target, 

  • --c-max-rate max_rate, 

  • --c-plan-ahead plan_time

    • Dynamically control the speed of resynchronization. This mechanism can be used by setting the c-plan-ahead parameter to a positive value. The maximum bandwidth is limited by the c-max-rate parameter. The c-plan-ahead parameter defines how quickly bsr adapts to changes in the resynchronization rate. It should be set to at least 5 times the network round-trip time (RTT). When c-fill-target is defined, it tries to fill the buffer with a defined amount of data along the data path, and has a defined delay if c-delay-target is defined. The common value range for c-fill-target for "normal" data paths is 4K to 100K. If you use drx, we recommend using c-delay-target instead of c-fill-target. The c-delay-target parameter is used when the c-fill-target parameter is undefined or set to 0. The c-delay-target parameter should be set to at least 5 times the network round trip time. The c-max-rate option should be set to either the available bandwidth or the available disk bandwidth between the bsr host and the system hosting drx. The default values ​​for these parameters are: c-plan-ahead = 20 (in 0.1 second increments), c-fill-target = 0 (in sector increments), c-delay-target = 1 (in 0.1 second increments) and c-max-rate = 102400 (KiB/s unit).

  • --c-min-rate min_rate

    • The primary and synchronization source nodes must schedule application I/O requests and synchronization requests. The c-min-rate parameter limits the amount of bandwidth available for resynchronization I/O. The rest of the bandwidth is used for replication of application I/O. If the c-min-rate value is 0, it means there is no limit to the resynchronization I/O bandwidth. This can significantly slow down application I/O. Use the value of 1 (1 KiB / s) for the lowest resynchronization rate. The default value of c-min-rate is 250 in KiB/s.

  • --resync-rate rate

    • Defines the bandwidth available for resynchronization. bsr allows general application I/O even during resynchronization. If resynchronization takes up too much bandwidth, application I/O can be very slow and this parameter can be avoided. This option only works if the dynamic resync controller is disabled.

bsrsetup check-resize minor

Remembers the current size of the specified replicated lower devices. Used by bsradm. Size information is stored in the /var/lib/bsr/bsr-minor-minor.lkbd file.

bsrsetup new-peer resource peer_node_id,

bsrsetup net-options resource peer_node_id

The new-peer command creates a connection within the resource. Resources must be created with bsrsetup new-resource. The net-options command changes the network options of an existing connection. Before activating a connection with the connect command, you must add at least one path with the new-path command. The available options are:

  • --after-sb-0pri policy

    • Defines how to respond when a split brain scenario is detected and neither of the two nodes plays the Primary role. The split brain is always determined between two nodes and detects when two nodes are connected. The policies defined are:

      • disconnect Simply disconnect.

      • discard-younger-primary,

      • discard-older-primary Discard the node that became primary lately(discard-younger-primary) or the node that became primary first(discard-older-primary). If both nodes have become Primary independently, the discard-least-changes policy is used.

      • discard-zero-changes If data is written on only one node, resynchronization is performed based on this node. If both nodes have written data, they disconnect.

      • discard-least-changes Synchronize based on the node that wrote more data.

      • discard-node-nodename Always discard named nodes.

  • --after-sb-1pri policy

    • Define what to do if a split brain is detected with one primary node and one secondary node. (The split brain decision is always one of the two nodes because it detects a split brain scenario when two nodes are connected.) The policies defined are:

      • disconnect Simply disconnect.

      • consensus If a victim node can be selected, it is automatically resolved. Otherwise, it works like disconnect.

      • discard-secondary Discard the secondary node.

  • --after-sb-2pri policy

    • Define how to react when a split brain scenario is detected and both nodes act as primary. (The split brain decision is always one of the two nodes because it detects a split brain scenario when two nodes are connected.) The policy defined is: 2 For primary split brain, only manual recovery via disconnect is available.

      • disconnect Simply disconnect.

  • --connect-int time

    • As soon as a connection between two nodes is configured with bsrsetup connect, the connection is attempted to be established. If this fails, bsr waits for connect-int seconds and repeats. The default value of connect-int is 3 seconds.

  • --csums-alg hash-algorithm

    • Typically, when two nodes are resynchronized, the synchronization target requests out-of-sync data from the synchronization source, and the synchronization source sends the data.

      In many usage patterns, many of the blocks are actually the same. When the --csums-alg algorithm is specified, when requesting unsynchronized data, the synchronization target also sends a hash of the data currently held. The synchronization source compares this hash to its own data. If the hashes are different, new data is sent to the synchronization target and if the hashes are the same, the data is the same. This reduces the required network bandwidth, but increases CPU utilization and increases SyncTarget's read I/O. csums-alg can be set to one of the secure hash algorithms supported by the kernel. See the shash algorithm listed in /proc/crypto. By default, csums-alg is not set.

  • --data-integrity-alg alg

    • bsr typically relies on data integrity checking built into the TCP/IP protocol, but if a data integrity algorithm is configured, it is used to verify that data received over the network matches what the sender sent. When a data integrity error is detected, bsr closes the network connection and reconnects to trigger resynchronization. data-integrity-alg can be set to one of the secure hash algorithms supported by the kernel. See the shash algorithm listed in /proc/crypto. By default, this mechanism is off. We do not recommend this option in production environments due to the associated CPU overhead.

  • --fencing fencing_policy

    • Fencing is a preventive measure to prevent situations where both nodes become disconnected and become both primary. This is also called a split brain situation. bsr supports the following fencing policies:

      • dont-care No fencing action is taken. This is the default policy.

      • resource-only When a node becomes the disconnected Primary node, it tries to block the peer. This is done by calling the "fence-peer" handler. The handler must reach the peer through an alternate communication path and call 'bsradm outdate minor'.

      • resource-and-stonith When the node becomes the disconnected Primary node, it stops all IO operations and calls the fence-peer handler. The fence-peer handler must reach the peer through an alternate communication path and call 'bsradm outdate minor'. If you are unable to do so, you must block (power control) the other party. IO resumes as soon as the situation is resolved. If the fence peer handler fails, you must determine that a potential split brain has occurred and manually repair it.

  • --ko-count number

    • Defines the number of transmission retries on the TX node side when buffering transmissions.

  • --max-buffers number

    • Defines the maximum buffer size of the receiving peer-request. The unit is PAGE_SIZE (4 KiB on most systems). The minimum possible setting is hard coded to 32 (= 128 KiB). This buffer is used to hold blocks of data while writing to or reading from disk. If more than max-buffers pages are in use, further allocation of this pool is limited. If the receiving side cannot handle the I / O load, the max-buffers should be increased.

  • --max-epoch-size number

    • Defines the maximum number of write requests that bsr can issue before issuing a write barrier. The default is 2048, with a minimum of 1 and a maximum of 20000. Setting this parameter to a value less than 10 can degrade performance.

  • --on-congestion policy, 

  • --congestion-fill threshold, 

  • --congestion-extents threshold

    • By default, bsr waits when the TCP send queue is full. In this case, the application cannot generate additional write requests until the send queue is available again. If you are using bsr with a proxy, we recommend using a pull-ahead congestion policy that allows you to put bsr into Ahead/Behind mode before the transmission queue is full. Then bsr records the difference between itself and the peer in the bitmap, but no longer replicates it to the peer. When enough buffer space becomes available again, the node resynchronizes with the peer and switches back to normal replication. This has the advantage of not blocking application I/O even when the queue is full, but it has the disadvantage that the peer node may lag far behind the original. And during resynchronization, the peer node is in an Inconsistent state. The available congestion policies are blocking (default), disconnect, and pull-ahead. The congestion-fill parameter defines the amount of data that is being replicated on this connection. The default is 0 (disable congestion control mechanism) and is up to 1 TB. The congestion-extents parameter defines the number of bitmap ranges that can be active before switching to Ahead/Behind mode. The congestion-extents parameter is only valid when set to a value less than al-extents.

  • --ping-int interval

    • If the TCP/IP connection to the peer has been idle for more than 1 second, bsr sends a ping packet to quickly detect a failed peer or network connection. The default is 3 seconds, with a minimum of 1 and a maximum of 120 seconds. The unit is seconds.

  • --ping-timeout timeout

    • Define the reply timeout for ping packets. If the peer does not respond within the ping timeout, bsr closes the connection and tries to connect again. The default is 3 seconds, with a minimum of 0.1 seconds and a maximum of 3 seconds. The unit is one tenth of a second.

  • --protocol name

    • Defines the protocol specified for the replication connection. The supported protocols are:

      • A Complete local I / O as soon as it is copied to the local disk and TCP/IP transport buffer.

      • B Writes to the local disk and returns an (Receive)ACK as soon as the peer receives replication data. When ACK is received locally, I/O is completed.

      • C Writes to the local disk and the peer writes the replicated data to disk and returns (Write)ACK. When a write ACK is received locally, I/O is completed.

  • --rcvbuf-size size

    • Configures the size of the TCP/IP receive buffer. If the value is 0 (default), the buffer size is dynamically adjusted. This parameter usually does not need to be set, but can be set to a value of up to 10 MiB. The default unit is bytes, which is not supported on Windows.

  • --sndbuf-size size

    • Set the size of TX buffer allocated by the sending worker thread. You can set up to 1TB.

  • --tcp-cork

    • By default, bsr uses the tcp-cork option to suppress the kernel from sending small messages. This increases the packet size on the network. This optimization can degrade the performance of some network stacks, and there is a delay during the time to collect packets. You can turn this optimization off using the tcp-cork parameter.

  • --timeout time

    • Define the response timeout over the network. If the peer node does not send an expected response within the specified timeout, it is considered unresponsive and closes the TCP/IP connection. The timeout value must be lower than connect-int and smaller than ping-int. The default is 5 seconds, specified in units of tenths of a second.

  • --use-rle

    • use-rle determines if run length encoding should be used. Each cloned device in a cluster node has a separate bitmap for each peer device. Bitmaps are used to track the differences between local and peer devices. Depending on the cluster state, the disk range may be marked as different from the peer in the device's bitmap, peer device's bitmap, or both bitmaps. When two cluster nodes are connected, they exchange their bitmaps with each other and compute the union of the local and peer bitmaps, respectively, to determine the overall difference. For very large devices, bitmaps are relatively large, so run length encoding is usually used to increase compression, which saves time and bandwidth required for bitmap transmission. It is enabled by default.

  • --verify-alg hash-algorithm

    • Online verify (bsradm verify) calculates and compares checksums of disk blocks (i.e. hash values) to detect differences. The verify-alg parameter determines the algorithm to use for these checksums. Before using online validation, you must set it up as one of the secure hash algorithms supported by the kernel. See the shash algorithm listed in /proc/crypto. We recommend that you schedule online confirmations regularly, for example, once a month, when the operating load is low.

bsrsetup new-path resource peer_node_id local-addr remote-addr

The new-path command creates a path within the connection. The connection must be made with bsrsetup new-peer. local-addr and remote-addr must describe local and remote protocols, network addresses, and ports in the form [address-family:] address [: port]. ipv4, ipv6 are supported. If no address family is specified, "ipv4" is assumed. For all address families except ipv6, the address uses IPv4 address notation (e.g. 1.2.3.4). For ipv6, addresses are enclosed in parentheses and use IPv6 address notation (eg [fd01: 2345: 6789: abcd :: 1]). The default port is 7788.

bsrsetup connect resource peer_node_id

The connect command activates the connection. That is, the bsr driver binds and listens to all local addresses in the connection path. Tries to establish one or more connection paths. The available options are:

  • --tentative

    • Decide if you can establish a connection to a peer without actually making a connection or resynchronizing, and if you need resynchronization (and in which direction). You can check the system log to see in advance what bsr will do without the --tentative option.

  • --discard-my-data

    • Discard the local data and resynchronize with the peer with the latest data. Use this option to manually recover from split brain situations.

bsrsetup del-peer resource peer_node_id

The del-peer command removes a connection from a resource.

bsrsetup del-path resource peer_node_id local-addr remote-addr

The del-path command removes the path from the connection. However, if you need to keep the connection already connected, this command will fail. To remove all paths, you must first disconnect.

bsrsetup cstate resource peer_node_id

Displays the current status of the connection. Connections are identified by the peer's node-id.

bsrsetup del-minor minor

Remove the replicated device. Subsequent devices will no longer be attached after this.

bsrsetup del-resource resource

Remove resources. To do this, you must first remove all volumes and connections (bsrsetup del-minor, bsrsetup disconnect). Or you can use bsrsetup down to remove resources with all volumes and connections.

bsrsetup detach minor

Detach the lower device from the replicated device. The available options are:

  • --force Force detach and return immediately. This will put the lower device into a failed state until all pending I/O is complete, then detach the device. I/Os that have not yet been submitted to a lower device (for example, because the device's I/O has been suspended) are considered failed.

bsrsetup disconnect resource peer_node_id

Remove the connection to the peer host. Connections are identified by the peer's node-id.

bsrsetup down {resource | all}

Stop resources by removing all volumes, connections, and the resource itself.

bsrsetup dstate minor

Displays the current disk status of the child device.

bsrsetup events2 {resource | all}

Displays the current state of all configured bsr objects and any changes in state. Output starts with the type of event. There are types of events, such as when an existing object is created, destroyed, or changed, an event handler is called or returned. The following describes the resource, device, connection, peer device, and helper object to which the event applies. The rest identifies the object and displays the object's status. The following options are available.

  • --now Report the current status and exit. The default is to constantly see the status change.

  • --statistics Include statistics in the output.

bsrsetup get-gi resource peer_node_id volume

Displays the data generation identifier of the device on a specific connection. Devices are identified by volume number, and connections by endpoints. See the "bsrsetup connect" command. The output now consists of the first two historical UUIDs, which consist of a UUID, bitmap UUID, and a set of flags. The current UUID and historical UUID are device-specific. Bitmap UUIDs and flags vary by peer device. This command only displays the first two historical UUIDs. bsr maintains one historical UUID for each peer device that is internally possible.

bsrsetup invalidate minor

Synchronize the device's local data with the peer's data. All space used by the peer's filesystem is marked out of sync and resynchronized with the specified local data.

bsrsetup invalidate-remote resource peer_node_id volume

Synchronize peer data with local data. All space used by the local file system is marked out of sync and resynchronized with the specified peer data.

bsrsetup new-current-uuid minor

Create a new current UUID and rotate all other UUID values. The available options are:

  • --clear-bitmap

    • Create a new current UUID and clear the sync bitmap. It can be used to skip the initial sync by removing the bitmap GI. This method only works for metadata that has just been created.

bsrsetup new-minor resource minor volume

Create a new clone device within the resource. This command creates a block device inode for the clone device (/dev/bsr minor by default). The volume number identifies the device within the resource.

bsrsetup new-resource resource node_id,

bsrsetup resource-options resource

The new-resource command creates a new resource. The resource-options command changes the resource options of the created resource. The available options are:

  • --auto-promote bool-value

    • Not supported.

  • --cpu-mask cpu-mask

    • Not supported.

  • --on-no-data-accessible policy

    • Determines how to handle I/O requests when the requested data is not locally accessible (for example, if all disks fail). Not supported by bsr.

  • --peer-ack-window value

    • On each node and each device, bsr maintains a bitmap of the difference between local and remote data for each peer device. For example, in a three-node setup (nodes A, B, C) with a single device, all nodes maintain one bitmap for each peer. When a node receives a write request, I know how to update the bitmap for the write node, but I don't know how to update the bitmap between different nodes. In this example, when a write request is propagated from Node A to B and C, Node B and C have the same data as Node A, but I don't know if both have the same data. As a solution to this, write nodes sometimes send peer-ack packets to peers to tell each other what they are in. The peer-ack-window parameter specifies the amount of data that the primary node can send before sending a peer-ack packet. A low value increases network traffic. A larger value reduces network traffic, but increases memory consumption on the secondary node and increases the resynchronization time between secondary nodes after a primary node failure. (Note: peer-ack packets may be sent for other reasons (eg, membership change or expiration of the “peer-ack-delay timer”). The default value of peer-ack-window is 2 MiB, the default unit is sector) .

  • --peer-ack-delay expiry-time

    • If a new write request is not issued during the expiration time after the last completed write request, a peer-ack packet is sent. If a new write request is issued before the timer expires, the timer resets to the expiration time. (Note: peer-ack packets may be sent for other reasons, such as changing membership or the "peer-ack-window" option) This parameter can affect the resynchronization behavior of the remote node. The peer node must wait until it receives a peer-ack to release the lock on the AL extent. Resynchronization between peers may have to wait for these locks. The default for peer-ack-delay is 100 milliseconds, and the default unit is milliseconds.

bsrsetup outdate minor

Displays data of sub-devices as outdated. Used for fencing and prevents future devices from becoming primary resources.

bsrsetup pause-sync resource peer_node_id volume

Set the local pause flag to stop resynchronization between the local device and peer device. Resynchronization can only be resumed if the pause flags on both sides of the connection are cleared.

bsrsetup primary resource

Change the role of the node in the resource to primary. This allows replicated devices of this resource to be mounted or opened for writing. Available options:

  • --overwrite-data-of-peer

    • This option is an alias of --force option.

  • --force

    • Even if you don't guarantee that your device has up-to-date data, set the resource as the Primary resource. This option is used when replacing one of the nodes in the newly created cluster with the Primary node or manually recovering from a disaster. This can lead to split brain scenarios. In addition, it is recommended to perform an integrity check (eg, file system check) at least when forcing an inconsistent device to the latest one. bsr usually allows only one node in the cluster to play the Primary role. This allows bsr to restrict access to devices on the node's resources.

bsrsetup resize minor

Resize the subdevice of the replicated device on all nodes. This command is called after resizing the replicated device because all nodes' low-level devices have grown. The available options are:

  • --assume-peer-has-space

    • Resize the device even if some of the current peer devices are not connected. bsr tries to resize the peer device the next time it connects. Rejects connections to peer devices that are too small.

  • --size val

    • This option allows you to reduce the available size of bsr devices online. It is the user's responsibility to ensure that the file system of the device is not truncated due to such actions.

  • --al-stripes val 

    • This option can be used to change the online layout of the activity log. For internal metadata, you can simultaneously reduce the size the user sees (not resize it) or increase the available space on the backup device.

bsrsetup resume-io minor

Resume I/O on the replicated device.

bsrsetup resume-sync resource peer_node_id volume

Clear the local sync pause flag so that resynchronization resumes.

bsrsetup role resource

Shows the current role of the resource.

bsrsetup secondary resource

Change the role of the node in the resource to secondary. On Linux, the command will fail if the replicated device is in use. On Windows, demote to Secodary regardless of whether it is used or not.

bsrsetup show {resource | all}

Displays the current configuration settings for a resource or all resources. The available options are:

  • --show-defaults Displays configuration parameters with all configuration parameters and default values. Normally, parameters with default values are not displayed.

bsrsetup show-gi resource peer_node_id volume

Displays the data generation identifier of the device on a particular connection and describes the output. Same as the "bsrsetup get-gi" command except for the description.

bsrsetup state

An alias for the bsrsetup role. Deprecated.

bsrsetup status {resource | all}

Displays the status of a resource or all resources. The output consists of one paragraph for each configured resource. Each paragraph has one line for each resource, one line for each device, and one line for each connection. The device and connection lines are indented. After the connection line, there is one line for each peer device. These lines are indented on the connecting line. Long lines are wrapped with terminal width and indented to show how the lines are joined. The available options are:

  • --verbose Include additional information in the output (even if it is duplicated or irrelevant).

  • --statistics Output including statistical information.

  • --color={always | auto | never} Color the output. If you use --color = auto, bsrsetup will display colors only when standard output is connected to the terminal.

bsrsetup suspend-io minor

Not Supported.

bsrsetup verify resource peer_node_id volume

Start an online verification, change or stop the test if you see a specific part of the device. The command must have a specified peer connected. The online check compares each disk block of local and peer nodes. Other blocks between nodes are written as OOS, but are not automatically resynchronized. To synchronize, you need to disconnect and reconnect the resource. You can monitor the progress in the output of bsrsetup status --statistics. Available options:

  • --start position

    • Define where to start the online verification. This parameter is ignored if an online verification is already in progress. If the start parameter is not specified, the online verification starts after the point where it left off if it was previously stopped, or from the beginning if it has never been started. Disks are located by default in disk sectors (512 bytes).

  • --stop position

    • Define where to stop the online verification. If online verification is already in progress, the stop position of the active online verification process changes. Use this option to stop the online verification. Disks are located by default in disk sectors (512 bytes).

bsrsetup wait-connect-volume resource peer_node_id volume,

bsrsetup wait-connect-connection resource peer_node_id,

bsrsetup wait-connect-resource resource,

bsrsetup wait-sync-volume resource peer_node_id volume,

bsrsetup wait-sync-connection resource peer_node_id,

bsrsetup wait-sync-resource resource

The wait-connect-* command waits for the peer's device to appear. The wait-sync-* command waits until the peer's device is up to date. bsr is not recommended for use with commands that are rarely used.

bsrsetup forget-peer resource peer_node_id

The forget-peer command removes all traces of peer nodes from metadata. You can free up bitmap slots in the metadata and use them to allocate additional bitmap slots if a node you have never seen connects. You must disconnect to use this command. If the peer reconnects later, bitmap based resynchronization is replaced by full synchronization.

bsrmeta

bsrmeta [--force] [--ignore-sanity-checks] { device} {v06 minor | v07  meta_dev index | v08 meta_dev index | v09  meta_dev index} { command} [cmd args...]

DESCRIPTION

The bsrmeta utility is used to create, display and modify bsr's on-disk metadata. Users typically interact with the bsradm utility, which provides a higher level interface to bsr than bsrarm. (See bsramm's dry-run option for how to use bsradm.) This utility is only available for devices not currently used by the kernel. The first argument (device) specifies the bsr device associated with the volume, or "-" if no device is associated with the volume. If a bsr device is specified, the bsr utility checks to see if the bsr device has a volume attached first to prevent metadata corruption on the currently active volume. The second argument is the version of the metadata to use (v06, v07, v08, v09). Designate. In most versions of metadata, the third argument (meta_dev) specifies the device that contains the metadata. This argument can be the same as the device. The fourth argument (index) is the keyword internal (for internal metadata), flex-internal variable (v07 for variable-size metadata, v07 is fixed-size internal metadata by default), flex-external (of variable size) Case). External metadata or numeric matadata index (for fixed-size external metadata) See Meta disk parameters in bsr.conf.

 

OPTIONS

--force

Assuming yes and answering all questions bsrmeta will ask.

--ignore-sanity-checks

In general, bsrmeta performs some defect checking before writing to the metadata device. For example, if the device contains a file system, it refuses to write meta to protect the file system from corruption. Use this option to ignore this defect check.

 

COMMANDS

create-md [--peer-max-bio-size=val] (metadata versions v06, v07, and v08),

create-md {number-of-bitmap-slots} [--peer-max-bio-size=val] [ --al-stripes=val] [--al-stripe-size-kB= val] (metadata version v09)

Initialize metadata. This is required before attaching the bsr resource. When bsrmeta finds the old version of bsr metadata on the device, it asks if it needs to convert the format. bsradm sets the bitmap slot count argument to the number of peers in the resource when invoking bsrmeta's create-md command for the device. To reserve additional bitmap slots (so you can add more peers in the future), call bbsrmeta instead. If you use the device before the first connection to the peer, bsr assumes that the peer can only handle 4KiB requests by default. Use the "peer-max-bio-size" option to set a more optimistic value. Use this if the bsr version to which this device will connect is known. If you want to use more than 6433 activity log ranges or configure over stripe RAID, you need to specify the number of stripes (--al-stripes, default 1) and the size of stripes (--al-stripe- size-kB, default 32). To use a larger linear on disk ring buffer, keep the number of stripes to 1 and increase only the size.

get-gi [--node-id=id]

Displays the data generation identifier of the device on a specific connection. bsr supports multiple peers. Use the node-id option to define the data generation identifier of the peer to be displayed.

show-gi [--node-id=id]

Similar to get-gi, but with descriptive information.

dump-md

Dump the device's metadata, including bitmap and activity log, in text format.

outdate

The data of the lower device is marked as outdated.

dstate

Displays the current disk status of the lower device.

check-resize

Check the device size of the sub-device and the last known device size (stored as /var/lib/bsr/bsr-minor-minor.lkbd by bsrsetup check-resize). For internal metadata, the size of the low-level device If it changes and you can find the metadata in the old location, move the metadata from the end of the block device to the new location.

apply-al

Apply the activity log of the specified device. This is needed before the device is reattached by the kernel.

 

EXPERT COMMANDS

The bsrmeta utility can be used to fine-tune metadata. This can lead to metadata corruption or data corruption automatically. Use with caution.

set-gi gi [--node-id=id] 

Set the generation identifier. bsr supports multiple peers and is set up with the same syntax as in the get-gi output. You can use the --node-id option to define the data generation identifier of the peer to be set.

restore-md dump_file

Replace the device's metadata with the contents of dump_file. The dump file format is defined by the output of the dump-md command.

 

bsrcon

bsrcon command {argument...}

 

DESCRIPTION

Set up the BSR's network, logs, volumes, handlers, etc.

 

COMMANDS

/nodelayedack [ip|guid]

Disables the delayed ACK behavior of TCP/IP. This is required for replication to work and is Windows only.

 

/delayedack_enable [ip|guid]

Enables delayed ACK behavior for TCP/IP. Windows only.

 

/release_vol [letter]

This command completely releases the volume lock that controls the Windows volume. This command must be performed as the final step in deleting a resource and is Windows only.

If you perform this command, you must reorganize the resource and perform a full synchronization again.

 

/bsrlock_use

Commands to temporarily enable or disable locks on a volume. This is for debugging purposes and is Windows only.

 

/bsrlock_status

Commands to view the lock status of a volume. Windows only.

 

/info

Prints the volume's partitions, mount points, and other related information. Windows only.

 

/handler_use [0,1]

Enables or disables the BSR's handler.

 

/maxlogfile_cnt [LogFileMaxCount : 0 ~ 1000]

 

/climaxlogfile_cnt [adm, setup, meta] [LogFileMaxCount : 0 ~ 255]

 

/minlog_lv [sys, dbg] [Level : 0~7] level info, emerg(0) alert(1) criti(2) err(3) warning(4) notice(5) info(6) debug(7)

 

/minlog_lv feature [flag : 0,1,2,4] level info, none(0) oos(1) latency(2) verify(4)