Overview

bsrmon is a performance monitoring tool that tracks the entire replication process of the bsr engine that performs a BSR engine performing replication and records the time required for taken by each logical section of the engine's logic in real time to identify bottlenecks in the engine operation's behavior. The bsr BSR kernel engine maintains performance-related statistical information such as time records for each section and accumulated calculation values at the time the engine was operated in the form of a file log through the internal jiffies record code. Administrators can monitor the statistics information of these kernel engines by querying them statistics in the form of file logs, including a section-by-section time history of when the engine was running through internal time (jiffies) records and cumulative calculations. Administrators can monitor these statistical information of the kernel engine by querying with the bsrmon utility. bsrmon is supported since from version 1.6.1.

Functions

Features provided by bsrmon include:

/start

...

...

Activation of the performance monitor is performed through the following process.

• windows

  • set HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\bsr\bsrmon_run registry value to 1

  • Activating the performance data aggregation logic of the bsr engine

  • Enable bsrservice's bsrmon /file execution logic

• linux

  • Set the /etc/bsr.d/.bsrmon_run value to 1.

  • Activating the performance data aggregation logic of the bsr engine

  • /lib/bsr/bsrmon-run Run the script. Run the bsrmon /file command periodically

/stop

Disable Performance Monitor. When disabled, the engine stops aggregating performance data and logging files.

Deactivation is performed as follows.

• windows

  • Set HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\bsr\bsrmon_run registry value to 0

  • Disable performance data aggregation logic of bsr engine

  • Disable bsrservice's bsrmon /file execution logic

• linux

  • Set /etc/bsr.d/.bsrmon_run value to 0

  • Disable performance data aggregation logic of bsr engine

  • Stop the /lib/bsr/bsrmon-run script

/status

Query the operating status of the performance monitor.

/file

Saves performance data and memory information to a file. This is a hidden command and is used by the bsrservice and bsrmon-run scripts.

The file storage location is as follows.

• windows : C:\Program Files\bsr\log\perfmon

• linux : /var/log/bsr/perfmon/

Data is saved only for the inquiry time and the numerical value for each item, and when the /watch or /report command is executed, the data is parsed and output according to the format.

2021-07-20_05:55:21.443 955 955 244480 244480 1042 1042 266752 266752

/watch {types} [/scroll]

Data that is being aggregated by type is output in real time. When the /scroll option is used, the output is displayed in newline format.

/report {types} [/f {filename}] [/d {YYYY-MM-DD}] [/s {hh:mm[:ss]}] [/e {hh:mm[:ss]}]

Outputs the statistics of the data recorded by the file for each type.

When generating statistics, meaningless data such as numerical values are excluded from the target sample.

• Data written to zero with no I/O occurring

• I/O data generated temporarily (when the I/O number is not consecutively more than 2 times)

• ex) vnr0_IO_STAT

  Code Block

  2021-07-07_21:27:52.304 0 0 0 0 0 0 0 0 // Excluded from samples as recorded performance data is zero 2021-07-07_21:27:53.325 0 0 0 0 2 2 1 1 // Excluded from samples as transient I/O performance data 2021-07-07_21:27:54.346 0 0 0 0 0 0 0 0 ... 2021-07-07_21:30:28.549 0 0 0 0 408 408 104448 104448 2021-07-07_21:30:29.580 0 0 0 0 409 409 104704 104704 // Performance data is not zero and is being recorded continuously. selected as samples 2021-07-07_21:30:30.605 0 0 0 0 404 404 103424 103424

[/f {filename}]

the file name for the report. Used for statistical calculation of backed up files.

[/d {YYYY-MM-DD}] [/s {hh:mm[:ss]}] [/e {hh:mm[:ss]}]

This option allows you to inquire the figures for a specific section.

• /d : Outputs statistics of data recorded on that date. If not entered, statistics for the entire period are output.

• /s : Outputs statistics of data recorded from that time. If not input, statistics of recorded data are output from 00:00:00.

• /e : Outputs statistics of data recorded up to that time. If not input, statistics of data collected until 23:59:59 are output.

/set {period, file_size, file_cnt} {value}

Command to adjust the numerical values related to monitoring.

period

Set file saving and monitoring cycle. It is set in seconds, and the default value is 1 second.

file_size

Sets the file rolling size. Set in MB unit, the default value is 50 MB.

file_cnt

Sets the number of file rolling. The default value is three.

/io_delay_test {flag} {delay point} {delay time}

Verifies the functionality of the bsr performance monitor by intentionally incurring I/O performance degradation. This feature is for developers.

/debug cmds options

Command for windows to inquire debugfs information.

types

Performance monitoring targets.

iostat, ioclat, reqstat, peer_reqstat, alstat, network, sendbuf, memstat

iostat {resource} {vnr}

Aggregate data for read and write I/O of replication volumes (iops, ios, kbs, kb)

is only available in kernels with the debugfs feature enabled, and debugfs must be mounted in the /sys/kernel/debug path.

You can check if the debugfs feature is enabled using the following command.

# grep CONFIG_DEBUG_FS /boot/config-`uname -r`
CONFIG_DEBUG_FS=y

Since debugfs is automatically mounted starting with CentOS7, and not automatically mounted in CentOS6, you can use the performance monitor feature after mounting it in the /sys/kernel/debug path. If you want to use the performance monitor feature even after a system reboot, you need to register that mount path in fstab.

• Code Block
  # mkdir /sys/kernel/debug # mount -t debugfs debugfs /sys/kernel/debug

Types

The target types of the performance measurement. Corresponds to the types argument of bsrmon.

iostat, ioclat, reqstat, peer_reqstat, alstat, network, sendbuf, memstat

iostat {resource} {vnr}

Aggregate data for read and write I/O on the replica volume (io count, kbs, kb)

> bsrmon /watch iostat r0 0
2020-12-06_08:22:36.223
  read : IO count=0, BW=0kb/s (0KB)
  write: IO count=216, BW=110208kb/s (220416KB)
2020-12-06_08:22:37.640
  read : IO count=0, BW=0kb/s (0KB)
  write: IO count=34, BW=16896kb/s (16896KB)
2020-12-06_08:22:39.32
  read : IO count=0, BW=0kb/s (0KB)
  write: IO count=326, BW=155176kb/s (155176KB)
2020-12-06_08:22:40.106
  read : IO count=0, BW=0kb/s (0KB)
  write: IO count=330, BW=164624kb/s (164624KB)
  ...

> bsrmon /report iostat r0 0
Report r0 [IO STAT - vnr0]
 Run: 2021-07-20_08:14:27.405 - 2021-08-04_18:59:53.573
  read : io count=24855, bw=6325194kbyte
    BW (kbyte/s): min=40, max=284928, avg=248578, samples=24
  write: io count=8360345, bw=2139182512kbyte
    BW (kbyte/s): min=12, max=401408, avg=320256, samples=6671

> bsrmon /report iostat r0 0 /f vnr0_IO_STAT_2021-07-20_081427.005 
Report r0 [vnr0_IO_STAT_2021-07-20_081427.005]
 Run: 2021-07-07_21:24:24.383 - 2021-07-20_08:14:26.371
  read : ios=6332219, bw=1595360917kbyte
    IOPS        : min=1, max=1233, avg=980, samples=6452
    BW (kbyte/s): min=4, max=428032, avg=247033, samples=6452
  write: ios=22550577, bw=3961426883kbyte
    IOPS        : min=1, max=6478, avg=786, samples=28468
    BW (kbyte/s): min=4, max=1085440, avg=138461, samples=28484

ioclat {resource} {vnr}

local/master I/O 의 complete latency (usec)

> bsrmon /watch ioclat r0 0
2020-12-

08_

09:

33:

19.

171
  local clat 

 (usec): 

min=

520, 

max=

112000, 

avg=10347
  master clat (usec): min=610, max=256852, avg=28202
2020-12-

08_

09:

33:

20.

351
  local 

clat  (usec): 

min=

780, 

max=

478387, 

avg=44316
  master clat (usec): min=1499, max=492829, avg=114106
2020-12-

08_

09:

33:

21.

509
  local 

clat  (usec): 

min=

478, 

max=

19805, 

avg=

4523
  master 

clat (usec): 

min=577, 

max=24335, avg=6303
  ...