1 Overview
2 Types
- 2.1 iostat {resource} {vnr}
- 2.2 ioclat {resource} {vnr}
- 2.3 reqstat {resource} {vnr}
- 2.4 peer_reqstat {resource} {vnr}
- 2.5 alstat {resource} {vnr}
- 2.6 network {resource}
- 2.7 sendbuf {resource}
- 2.8 memstat
- 2.9 resync_ratio {resource} {vnr}
3 Commands
- 3.1 /start
- 3.2 /stop
- 3.3 /status
- 3.4 /file
- 3.5 /show
- 3.6 /watch
- 3.7 /report
- 3.8 /set
- 3.9 /get
- 3.10 /io_delay_test
- 3.11 /debug cmds options

Overview

bsrmon is a performance monitoring tool that tracks the entire replication process of a BSR engine performing replication and records the time taken by each section of the engine's logic in real time to identify bottlenecks in the engine's behavior. The BSR kernel engine maintains performance-related statistics in the form of file logs, including a section-by-section time history of when the engine was running through internal time (jiffies) records and cumulative calculations. Administrators can monitor these statistical information of the kernel engine by querying with the bsrmon utility. bsrmon is supported from version 1.6.1.

In a linux environment, the Performance Monitor feature is only available in kernels with the debugfs feature enabled, and debugfs must be mounted in the /sys/kernel/debug path.

You can check if the debugfs feature is enabled using the following command.

# grep CONFIG_DEBUG_FS /boot/config-`uname -r` CONFIG_DEBUG_FS=y

Since debugfs is automatically mounted starting with CentOS7, and not automatically mounted in CentOS6, you can use the performance monitor feature after mounting it in the /sys/kernel/debug path. If you want to use the performance monitor feature even after a system reboot, you need to register that mount path in fstab.

# mkdir /sys/kernel/debug # mount -t debugfs debugfs /sys/kernel/debug

Types

The target types of the performance measurement. Corresponds to the types argument of bsrmon.

iostat, ioclat, reqstat, peer_reqstat, alstat, network, sendbuf, memstat

iostat {resource} {vnr}

Aggregate data for read and write I/O on the replica volume (io count, kbs, kb)

Real-time monitoring of I/O performance data

> bsrmon /watch iostat r0 0
2020-12-06_08:22:36.223
  read : IO count=0, BW=0kb/s (0KB)
  write: IO count=216, BW=110208kb/s (220416KB)
2020-12-06_08:22:37.640
  read : IO count=0, BW=0kb/s (0KB)
  write: IO count=34, BW=16896kb/s (16896KB)
2020-12-06_08:22:39.32
  read : IO count=0, BW=0kb/s (0KB)
  write: IO count=326, BW=155176kb/s (155176KB)
2020-12-06_08:22:40.106
  read : IO count=0, BW=0kb/s (0KB)
  write: IO count=330, BW=164624kb/s (164624KB)
  ...

→ Situations that write I/O is constantly occurring

Statistical output of I/O performance data

→ Data aggregated for the period 2021-07-20_08:14:27.405 - 2021-08-04_18:59:53.573

→ 24 read I/O samples. 248 average, 40 minimum, 284 maximum Mbytes/s of IO processed.

→ 6670 write I/O samples. 320 average, 12 minimum, 401 maximum Mbytes/s of IO processed.

Calculate statistics using backup files

→ Extract data from the file vnr0_IO_STAT_2021-07-20_081427.005

→ Data aggregated for the period 2021-07-07_21:24:24.383 - 2021-07-20_08:14:26.371

→ 6452 read I/O samples. Measured with an average of 980 IOPS, a minimum of 1 IOPS, and a maximum of 1233 IOPS.

→ 28468 write I/O samples. Measured with an average of 786 IOPS, a minimum of 1 IOPS, and a maximum of 6478 IOPS.

ioclat {resource} {vnr}

local/master I/O 의 complete latency (usec)

Real-time monitoring of I/O complete latency

→ Situations where write I/O is constantly occurring

→ Output minimum, maximum, and average values for the time it took for local/master I/O to complete.

I/O complete latency statistics output

→ Data aggregated for the period 2020-12-07_02:42:31.440 - 2020-12-08_09:37:44.115

→ 50 local clat samples. measured to have taken an average of 8601, a minimum of 153, and a maximum of 1635570 usec.

→ 50 master clat samples. measured to have taken an average of 21801, a minimum of 217, and a maximum of 1667260 usecs.

reqstat {resource} {vnr}

Aggregate data on the time taken to reach a specific segment based on when the request was created (usec)

requests : Number of requests processed during the monitoring period (cycles)
before_queue : Time spent before submitting work queued up
before_al_begin : Time spent before getting LRU of ACT_LOG
in_actlog : Time spent before RQ_IN_ACT_LOG was set.
submit : Time taken for request submit to be executed
bio_endio: Time taken for bio end logic to be executed
pre_send: Time spent before sending data to peer
acked: Time spent before RQ_NET_PENDING was removed after sending.
net_done: Time spent until RQ_NET_DONE
destroy : Time taken until request is released

Time for each leg of the AL UPTATE to complete (usec)

al_update : Number of active logs updated during the monitoring period (cycle)
before_bm_write : Before bsr_bm_write_hinted() execution
after_bm_write : After execution of bsr_bm_write_hinted()
after_sync_page : after execution of bsr_md_sync_page_io()

Real-time monitoring of request performance data

→ Write I/O is occurring continuously

→ Disconnected from peer bsr03

→ AL update is not performed

Output request performance data statistics

→ 38180 requests were processed during the period 2020-12-06_05:46:36.678 - 2020-12-08_09:45:46.219

→ al_update never occurred

peer_reqstat {resource} {vnr}

Aggregate data on the time taken to reach a certain interval based on when the peer request was created (USEC)

peer_requests : Number of peer requests processed during the monitoring period (cycle)
submit : Time taken for a peer request submit to be executed
bio_endio: Time taken for the bio completion logic of a peer request to be executed.
destroy: Time taken for peer request to be destroyed

Real-time monitoring of peer request performance data

→ 100 peer requests for peer bsr-02 are processed

→ No peer requests for peer bsr-03 occurred

peer request performance statistics output

→ Aggregated data for the period 2021-07-11_22:23:08.890 - 2021-07-19_21:56:18.179

→ BIO completion logic took an average of 0.1 seconds to execute

alstat {resource} {vnr}

Aggregate data for usage figures in active log

Aggregate used values, maximum value of used values
Aggregate incremental values of HITS, MISSES, STARVING, LOCKED, CHANGED
Count of AL_WAIT RETRY
Aggregation of AL shortage causes
- starving, pending, used, busy, wouldblock

Real-time monitoring of active log performance data

→ 3 AL slots in use at that time

→ 1 al slot is pending

→ dirty, locked flag is set

active log performance statistics output

→ al_extents value changed from 6001 to 67 on 2021-08-03_22:47:52.895

→ When AL is set to 67, all AL slots are used 59 times, STARVING situation occurs 2 times, and AL_WAIT RETRY occurs 501 times

network {resource}

Replication network sending and receiving speed (byte/s)

sendbuf {resource}

Transmit buffer usage (bytes)

memstat

Memory usage by users and modules

In user space, the memory used by the bsradm, bsrsetup, bsrcon, bsrmon, and bsrservice processes is counted.

windows
- Memory information obtained through GetProcessMemoryInfo()
  - WorkingSetSize, QuotaPagedPoolUsage, QuotaNonPagedPoolUsage, PagefileUsage
linux
- Memory information obtained through the ps command
  - RSZ: Physical memory usage
  - vsz : Virtual memory usage

The following information about the memory used by the kernel module is aggregated by OS.

windows
- Nonpaged, paged memory usage allocated by the driver with the 'BS--' tag
linux
- slab cache information
- /sys/kernel/slab/bsr_req
- /sys/kernel/slab/bsr_al
- /sys/kernel/slab/bsr_bm
- /sys/kernel/slab/bsr_ee

resync_ratio {resource} {vnr}

Real-time replication, the amount of synchronization transfers and the ratio of synchronization transfers to it.

Updated only on the source node when synchronization is in progress.

Commands

/start

Enables the Performance Monitor feature and starts aggregating performance data and logging files. By default, Performance Monitor is enabled.

Enabling Performance Monitor is done through the following process

windows
- HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\bsr\bsrmon_run registry value to 1
- Enable the BSR Engine's performance data aggregation logic
- Enable bsrmon /file execution logic in bsrservice
linux
- Set the value of /etc/bsr.d/.bsrmon_run to 1
- Enable BSR Engine's performance data aggregation logic
- Run the /lib/bsr/bsrmon-run script. Run the bsrmon /file command periodically

/stop

Disable the performance monitor. Disabling stops the engine's performance data aggregation and file logging.

Deactivation is performed as follows

windows
- HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\bsr\bsrmon_run Set the registry value to 0
- Disable the BSR Engine's performance data aggregation logic
- Disable the bsrmon /file run logic in bsrservice
linux
- Set the value of /etc/bsr.d/.bsrmon_run to 0
- Disable BSR Engine's performance data aggregation logic
- Exit the /lib/bsr/bsrmon-run script

/status

Query the behavioral status of the performance monitor.

/file

Save performance data and memory information to a file. This is a hidden command and is used by the bsrservice and bsrmon-run scripts.

The file is stored in the following location

windows : C:\Program Files\bsr\log\perfmon
linux : /var/log/bsr/perfmon/

Only the lookup time and the numbers for each item are stored, and the data is parsed and formatted for output when you run the /show, /watch, or /report command.

/show

[/t {types[,...]|all}] [/r {resource[,...]|all}] [/j|/json] [/c|/continue]
Outputs performance data for all entries for all resources at once. The output target is the last data written to the performance file.
[/t {types[,...]|all}] [/r {resource[,...]|all}]
Provides the option to specify specific performance items and resources for output. Enter performance item and resource names separated by ',' and without spaces.
[/j|/json]
Outputs data in JSON format.
[/c|/continue]
Outputs performance data periodically. The output period follows the period value set in bsrmon.
Usage examples
Output everything on all resources

Output specific items from all resources

Output specific items from specific resources

/watch

{types} [/scroll]

Displays the data being aggregated by type in real-time. When using the /scroll option, the output is displayed in a line-by-line fashion.

/report

{types} [/f {filename}] [/p {peer_name[,...]}] [/d {YYYY-MM-DD}] [/s {timestamp}] [/e {timestamp}]

Output statistics of data written to files (including data from backed up files) by type.

[/f {filename}]

Enter the name of the file to report on. This is used to calculate statistics for a specific file.

[/p {peer_name[,...]}]

Enter the name of the peer to report on. This is used to calculate statistics for a specific peer. Separate multiple PEER entries with commas (,) without spaces. If not entered, output statistics to the PEERs of the resource configured on the server.

[/d {YYYY-MM-DD}] [/s {YYYY-MM-DD|hh:mm[:ss]|YYYY-MM-DD_hh:mm[:ss]}] [/e {YYYY-MM-DD|hh:mm[:ss]|YYYY-MM-DD_hh:mm[:ss]}]

This option allows you to query figures for a specific period. The /d option outputs statistics of data recorded on that date. /s specifies the start date and time of the query. /e specifies the end date and time of the query. When entering the /s, /e options, the date and time are separated by an underscore (_).

/set

{period, file_size, file_cnt} {value}
Commands to adjust numbers related to monitoring.
period
Sets the file saving and monitoring interval. Set in seconds, the default value is 1 second.
file_size
Set the file rolling size. Set in MB, the default value is 50 MB.
file_cnt
Set the number of file rolls. The default value is 3.

/get

{all, period, file_size, file_cnt}

Commands to get numbers related to monitoring.

/io_delay_test

{flag} {delay point} {delay time}

Intentionally cause I/O performance degradation to validate the functionality of the bsr performance monitor. This is a developer feature.

/debug cmds options

Commands for windows to retrieve debugfs information.