Versions Compared

Old Version 1

changes.mady.by.user SungEun Kim

Saved on

compared with

New Version 2

changes.mady.by.user SungEun Kim

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Table of Contents
minLevel1
maxLevel3

Overview

bsrmon is a performance monitoring tool that tracks the entire replication process of the bsr engine that performs a BSR engine performing replication and records the time required for taken by each logical section of the engine's logic in real time to identify bottlenecks in the engine operation's behavior. The bsr BSR kernel engine maintains performance-related statistical information such as time records for each section and accumulated calculation values at the time the engine was operated in the form of a file log through the internal jiffies record code. Administrators can monitor the statistics information of these kernel engines by querying them statistics in the form of file logs, including a section-by-section time history of when the engine was running through internal time (jiffies) records and cumulative calculations. Administrators can monitor these statistical information of the kernel engine by querying with the bsrmon utility. bsrmon is supported since from version 1.6.1.

Functions

Features provided by bsrmon include:

/start

...

Info

In a linux environment, the Performance Monitor feature

...

Activation of the performance monitor is performed through the following process.

  • windows

    • set HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\bsr\bsrmon_run registry value to 1

    • Activating the performance data aggregation logic of the bsr engine

    • Enable bsrservice's bsrmon /file execution logic

  • linux

    • Set the /etc/bsr.d/.bsrmon_run value to 1.

    • Activating the performance data aggregation logic of the bsr engine

    • /lib/bsr/bsrmon-run Run the script. Run the bsrmon /file command periodically

/stop

Disable Performance Monitor. When disabled, the engine stops aggregating performance data and logging files.

Deactivation is performed as follows.

  • windows

    • Set HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\bsr\bsrmon_run registry value to 0

    • Disable performance data aggregation logic of bsr engine

    • Disable bsrservice's bsrmon /file execution logic

  • linux

    • Set /etc/bsr.d/.bsrmon_run value to 0

    • Disable performance data aggregation logic of bsr engine

    • Stop the /lib/bsr/bsrmon-run script

/status

Query the operating status of the performance monitor.

/file

Saves performance data and memory information to a file. This is a hidden command and is used by the bsrservice and bsrmon-run scripts.

The file storage location is as follows.

  • windows : C:\Program Files\bsr\log\perfmon

  • linux : /var/log/bsr/perfmon/

Data is saved only for the inquiry time and the numerical value for each item, and when the /watch or /report command is executed, the data is parsed and output according to the format.

Code Block
2021-07-20_05:55:21.443 955 955 244480 244480 1042 1042 266752 266752

/watch {types} [/scroll]

Data that is being aggregated by type is output in real time. When the /scroll option is used, the output is displayed in newline format.

/report {types} [/f {filename}] [/d {YYYY-MM-DD}] [/s {hh:mm[:ss]}] [/e {hh:mm[:ss]}]

Outputs the statistics of the data recorded by the file for each type.

When generating statistics, meaningless data such as numerical values are excluded from the target sample.

  • Data written to zero with no I/O occurring

  • I/O data generated temporarily (when the I/O number is not consecutively more than 2 times)

  • ex) vnr0_IO_STAT

    Code Block
    2021-07-07_21:27:52.304 0 0 0 0 0 0 0 0 // Excluded from samples as recorded performance data is zero
2021-07-07_21:27:53.325 0 0 0 0 2 2 1 1 // Excluded from samples as transient I/O performance data
2021-07-07_21:27:54.346 0 0 0 0 0 0 0 0
...
2021-07-07_21:30:28.549 0 0 0 0 408 408 104448 104448 
2021-07-07_21:30:29.580 0 0 0 0 409 409 104704 104704 // Performance data is not zero and is being recorded continuously. selected as samples
2021-07-07_21:30:30.605 0 0 0 0 404 404 103424 103424

[/f {filename}]

the file name for the report. Used for statistical calculation of backed up files.

[/d {YYYY-MM-DD}] [/s {hh:mm[:ss]}] [/e {hh:mm[:ss]}]

This option allows you to inquire the figures for a specific section.

  • /d : Outputs statistics of data recorded on that date. If not entered, statistics for the entire period are output.

  • /s : Outputs statistics of data recorded from that time. If not input, statistics of recorded data are output from 00:00:00.

  • /e : Outputs statistics of data recorded up to that time. If not input, statistics of data collected until 23:59:59 are output.

/set {period, file_size, file_cnt} {value}

Command to adjust the numerical values related to monitoring.

period

Set file saving and monitoring cycle. It is set in seconds, and the default value is 1 second.

file_size

Sets the file rolling size. Set in MB unit, the default value is 50 MB.

file_cnt

Sets the number of file rolling. The default value is three.

/io_delay_test {flag} {delay point} {delay time}

Verifies the functionality of the bsr performance monitor by intentionally incurring I/O performance degradation. This feature is for developers.

/debug cmds options

Command for windows to inquire debugfs information.

types

Performance monitoring targets.

iostat, ioclat, reqstat, peer_reqstat, alstat, network, sendbuf, memstat

iostat {resource} {vnr}

Aggregate data for read and write I/O of replication volumes (iops, ios, kbs, kb)

  • Statistical output of I/O performance data

> bsrmon /report iostat r0 0 Report r0 [IO STAT

is only available in kernels with the debugfs feature enabled, and debugfs must be mounted in the /sys/kernel/debug path.

You can check if the debugfs feature is enabled using the following command.

Expand
titleUsage Example

  • Real-time monitoring of I/O performance data

Code Block
> bsrmon /watch iostat r0 0
2020-12-06_08:22:36.223
  read : IOPS=0 (IOs=0), BW=0kb/s (0KB)
  write: IOPS=216 (IOs=432), BW=110208kb/s (220416KB)
2020-12-06_08:22:37.640
  read : IOPS=0 (IOs=0), BW=0kb/s (0KB)
  write: IOPS=34 (IOs=34), BW=16896kb/s (16896KB)
2020-12-06_08:22:39.32
  read : IOPS=0 (IOs=0), BW=0kb/s (0KB)
  write: IOPS=326 (IOs=326), BW=155176kb/s (155176KB)
2020-12-06_08:22:40.106
  read : IOPS=0 (IOs=0), BW=0kb/s (0KB)
  write: IOPS=330 (IOs=330), BW=164624kb/s (164624KB)
  ...

→ A situation in which write I/O is continuously occurring

Code Block
Code Block
# grep CONFIG_DEBUG_FS /boot/config-`uname -r`
CONFIG_DEBUG_FS=y

Since debugfs is automatically mounted starting with CentOS7, and not automatically mounted in CentOS6, you can use the performance monitor feature after mounting it in the /sys/kernel/debug path. If you want to use the performance monitor feature even after a system reboot, you need to register that mount path in fstab.

  • Code Block
    # mkdir /sys/kernel/debug
# mount -t debugfs debugfs /sys/kernel/debug

Types

The target types of the performance measurement. Corresponds to the types argument of bsrmon.

iostat, ioclat, reqstat, peer_reqstat, alstat, network, sendbuf, memstat

iostat {resource} {vnr}

Aggregate data for read and write I/O on the replica volume (io count, kbs, kb)

Expand
titleUsage examples

  • Real-time monitoring of I/O performance data

Code Block
> bsrmon /watch iostat r0 0
2020-12-06_08:22:36.223
  read : IO count=0, BW=0kb/s (0KB)
  write: IO count=216, BW=110208kb/s (220416KB)
2020-12-06_08:22:37.640
  read : IO count=0, BW=0kb/s (0KB)
  write: IO count=34, BW=16896kb/s (16896KB)
2020-12-06_08:22:39.32
  read : IO count=0, BW=0kb/s (0KB)
  write: IO count=326, BW=155176kb/s (155176KB)
2020-12-06_08:22:40.106
  read : IO count=0, BW=0kb/s (0KB)
  write: IO count=330, BW=164624kb/s (164624KB)
  ...

→ Situations that write I/O is constantly occurring

  • Statistical output of I/O performance data

Code Block
> bsrmon /report iostat r0 0
Report r0 [IO STAT - vnr0]
 Run: 2021-07-20_08:14:27.405 - 2021-08-04_18:59:53.573
  read : io count=24855, bw=6325194kbyte
    BW (kbyte/s): min=40, max=284928, avg=248578, samples=24
  write: io count=8360345, bw=2139182512kbyte
    BW (kbyte/s): min=12, max=401408, avg=320256, samples=6671

→ Data aggregated for the period 2021-07-20_08:14:27.405 - 2021-08-04_18:59:53.573

→ 24 read I/O samples. 248 average, 40 minimum, 284 maximum Mbytes/s of IO processed.

→ 6670 write I/O samples. 320 average, 12 minimum, 401 maximum Mbytes/s of IO processed.

  • Calculate statistics using backup files

Code Block
> bsrmon /report iostat r0 0 /f vnr0_IO_STAT_2021-07-20_081427.005 
Report r0 [vnr0_IO_STAT_2021-07-20_081427.005]
 Run: 2021-07-07_21:24:24.383 - 2021-07-20_08:14:26.371
  read : ios=6332219, bw=1595360917kbyte
    IOPS        : min=1, max=1233, avg=980, samples=6452
    BW (kbyte/s): min=4, max=428032, avg=247033, samples=6452
  write: ios=22550577, bw=3961426883kbyte
    IOPS        : min=1, max=6478, avg=786, samples=28468
    BW (kbyte/s): min=4, max=1085440, avg=138461, samples=28484

→ Extract data from the file vnr0_IO_STAT_2021-07-20_081427.005

→ Data aggregated for the period 2021-07-07_21:24:24.383 - 2021-07-20_08:14:26.371

→ 6452 read I/O samples. Measured with an average of 980 IOPS, a minimum of 1 IOPS, and a maximum of 1233 IOPS.

→ 28468 write I/O samples. Measured with an average of 786 IOPS, a minimum of 1 IOPS, and a maximum of 6478 IOPS.

ioclat {resource} {vnr}

local/master I/O 의 complete latency (usec)

  • Calculate statistics using a backup file

Expand
titleUsage examples

  • Real-time monitoring of I/O complete latency

Code Block
> bsrmon /watch ioclat r0 0
2020-12-08_09:33:19.171
  local clat  (usec): min=520, max=112000, avg=10347
  master clat (usec): min=610, max=256852, avg=28202
2020-12-08_09:33:20.351
  local clat  (usec): min=780, max=478387, avg=44316
  master clat (usec): min=1499, max=492829, avg=114106
2020-12-08_09:33:21.509
  local clat  (usec): min=478, max=19805, avg=4523
  master clat (usec): min=577, max=24335, avg=6303
  ...

→ Situations where write I/O is constantly occurring

→ Output minimum, maximum, and average values for the time it took for local/master I/O to complete.

  • I/O complete latency statistics output

Code Block
> bsrmon /report ioclat r0 0 
Report r0 [IO COMPLETE - vnr0]
 Run:
2021
2020-12-07
-20
_
08
02:
14
42:
27
31.
405
440 -
2021
2020-12-08
-04
_
18
09:
59
37:
53
44.
573
115
read : ios=24855, bw=6325194kbyte IOPS 
local clat  (usec): min=
10
153, max=
1113
1635570, avg=
972
8601, samples=
24
50
  master
BW
clat (
kbyte/s
usec): min=
40
217, max=
284928
1667260, avg=
248578
21801, samples=
24 write: ios=8360345, bw=2139182512kbyte IOPS : min=3, max=1568, avg=1252, samples=6670 BW (kbyte/s): min=12, max=401408, avg=320256, samples=6671

→ Data aggregated during the period 2021-07-20_08:14:27.405 - 2021-08-04_18:59:53.573

→ 24 read I/O samples. Measured at an average of 972 iops, a minimum of 10 iops, and a maximum of 1113 iops

→ 6670 write I/O samples. Measured at 1252 iops average, 3 iops min, 1568 iops max

Code Block
> bsrmon /report iostat r0 0 /f vnr0_IO_STAT_2021-07-20_081427.005 
Report r0 [vnr0_IO_STAT_2021-07-20_081427.005]
 Run: 2021-07-07_21:24:24.383 - 2021-07-20_08:14:26.371
  read : ios=6332219, bw=1595360917kbyte
    IOPS        : min=1, max=1233, avg=980, samples=6452
    BW (kbyte/s): min=4, max=428032, avg=247033, samples=6452
  write: ios=22550577, bw=3961426883kbyte
    IOPS        : min=1, max=6478, avg=786, samples=28468
    BW (kbyte/s): min=4, max=1085440, avg=138461, samples=28484

→ Extract data from file vnr0_IO_STAT_2021-07-20_081427.005

→ Data aggregated for the period 2021-07-07_21:24:24.383 - 2021-07-20_08:14:26.371

→ 6452 read I/O samples. Measured at an average of 980 iops, a minimum of 1 iops, and a maximum of 1233 iops

→ 28468 write I/O samples. Measured average 786 iops, min 1 iops, max 6478 iops

ioclat {resource} {vnr}

Complete latency of local/master I/O (usec)

> bsrmon /watch ioclat r0 0 2020-12-08_09:33:19.171 local clat
Expand
titleUsage Example

  • Real-time monitoring of I/O complete latency

Code Block
50

→ Data aggregated for the period 2020-12-07_02:42:31.440 - 2020-12-08_09:37:44.115

→ 50 local clat samples. measured to have taken an average of 8601, a minimum of 153, and a maximum of 1635570 usec.

→ 50 master clat samples. measured to have taken an average of 21801, a minimum of 217, and a maximum of 1667260 usecs.

reqstat {resource} {vnr}

Aggregate data on the time taken to reach a specific segment based on when the request was created (usec)

  • requests : Number of requests processed during the monitoring period (cycles)

  • before_queue :Time spent before submitting work queued up

  • before_al_begin : Time spent before getting LRU of ACT_LOG

  • in_actlog : Time spent before RQ_IN_ACT_LOG was set.

  • submit : Time taken for request submit to be executed

  • bio_endio: Time taken for bio end logic to be executed

  • pre_send: Time spent before sending data to peer

  • acked: Time spent before RQ_NET_PENDING was removed after sending.

  • net_done: Time spent until RQ_NET_DONE

  • destroy : Time taken until request is released

Time for each leg of the AL UPTATE to complete (usec)

  • al_update : Number of active logs updated during the monitoring period (cycle)

  • before_bm_write : Before bsr_bm_write_hinted() execution

  • after_bm_write : After execution of bsr_bm_write_hinted()

  • after_sync_page : after execution of bsr_md_sync_page_io()

  • I/O complete latency statistics output

> bsrmon /report ioclat r0 0 Report r0 [IO COMPLETE - vnr0] Run: 2020-12-07_02:42:31.440 - 2020-12-08_09:37:44.115 local clat
Expand
titleUsage examples

  • Real-time monitoring of request performance data

Code Block
> bsrmon /watch reqstat r0 0
2020-12-08_09:42:18.549
  requests  : 445
    before_queue    (usec): min=0, max=19, avg=0
    before_al_begin (usec): min=0, max=4907, avg=11
    in_actlog       (usec): min=1, max=7191, avg=35
    submit          (usec): min=3, max=7215, avg=46
    bio_endio       (usec): min=4, max=7219, avg=50
    destroy         (usec): min=577, max=214980, avg=15711
  al_uptate  : 0
    before_bm_write (usec): min=
520
0, max=
112000
0, avg=
10347
0
master
after_bm_write
clat
 (usec): min=
610
0, max=
256852
0, avg=
28202 2020-12-08_09:33:20.351
0
local
clat
 after_sync_page (usec): min=
780
0, max=
478387
0, avg=
44316
0
  PEER bsr03:
master
clat
pre_send (usec): min=
1499
0, max=
492829
0, avg=
114106 2020-12-08_09:33:21.509
0
    acked
local
clat
  (usec): min=
478
0, max=
19805
0, avg=
4523
0
master
clat
net_done (usec): min=
577
0, max=
24335
0, avg=
6303
0
  PEER 100.100.
.

→ A situation in which write I/O is continuously occurring

→ Outputs the minimum, maximum, and average values of the time taken for local/master I/O to complete

Code Block
10.31:7792:
    pre_send (usec): min=571, max=214862, avg=15703
    acked    (usec): min=
153
555, max=
1635570
210004, avg=
8601,
7966
samples=50
master
clat
net_done (usec): min=
217
575, max=
1667260
214978, avg=
21801, samples=50

→ Data aggregated for the period 2020-12-07_02:42:31.440 - 2020-12-08_09:37:44.115

→ 50 local clat samples. Measured to have consumed an average of 8601, a minimum of 153, and a maximum of 1635570 usec

→ 50 master clat samples. Measured to have consumed an average of 21801, a minimum of 217, and a maximum of 1667260 usec

reqstat {resource} {vnr}

Aggregate data on the time taken from the time of request creation to a specific section (usec)

...

requests : The number of requests processed during the monitoring period (cycle)

...

before_al_begin : Time elapsed before getting the lru of act_log

...

in_actlog : Time taken for RQ_IN_ACT_LOG to be set

...

submit : Time taken until request submit is executed

...

bio_endio : Time taken for bio completion logic to be executed

...

pre_send : Time taken before sending data to peer

...

acked : Time elapsed until RQ_NET_PENDING is removed after send

...

net_done : Time taken to RQ_NET_DONE

...

destroy : The time it takes for the request to be released

Time required for each section until al uptate is completed (usec)

  • al_update : The number of active logs updated during the monitoring period (cycle)

  • before_bm_write : Before executing bsr_bm_write_hinted()

  • after_bm_write : After executing bsr_bm_write_hinted()

  • after_sync_page : After executing bsr_md_sync_page_io()

> bsrmon /watch reqstat r0 0 2020-12-08_09:42:18.549 requests : 445 before_queue
Expand
titleUsage Example

  • Real-time monitoring of request performance data

Code Block
15710
...

→ Write I/O is occurring continuously

→ Disconnected from peer bsr03

→ AL update is not performed

Code Block
> bsrmon /report reqstat r0 0 
Report r0 [REQUEST STAT - vnr0]
 Run: 2020-12-06_05:46:36.678 - 2020-12-08_09:45:46.219
  requests : total=38180
    before_queue    (usec): min=0, max=0, avg=0, samples=0
    before_al_begin (usec): min=6966, max=460112, avg=4264, samples=2
    in_actlog       (usec): min=
0
1, max=
19
589946, avg=
0 
549, samples=28
before_al_begin
(usec): min=0, max=4907, avg=11
 submit
in_actlog 
   
(usec): min=
1
2, max=
7191
721793, avg=
35 submit (usec): min=3, max=7215, avg=46
349, samples=72
    bio_endio       (usec): min=
4
3, max=
7219
589987, avg=
50
579, samples=28
    destroy         (usec): min=
577
83, max=
214980
2362749, avg=
15711
46380, samples=161
  al_uptate 
: total=0
    before_bm_write (usec): min=0, max=0, avg=0, samples=0
    after_bm_write  (usec): min=0, max=0, avg=0, samples=0
    after_sync_page (usec): min=0, max=0, avg=0, samples=0
  PEER
bsr03
100.100.10.31:7792:
    pre_send (usec): min=
0
7, max=
0
1667187, avg=22364, samples=
0
161
    acked    (usec): min=
0
11, max=
0
1635581, avg=
0
15737, samples=161
    net_done (usec): min=
0
82, max=
0
2362749, avg=
0
39406, samples=161
  PEER
100.100.10.31:7792:
bsr03:
    pre_send (usec): min=
571
0, max=
214862
0, avg=0, samples=
15703
0
    acked    (usec): min=
555
0, max=
210004
0, avg=
7966
0, samples=0
    net_done (usec): min=
575
0, max=
214978
0, avg=
15710 ...

→ A situation in which write I/O is continuously occurring

→ Disconnect from peer bsr03

→ al update is not performed

Request performance data statistics output
0, samples=0

→ 38180 requests were processed during the period 2020-12-06_05:46:36.678 - 2020-12-08_09:45:46.219

→ al_update never occurred

peer_reqstat {resource} {vnr}

Aggregate data on the time taken to reach a certain interval based on when the peer request was created (USEC)

Expand
titleUsage examples

  • Real-time monitoring of peer request performance data

Code Block
> bsrmon /
report
watch peer_reqstat r0 0
Report r0 [REQUEST STAT - vnr0] Run: 2020-12-06_05:46:36.678 - 2020-12-08_09:45:46.219 
2021-07-19_21:52:02.191
  PEER bsr-03:
    peer requests :
total=38180
0
before_queue
submit    (usec): min=0, max=0, avg
=0, samples
=0
before
bio_
al_begin
endio (usec): min=
6966
0, max=
460112
0, avg=
4264,
0
samples=2
    destroy
in_actlog 
  
(usec): min=
1
0, max=
589946
0, avg=
549, samples=28
0
  PEER bsr-02:
    peer requests
submit
: 100
    submit    (usec): min=
2
421, max=
721793
6907, avg=
349, samples=72
2184
    bio_endio
 (usec): min=
3
1021, max=
589987
7312, avg=
579, samples=28
2563
    destroy   
(usec): min=
83
1739, max=
2362749
7955, avg=
46380, samples=161 al_uptate : total=0 before_bm_write (usec): min=0, max=0, avg=0, samples=0 after_bm_write (usec): min=0, max=0, avg=0, samples=0 after_sync_page
3244

→ 100 peer requests for peer bsr-02 are processed

→ No peer requests for peer bsr-03 occurred

  • peer request performance statistics output

Code Block
> bsrmon /report peer_reqstat r0 0
Report r0 [PEER REQUEST STAT - vnr0]
 Run: 2021-07-11_22:23:08.890 - 2021-07-19_21:56:18.179
  PEER bsr-02:
    peer requests : total=344054
    submit    (usec): min=
0
1, max=
0
36103, avg=
0
99, samples=
0
6902
PEER
100.100.10.31:7792: pre_send
 bio_endio (usec): min=
7
1, max=
1667187
116988, avg=
22364
96161, samples=
161
9517
acked
destroy   
(usec): min=
11
47, max=
1635581
117495, avg=
15737
96340, samples=
161
9518
  PEER
net_done (usec): min=82, max=2362749, avg=39406, samples=161 PEER bsr03: 
bsr-03:
    peer requests : total=133288
    submit
pre_send 
(usec): min=
0
1, max=
0
1670, avg=
0
5, samples=
0
6037
acked 
bio_endio (usec): min=
0
1, max=
0
117000, avg=
0
104660, samples=
0
8708
    destroy
net_done
  (usec): min=
0
63, max=
0
125871, avg=
0
104839, samples=
0
8709

→

38180 requests processed during

Aggregated data for the period

2020

2021-

12

07-

06

11_

05

22:

46

23:

36

08.

678

890 -

2020

2021-

12

07-

08

19_

09

21:

45

56:

46

18.

219

→ al_update never occurs

...

179

→ BIO completion logic took an average of 0.1 seconds to execute

alstat {resource} {vnr}

Aggregate data (usec) about the time taken from the peer request creation time to a specific section

...

peer_requests : Number of peer requests processed during the monitoring period (cycle)

...

submit : Time taken until peer request submit is executed

...

bio_endio : Time taken until the bio completion logic of the peer request is executed

...

for usage figures in active log

> bsrmon /report peer_reqstat r0 0 Report r0 [PEER REQUEST STAT - vnr0] Run: 2021-07-11_22:23:08.890 - 2021-07-19_21:56:18.179 PEER bsr-02:
Expand
titleUsage Exampleexamples

  • Real-time monitoring of peer request active log performance data

Code Block
> bsrmon /watch
peer_reqstat
alstat r0 0

2021-
07
08-04_19
_21
:
52
10:
02.191
11.463
  used
PEER
bsr-03
:
peer
requests
:
0
 3/67 (max=8)
submit
hits
(usec)
:
min=0,
max=0,
avg=0
bio_endio
579 (
usec): min=0, max=0, avg=0
total=2843)
  misses  :
destroy
  2250 (
usec): min=0, max=0, avg=0
total=17564)
  starving:
PEER
bsr-02:
 0 (total=0)
peer
locked
requests
 :
100
submit
   0 (
usec)
total=0)
  changed :
min=421,
max=6907,
avg=2184
bio_endio
749 (
usec): min=1021, max=7312, avg=2563
total=5854)
  al_wait retry :
destroy
  0 (
usec): min=1739
total=0, max=
7955, avg=3244

→ 100 peer requests processed for bsr-02

→ No peer request for bsr-03

Code Block
0)
  pending_changes :  1/64
  error   : 0
    NOBUFS - starving : 0
peer
requests
:
total=344054
- pending slot : 0
submit
(usec):
min=1,
max=36103,
avg=99,
samples=6902
   - used
bio_endio (usec): min=1, max=116988, avg=96161, samples=9517 destroy (usec): min=47, max=117495, avg=96340, samples=9518 PEER bsr-03: peer requests : total=133288 submit (usec): min=1, max=1670, avg=5, samples=6037 bio_endio (usec): min=1, max=117000, avg=104660, samples=8708 destroy (usec): min=63, max=125871, avg=104839, samples=8709

→ Data aggregated during the period 2021-07-11_22:23:08.890 - 2021-07-19_21:56:18.179

→ It took an average of 0.1 seconds for the bio completion logic to be executed.

alstat {resource} {vnr}

Aggregate data for usage figures in active log

> bsrmon /watch alstat r0 0 2021-08-04_19:10:11.463 used : 3/67 (max=8) hits : 579 (total=2843) misses : 2250 (total=17564) starving: 0 (total=0) locked : 0 (total=0) changed : 749 (total=5854)
Expand
titleUsage Example

  • Real-time monitoring of activity log performance data

Code Block
slot : 0
    BUSY : 0
    WOULDBLOCK : 0
  flags   : __LC_DIRTY __LC_LOCKED

→ 3 AL slots in use at that time

→ 1 al slot is pending

→ dirty, locked flag is set

Code Block
> bsrmon /report alstat r0 0
Report r0 [AL STAT - vnr0]
 Run: 2021-08-03_04:48:33.721 - 2021-08-03_22:47:28.326
  al_extents : 6001
    used     : max=0(all_slot_used=0), avg=0
    hits     : total=0
    misses   : total=0
    starving : total=0
    locked   : total=0
    changed  : total=0
    al_wait retry count : max=0, total=0
    pending_changes     : max=0, total=0
    error : total=0
      NOBUFS - starving     : total=0
             - pending slot : total=0
             - used    slot : total=0
      BUSY       : total=0
      WOULDBLOCK : total=0
 -> al_extents changed 
 Run: 2021-08-03_22:47:52.895 - 2021-08-04_19:27:06.522
  al_extents : 67
    used     : max=67(all_slot_used=59), avg=3
    hits     : total=337528
    misses   : total=2020409
    starving : total=2
    locked   : total=0
    changed  : total=673370
    al_wait retry count : max=2, total=501
    pending_changes
0 (total=0
  : max=64,
max=0)
total=258
pending_changes
error : total=1004
1/64
error
:
NOBUFS
0
- starving
NOBUFS
-
:
starving
total=501
:
0
            - pending slot : total=0
             - used    slot : total=503
0
      BUSY
:
0
WOULDBLOCK
 : total=0
flags
:
__LC_DIRTY __LC_LOCKED

→ There are 3 al slots in use at the time

→ 1 pending al slot

→ The dirty, locked flag is set

activity log performance statistics output
WOULDBLOCK : total=0

→ al_extents value changed from 6001 to 67 on 2021-08-03_22:47:52.895

→ When AL is set to 67, all AL slots are used 59 times, STARVING situation occurs 2 times, and AL_WAIT RETRY occurs 501 times

network {resource}

Replication network sending and receiving speed (byte/s)

Expand
titleUsage examples

  • Real-time monitoring of network performance data

Code Block
> bsrmon /
report
watch
alstat
network r0
0 Report r0 [AL STAT - vnr0] Run: 2021-08-03_04:48:33.721 - 2021-08-03_22:47:28.326 al_extents : 6001 used : max=0(all_slot_used=0), avg=0 hits : total=
2020-12-08_09:47:45.84
  PEER 100.100.10.31:7792:
    send (byte/s): 62932184
    recv (byte/s): 4820
  PEER bsr03:
    send (byte/s): 0
misses 
recv (byte/s):
total=
0
2020-12-08_09:47:46.160
starving : total=0
PEER 100.100.10.31:7792:
locked 
send (byte/s): 104896568
:
total=0
changed : total=0
recv (byte/s): 12320
al_wait retry count : max=0, total=0
PEER bsr03:
pending_changes : max=0, total=0
send (byte/s): 0
error 
recv (byte/s):
total=0 NOBUFS - starving : total=0 
0
...

→ peer bsr03 과는 연결 단절된 상태

→ peer 100.100.10.31:7792 로 데이터 전송이 진행되고 있는 중

  • network 성능 통계 출력

Code Block
> bsrmon /report network r0
Report r0 [NETWORK SPEED]
 Run: 2020-11-30_06:38:44.653 - 2020-12-08_09:53:31.455
  PEER 100.100.10.31:7792: send=52497905byte/s, receive=5902byte/s
- pending slot 
send (byte/s):
total
min=
0 
3, max=115395016, avg=52497905, samples=392
    recv (byte/s): min=3, max=15004, avg=5902, samples=367
-
used
 PEER
slot 
bsr03:
total=0 
send=0byte/s, receive=0byte/s
BUSY : total
send (byte/s): min=0, max=0, avg=0, samples=0
    recv
WOULDBLOCK : total=0 -> al_extents changed Run: 2021-08-03_22:47:52.895 - 2021-08-04_19:27:06.522 al_extents : 67 used : max=67(all_slot_used=59), avg=3 hits : total=337528 misses : total=2020409 starving : total=2 locked : total=0 changed : total=673370 al_wait retry count : max=2, total=501 pending_changes : max=64, total=258 error : total=1004 NOBUFS - starving : total=501
(byte/s): min=0, max=0, avg=0, samples=0

→ 평균적으로 52.4 MB 의 데이터가 peer 100.100.10.31 노드로 전송되고 있음

sendbuf {resource}

송신버퍼 사용량 (bytes)

Expand
title사용 예

  • 송신버퍼 성능 데이터 실시간 모니터링

Code Block
> bsrmon /watch sendbuf r0
2020-12-08_09:54:13.735
  PEER 100.100.10.31:7792:
    highwater: 11246, fill: 37094400bytes
        ap_in_flight: 8881 (9094144bytes)
        rs_in_flight: 2365 (28000256bytes)
    data stream
        size (bytes): 10485760000
        used (bytes): 27098576
         [P_DATA]  -  cnt: 7542  size: 8024616bytes
         [P_RS_DATA_REPLY]  -  cnt: 1596  size: 18942304bytes
         [P_BARRIER]  -  cnt: 549  size: 13176bytes
         [P_UNPLUG_REMOTE]  -
-
cnt:
pending
7406
slot
 size: 118496bytes
total=0
   control stream
        size
- used
(bytes): 5242880
slot
:
total=503
   used (bytes): 0
BUSY
 PEER bsr03:
    highwater:
total=0
0, fill: 0bytes
WOULDBLOCK
 ap_in_flight:
total=0

→ al_extents value changed from 6001 to 67 on 2021-08-03_22:47:52.895

→ If al is set to 67, the number of times that all al slots are used 59 times, starving situation occurs 2 times, and al_wait retry occurs 501 times.

network {resource}

Replication network transmit/receive speed (byte/s)

→ Disconnected from peer bsr03

→ Data transmission to
Expand
titleUsage Example

  • Real-time monitoring of network performance data

Code Block
> bsrmon /watch network r0
2020-12-08_09:47:45.84
  PEER 100.100.10.31:7792:
    send (byte/s): 62932184
    recv (byte/s): 4820
  PEER bsr03:
    send (byte/s): 0
    recv (byte/s): 0
2020-12-08_09:47:46.160
  PEER 100.100.10.31:7792:
    send (byte/s): 104896568
    recv (byte/s): 12320
  PEER bsr03:
    send (byte/s): 0
    recv (byte/s): 0
...
0 (0bytes)
        rs_in_flight: 0 (0bytes)
    data stream
        size (bytes): 10485761
        used (bytes): 0
    control stream
        size (bytes): 5242881
        used (bytes): 0
...

→ peer 100.100.10.31:7792, bsr03 에 송신버퍼가 할당되어 있음

→ peer bsr03 과는 연결 단절된 상태

→ peer 100.100.10.31:7792 로 데이터 전송이 진행되고 있는 중

→ peer 100.100.10.31:7792

is in progressoutput network performance statistics

에 계류중인 복제 및 동기화 데이터 갯수는 11246 개 (복제 데이터 8881개, 동기화 데이터 2365개)

Code Block
> bsrmon /report
network
sendbuf r0
Report r0 [
NETWORK
SEND
SPEED
BUFFER]
 Run: 2020-
11
12-
30
05_
06
13:
38
26:
44
59.
653
969 - 2020-12-08_09:
53
56:
31
33.
455
718
  PEER 100.100.10.31:7792:
send=52497905byte/s, receive=5902byte/s
data stream size=10485761byte, control stream size=5242881byte
send (byte/s
data-used (bytes): min=
3
2097232, max=
115395016
7603084, avg=
52497905
4787174, samples=
392
5
recv
cntl-used (
byte/s
bytes): min=
3
0, max=0, avg=0, samples=0
    highwater: min=1, max=
15004
8014, avg=
5902
760, samples=
367
999
  PEER bsr03: data stream
send
size=
0byte/s, receive=0byte/s
10485761byte, control stream size=5242881byte
send
data-used (
byte/s
bytes): min=0, max=0, avg=0, samples=0
recv
cntl-used (
byte/s
bytes): min=0, max=0, avg=0, samples=0

→

On average, 52.4 MB of data is being transferred to the peer 100.100.10.31 node.

sendbuf {resource}

Send buffer usage (bytes)

> bsrmon /watch sendbuf r0 2020-12-08_09:54:13.735 PEER 100.100.10.31:7792: data stream size (bytes): 10485761 used (bytes): 204920 [P_RS_DATA_REPLY] - cnt: 3 size: 209016 bytes control stream size (bytes): 5242881
Expand
titleUsage Example

  • Real-time monitoring of transmit buffer performance data

Code Block

평균적으로 10MB 중 4.7MB 의 버퍼가 사용되고 있음

memstat

유저 및 모듈에서 사용하는 메모리 사용량

유저 공간에서는 bsradm, bsrsetup, bsrcon, bsrmon, bsrservice 프로세스가 사용하는 메모리가 집계됩니다.

커널 모듈에서 사용하는 메모리는 os 별로 다음과 같은 정보가 집계됩니다.

Send buffer performance statistics output
Expand
title사용 예

  • linux 메모리 사용량 실시간 모니터링

Code Block
> bsrmon /watch memstat
2020-12-08_09:57:27.171
  module (bytes)
    BSR_REQ : 16334336
    BSR_AL  : 803760
    BSR_BM  : 4161536
    BSR_EE  : 2782560
  user (kbytes)
    name      pid    rsz        vsz       
    bsrmon    29304  1192       12724
used
(bytes):
bsrmon
0
PEER
37474
bsr03:
 1200
data
stream
  12720
size
(bytes):
10485761
 bsrmon    112177 1192
used
(bytes):
0
  12724
control
stream
 
    bsrmon
size
(bytes): 5242881
113913 1068       12724
used
(bytes):
0 ...

→ The send buffer is allocated to peer 100.100.10.31:7792, bsr03

→ Disconnected from peer bsr03

→ Data transmission to peer 100.100.10.31:7792 is in progress

 
    bsrmon    113978 1308       12728

→ bsrmon 프로세스가 여러개 실행되고 있는 상태. 종료되지 않은 bsrmon 프로세스가 존재

Code Block
> bsrmon /report
sendbuf r0
memstat
Report
r0
 [
SEND BUFFER
MEMORY]
 Run: 2020-12-
05
06_
13
19:
26
04:
59
12.
969
177 - 2020-12-08_09:
56
59:
33
17.
718
716
 module
PEER 100.100.10.31:7792: data stream size=10485761byte, control stream size=5242881byte
(bytes)
  BSR_REQ: 16303104 - 16459264
data-used (bytes): min=2097232, max=7603084, avg=4787174, samples=5
BSR_AL : 803760
cntl-used
(bytes):
min=0,
max=0,
avg=0,
samples=0
PEER
BSR_BM
bsr03
: 4063232
data
-
stream
5271552
size=10485761byte,
control
stream
size=5242881byte
data-used (bytes): min=0, max=0, avg=0, samples=0
BSR_EE : 2782560
cntl-used
(bytes):
min=0, max=0, avg=0, samples=0

→ On average, 4.7 MB of buffer out of 10 MB is used

memstat

Memory usage by users and modules

In user space, the memory used by the bsradm, bsrsetup, bsrcon, bsrmon, and bsrservice processes is aggregated.

As for the memory used by the kernel module, the following information is aggregated by OS.

> bsrmon /watch memstat 2020-12-08_09:57:27.171 module (bytes) BSR_REQ : 16334336 BSR_AL : 803760 BSR_BM : 4161536 BSR_EE : 2782560 user (kbytes) name pid rsz
Expand
titleUsage Example

  • Real-time monitoring of linux memory usage

Code Block
      
 user (kbytes)
  name          rsz                     vsz
  bsradm        0                       0
vsz
   
  bsrsetup
bsrmon
0
29304
1192
12724
bsrmon
0
37474
1200
12720
bsrmon

  bsrmeta
112177
1192
     0
12724
bsrmon
113913
1068
      0
12724
bsrmon
113978
1308
12728
→ Multiple 
bsrmon
processes running. A non-terminating bsrmon process exists
Code Block>
bsrmon
/report
memstat
Report
[MEMORY]
Run: 2020-12-06_19:04:12.177
 1068 -
2020-12-08_09:59:17.716
1368
module
(bytes)
BSR_REQ:
16303104
-
16459264
    12720 - 12732
BSR_AL
:
803760 BSR_BM : 4063232 - 5271552 BSR_EE : 2782560 user (kbytes) name rsz vsz bsradm 0 0 bsrsetup 0 0 bsrmeta 0 0 bsrmon 1068 - 1368 12720 - 12732

→ Output the memory usage range of the data collected during the period 2020-12-06_19:04:12.177 - 2020-12-08_09:59:17.716

→ BSR_REQ uses 16303104 ~ 16459264 bytes

→ Considering that there is no data collected in user space except for bsrmon, it can be seen that the cli commands were executed within 1 second(The measurement period of bsrmon is 1 second.)

→ 2020-12-06_19:04:12.177 - 2020-12-08_09:59:17.716 기간동안 수집된 데이터의 메모리 사용량 범위 출력

→ BSR_REQ 는 16303104 ~ 16459264 bytes 사용됨

→ 유저 공간에서 bsrmon 이외에는 수집된 데이터가 없는 것으로 보아, cli 명령들이 1초 이내에 수행 완료 되었음을 알 수 있다.

resync_ratio {resource} {vnr}

실시간 복제, 동기화 전송량과 이에 대한 동기화 전송 비율

동기화가 진행 중일때 소스 노드에서만 갱신됩니다.

Expand
title사용 예
Code Block
> bsrmon /watch resync_ratio r0 0
2022-04-12_15:34:52.206
svr06
    replcation(144100kb)/resync(18508kb),  resync ratio 11%

→ 연결되어있는 svr06 노드에 초당 복제 데이터가 144100kb, 동기화 데이터가 18508kb 전송되었으며 이에 대한 동기화 전송 비율은 11%입니다.

명령어

bsrmon에서 제공하는 명령어 입니다.

/start

성능 모니터 기능을 활성화 하고 성능 데이터 집계 및 파일 로깅을 시작합니다. 기본적으로 성능 모니터는 활성화 되어 있습니다.

성능모니터의 활성화는 다음의 과정을 통해 수행됩니다.

/stop

성능 모니터를 비활성화 합니다. 비활성화 시 엔진의 성능 데이터 집계 및 파일 로깅이 중단됩니다.

비활성화는 다음의 과정으로 수행합니다.

/status

성능 모니터의 동작 상태를 조회합니다.

/file

성능 데이터와 memory 정보를 파일로 저장합니다. 이것은 숨겨진 명령이며 bsrservice 와 bsrmon-run 스크립트에서 사용합니다.

파일 저장 위치는 다음과 같습니다.

데이터는 조회 시간과 각 항목별 수치만 저장되며, /show, /watch 또는 /report 명령 실행시 데이터 파싱 후 포멧에 맞춰 출력합니다.

Code Block
2021-07-20_05:55:21.443 955 955 244480 244480 1042 1042 266752 266752

/show

전체 리소스에 대한 모든 항목의 성능 데이터를 한번에 출력합니다. 출력 대상은 성능 파일에 마지막으로 기록된 데이터입니다.

특정 성능 항목 및 리소스를 지정하여 출력할 수 있는 옵션을 제공합니다. 성능 항목 및 리소스 명은 ','로 구분하며 공백 없이 입력합니다.

json 형식으로 데이터를 출력합니다.

성능 데이터를 주기적으로 출력합니다. 출력 주기는 bsrmon에 설정된 period 값을 따릅니다.

모든 리소스의 모든 항목 출력

Expand
titlebsrmon /show

  • memory, resource 별로 timestamp 가 출력됩니다.

  • iostat, ioclat, reqstat, peer_reqstat, al_stat, resync_ratio 는 vnr 로 구분하여 출력됩니다.

  • memory 성능 데이터는 os(windows/linux)에 따라 출력되는 항목이 다릅니다.

Code Block
[root@cent79_01 bsr-utils]# bsrmon /show
bsrmon {
    memory {
        system {
            total_memory        7990028; # kbytes
            used_memory 1843756; # kbytes
            free_memory 2511096; # kbytes
            buff/cache  3635176; # kbytes
        }
        module {
            slab {
                bsr_req 15982; # kbytes
                bsr_al  1020; # kbytes
                bsr_bm  7880; # kbytes
                bsr_ee  2384; # kbytes
                total_bio_set   520; # kbytes
                kmalloc 157; # kbytes
            }
            vmalloc     0; # kbytes
            total_page_pool     33344; # kbytes
        }
        user {
            top_process {
                name    gnome-shell;
                pid     2482;
                rsz     250212; # kbytes
                vsz     4082020; # kbytes
            }
            bsr_process {
                name    bsrmon;
                pid     19370;
                rsz     1364; # kbytes
                vsz     12780; # kbytes
            }
        }
        timestamp 2022-10-04_00:10:55.805;
    }
    resource r0 {
        vnr 0 {
            iostat {
                read_iops       0;
                read_iocnt      0;
                read_kbs        0; # kbytes/second
                read_kb 0; # kbytes
                write_iops      0;
                write_iocnt     0;
                write_kbs       0; # kbytes/second
                write_kb        0; # kbytes
            }
            ioclat {
                local_min       0; # usec
                local_max       0; # usec
                local_avg       0; # usec
                master_min      0; # usec
                master_max      0; # usec
                master_avg      0; # usec
            }
            reqstat {
                requests {
                    count       0;
                    before_queue_min    0; # usec
                    before_queue_max    0; # usec
                    before_queue_avg    0; # usec
                    before_al_begin_min 0; # usec
                    before_al_begin_max 0; # usec
                    before_al_begin_avg 0; # usec
                    in_actlog_min       0; # usec
                    in_actlog_max       0; # usec
                    in_actlog_avg       0; # usec
                    submit_min  0; # usec
                    submit_max  0; # usec
                    submit_avg  0; # usec
                    bio_endio_min       0; # usec
                    bio_endio_max       0; # usec
                    bio_endio_avg       0; # usec
                    destroy_min 0; # usec
                    destroy_max 0; # usec
                    destroy_avg 0; # usec
                }
                al_update {
                    count       0;
                    before_bm_write_min 0; # usec
                    before_bm_write_max 0; # usec
                    before_bm_write_avg 0; # usec
                    after_bm_write_min  0; # usec
                    after_bm_write_max  0; # usec
                    after_bm_write_avg  0; # usec
                    after_sync_page_min 0; # usec
                    after_sync_page_max 0; # usec
                    after_sync_page_avg 0; # usec
                }
                peer cent79_03 {
                    pre_send_min        0; # usec
                    pre_send_max        0; # usec
                    pre_send_avg        0; # usec
                    acked_min   0; # usec
                    acked_max   0; # usec
                    acked_avg   0; # usec
                    net_done_min        0; # usec
                    net_done_max        0; # usec
                    net_done_avg        0; # usec
                }
            }
            peer_reqstat {
                peer cent79_03 {
                    count       0;
                    submit_min  0; # usec
                    submit_max  0; # usec
                    submit_avg  0; # usec
                    bio_endio_min       0; # usec
                    bio_endio_max       0; # usec
                    bio_endio_avg       0; # usec
                    destroy_min 0; # usec
                    destroy_max 0; # usec
                    destroy_avg 0; # usec
                }
            }
            al_stat {
                al-extents      6001;
                al_used 0;
                al_used_max     0;
                hits    0;
                hits_total      2064;
                misses  0;
                misses_total    1998;
                starving        0;
                starving_total  0;
                locked  0;
                locked_total    0;
                changed 0;
                changed_total   249;
                al_wait_retry_cnt       0;
                al_wait_total_retry_cnt 0;
                al_wait_max_retry_cnt   0;
                pending_changes 0;
                max_pending_changes     64;
                error {
                    nobufs_starving     0;
                    nobufs_pending_slot 0;
                    nobufs_used_slot    0;
                    busy        0;
                    wouldblock  0;
                }
                flags   __LC_DIRTY,__LC_LOCKED;
            }
            resync_ratio {
                peer cent79_02 {
                    replication	0; # byte/second
                    resync	0; # byte/second
                    resync_ratio	0; # percent
                }
                peer cent79_03 {
                    replication	0; # byte/second
                    resync	0; # byte/second
                    resync_ratio	0; # percent
                }

        }
        vnr 1 {
            iostat {
                read_iops       0;
                read_iocnt      0;
                read_kbs        0; # kbytes/second
                read_kb 0; # kbytes
                write_iops      0;
                write_iocnt     0;
                write_kbs       0; # kbytes/second
                write_kb        0; # kbytes
            }
            ioclat {
                local_min       0; # usec
                local_max       0; # usec
                local_avg       0; # usec
                master_min      0; # usec
                master_max      0; # usec
                master_avg      0; # usec
            }
            reqstat {
                requests {
                    count       0;
                    before_queue_min    0; # usec
                    before_queue_max    0; # usec
                    before_queue_avg    0; # usec
                    before_al_begin_min 0; # usec
                    before_al_begin_max 0; # usec
                    before_al_begin_avg 0; # usec
                    in_actlog_min       0; # usec
                    in_actlog_max       0; # usec
                    in_actlog_avg       0; # usec
                    submit_min  0; # usec
                    submit_max  0; # usec
                    submit_avg  0; # usec
                    bio_endio_min       0; # usec
                    bio_endio_max       0; # usec
                    bio_endio_avg       0; # usec
                    destroy_min 0; # usec
                    destroy_max 0; # usec
                    destroy_avg 0; # usec
                }
                al_update {
                    count       0;
                    before_bm_write_min 0; # usec
                    before_bm_write_max 0; # usec
                    before_bm_write_avg 0; # usec
                    after_bm_write_min  0; # usec
                    after_bm_write_max  0; # usec
                    after_bm_write_avg  0; # usec
                    after_sync_page_min 0; # usec
                    after_sync_page_max 0; # usec
                    after_sync_page_avg 0; # usec
                }
                peer cent79_03 {
                    pre_send_min        0; # usec
                    pre_send_max        0; # usec
                    pre_send_avg        0; # usec
                    acked_min   0; # usec
                    acked_max   0; # usec
                    acked_avg   0; # usec
                    net_done_min        0; # usec
                    net_done_max        0; # usec
                    net_done_avg        0; # usec
                }
            }
            peer_reqstat {
                peer cent79_03 {
                    count       0;
                    submit_min  0; # usec
                    submit_max  0; # usec
                    submit_avg  0; # usec
                    bio_endio_min       0; # usec
                    bio_endio_max       0; # usec
                    bio_endio_avg       0; # usec
                    destroy_min 0; # usec
                    destroy_max 0; # usec
                    destroy_avg 0; # usec
                }
            }
            al_stat {
                al-extents      6001;
                al_used 0;
                al_used_max     0;
                hits    0;
                hits_total      0;
                misses  0;
                misses_total    0;
                starving        0;
                starving_total  0;
                locked  0;
                locked_total    0;
                changed 0;
                changed_total   0;
                al_wait_retry_cnt       0;
                al_wait_total_retry_cnt 0;
                al_wait_max_retry_cnt   0;
                pending_changes 0;
                max_pending_changes     64;
                error {
                    nobufs_starving     0;
                    nobufs_pending_slot 0;
                    nobufs_used_slot    0;
                    busy        0;
                    wouldblock  0;
                }
                flags   NONE;
            }
            resync_ratio {
                peer cent79_02 {
                    replication	0; # byte/second
                    resync	0; # byte/second
                    resync_ratio	0; # percent
                }
                peer cent79_03 {
                    replication	0; # byte/second
                    resync	0; # byte/second
                    resync_ratio	0; # percent
                }

        }
        network {
            peer cent79_02 {
                send    180; # byte/second
                recv    384; # byte/second
            }
            peer cent79_03 {
                send    0; # byte/second
                recv    0; # byte/second
            }
        }
        sendbuf {
            peer cent79_02 {
                ap_in_flight {
                    size        17301504; # bytes
                    count       33;
                }
                rs_in_flight {
                    size        0; # bytes
                    count       0;
                }
                highwater       33;
                fill    17301504; # bytes
                data_stream {
                    size        20971520; # bytes
                    used        0; # bytes
                    packet {
                        name    P_DATA
                        count   1;
                        size    0; # bytes
                    }
                }
                control_stream {
                    size        5242880; # bytes
                    used        0; # bytes
                }
            }
            peer cent79_03 {
                ap_in_flight {
                    size        0; # bytes
                    count       0;
                }
                rs_in_flight {
                    size        0; # bytes
                    count       0;
                }
                highwater       0;
                fill    0; # bytes
                data_stream {
                    size        20971520; # bytes
                    used        0; # bytes
                }
                control_stream {
                    size        5242880; # bytes
                    used        0; # bytes
                }
            }
        }
        timestamp 2022-09-29_22:54:25.064;
    }
    resource r1 {
        ...
        timestamp 2022-10-04_00:10:55.805;
    }
}

모든 리소스의 특정 항목 출력

Expand
titlebsrmon /show /t iostat
Code Block
[root@cent79_01 bsr-utils]# bsrmon /show /t iostat
bsrmon {
    resource r0 {
        vnr 0 {
            iostat {
                read_iops       0;
                read_iocnt      0;
                read_kbs        0; # kbytes/second
                read_kb 0; # kbytes
                write_iops      0;
                write_iocnt     0;
                write_kbs       0; # kbytes/second
                write_kb        0; # kbytes
            }
        }
        vnr 1 {
            iostat {
                read_iops       0;
                read_iocnt      0;
                read_kbs        0; # kbytes/second
                read_kb 0; # kbytes
                write_iops      0;
                write_iocnt     0;
                write_kbs       0; # kbytes/second
                write_kb        0; # kbytes
            }
        }
        timestamp 2022-09-29_22:54:25.064;
    }
    resource r1 {
        vnr 0 {
            iostat {
                read_iops       0;
                read_iocnt      0;
                read_kbs        0; # kbytes/second
                read_kb 0; # kbytes
                write_iops      0;
                write_iocnt     0;
                write_kbs       0; # kbytes/second
                write_kb        0; # kbytes
            }
        }
        timestamp 2022-10-04_00:17:52.249;
    }
}

특정 리소스의 특정 항목 출력

Expand
titlebsrmon /show /t network,sendbuf /r r0,r1
Code Block
[root@cent79_01 bsr-utils]# bsrmon /show /t network,sendbuf /r r0,r1
bsrmon {
    resource r0 {
        network {
            peer cent79_02 {
                send    180; # byte/second
                recv    384; # byte/second
            }
            peer cent79_03 {
                send    0; # byte/second
                recv    0; # byte/second
            }
        }
        sendbuf {
            peer cent79_02 {
                ap_in_flight {
                    size        17301504; # bytes
                    count       33;
                }
                rs_in_flight {
                    size        0; # bytes
                    count       0;
                }
                highwater       33;
                fill    17301504; # bytes
                data_stream {
                    size        20971520; # bytes
                    used        0; # bytes
                    packet {
                        name    P_DATA
                        count   1;
                        size    0; # bytes
                    }
                }
                control_stream {
                    size        5242880; # bytes
                    used        0; # bytes
                }
            }
            peer cent79_03 {
                ap_in_flight {
                    size        0; # bytes
                    count       0;
                }
                rs_in_flight {
                    size        0; # bytes
                    count       0;
                }
                highwater       0;
                fill    0; # bytes
                data_stream {
                    size        20971520; # bytes
                    used        0; # bytes
                }
                control_stream {
                    size        5242880; # bytes
                    used        0; # bytes
                }
            }
        }
        timestamp 2022-09-29_22:54:24.021;
    }
    resource r1 {
        network {
            peer cent79_02 {
                send    0; # byte/second
                recv    0; # byte/second
            }
            peer cent79_03 {
                send    0; # byte/second
                recv    0; # byte/second
            }
        }
        sendbuf {
            peer cent79_02 {
                ap_in_flight {
                    size        0; # bytes
                    count       0;
                }
                rs_in_flight {
                    size        0; # bytes
                    count       0;
                }
                highwater       0;
                fill    0; # bytes
                data_stream {
                    size        0; # bytes
                    used        0; # bytes
                }
                control_stream {
                    size        0; # bytes
                    used        0; # bytes
                }
            }
            peer cent79_03 {
                ap_in_flight {
                    size        0; # bytes
                    count       0;
                }
                rs_in_flight {
                    size        0; # bytes
                    count       0;
                }
                highwater       0;
                fill    0; # bytes
                data_stream {
                    size        0; # bytes
                    used        0; # bytes
                }
                control_stream {
                    size        0; # bytes
                    used        0; # bytes
                }
            }
        }
        timestamp 2022-10-04_00:19:53.026;
    }
}

/watch

types 별로 집계되고 있는 데이터를 실시간으로 출력합니다. /scroll 옵션 사용시 출력을 줄넘김 방식으로 표기합니다.

/report

types 별로 파일에 기록된 데이터(백업된 파일의 데이터 포함)의 통계를 출력합니다.

리포팅 대상 파일 명을 입력합니다. 특정 파일의 통계 산출을 위해 사용됩니다.

리포팅 대상 peer 의 이름을 입력합니다. 특정 peer의 통계 산출을 위해 사용합니다. 다중 peer 입력은 공백 없이 콤마(,)로 구분합니다. 미 입력시 서버에 구성된 리소스의 peer를 대상으로 통계를 출력합니다.

특정 기간의 수치를 조회할 수 있는 옵션입니다. /d 옵션으로 해당 날짜에 기록된 데이터의 통계를 출력합니다. /s 는 조회 시작 날짜 및 시간을 지정합니다. /e 로 조회 종료 날짜 및 시간을 지정합니다. /s, /e 옵션 입력시 날짜와 시간은 언더바(_)로 구분합니다.

Expand
title사용 예

  • bsrmon /report iostat r0 0 /d 2022-11-01

    • 11월 1일에 수집된 iostat 통계 출력

  • bsrmon /report iostat r0 0 /s 2022-11-01 /e 2022-11-10

    • 11월 1일부터 10일까지 수집된 iostat 통계 출력

  • bsrmon /report iostat r0 0 /s 2022-11-01_09:00 /e 2022-11-10_20:00

    • 11월 1일 9시부터 11월 10일 20시까지 수집된 iostat 통계 출력

  • bsrmon /report iostat r0 0 /s 09:00 /e 20:00

    • 수집되 모든 기간동안 9시부터 20시까지의 iostat 통계를 날짜별로 출력

/set

모니터링과 관련된 수치를 조정하는 명령입니다.

파일 저장과 모니터링 주기를 설정합니다. 초 단위로 설정하며 기본 값은 1초 입니다.

파일 롤링 크기를 설정합니다. MB 단위로 설정하며 기본 값은 50MB 입니다.

파일 롤링 수를 설정합니다. 기본 값은 3개 입니다.

/get

모니터링과 관련된 수치를 조회하는 명령입니다.

/io_delay_test

I/O 성능 저하를 의도적으로 발생시켜 bsr 성능 모니터의 기능을 검증합니다. 개발자용 기능 입니다.

/debug cmds options

debugfs 정보를 조회하기 위한 windows 용 명령어입니다.