Versions Compared

Old Version 9

changes.mady.by.user dgkim

Saved on

compared with

New Version Current

changes.mady.by.user SungEun Kim

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

7.1. Optimization Factor

Optimization of DRX is accomplished by adjusting the DRX internal parameters or by changing the settings of the system so that the buffering and compression functions provided by DRX can achieve optimal performance. The performance of DRX can be buffered up to 6 to 7 Gbps in a 10 Gbps network, and it can be said that the performance is adequate when the compression is 1.5 times more than that. However, these high-performance processing results ​​are based on the assumption that the DRX transmission network bandwidth of the TX side is sufficient, and usually can not show such throughput in the WAN interval. Therefore, in the case of such a high level I/O load state, It is difficult to optimize the performance of DRX in situations where random high level I/O load occurs in such a low transmission band, and it is necessary to check the bandwidth situation of WAN section in advance through proper level simulation for I/O load. DRX provides the drbdsim tool to perform these checks.

In general, the network bandwidth can be checked with tools such as iperf, and iperf calculates the maximum bandwidth the network can make. drbdsim injects I/O load into DRX from local drbdsim and receives remote relayed data from remote drbdsim to calculate the throughput that DRX can process on the network. Therefore, DRX will perform best when iperf's result is equal to drbdsim's or when drbdsim's is greater than it by compression. Please refer to Appendix F. drbdsim usage guide for more information on how to use drbdsim.

Based on the above, the factors of optimization are summarized as follows:

  • Replication I/O load
  • Network Bandwidth
  • DRX Buffering
  • Compression
Info

Latency and throughput optimization on the local I/O side should be handled with optimization on the replication side. For this, please refer to the following.

7.2. Performance Indicator

Performance for the DRX can be viewed through the CLI commands provided by the DRX.

7.2.1 BAB Performance : memusage

This command checks the usage rate and latency of the BAB.

...

  • You can check the usage rate of the BAB set by calculation of "2.4. Physical Memory Specifications".

  • You can see where bottlenecks have occurred by increasing or decreasing the latency value.

Output Sample

...

themeDJango
titlememusage
collapsetrue

...

Table of Contents

Optimization Factor

Optimization of DRX is accomplished by adjusting the DRX internal parameters or by changing the settings of the system so that the buffering and compression functions provided by DRX can achieve optimal performance. The performance of DRX can be buffered up to 6 to 7 Gbps in a 10 Gbps network, and it can be said that the performance is adequate when the compression is 1.5 times more than that. However, these high-performance processing results ​​are based on the assumption that the DRX transmission network bandwidth of the TX side is sufficient, and usually can not show such throughput in the WAN interval. Therefore, in the case of such a high level I/O load state, It is difficult to optimize the performance of DRX in situations where random high level I/O load occurs in such a low transmission band, and it is necessary to check the bandwidth situation of WAN section in advance through proper level simulation for I/O load. DRX provides the drxsim tool to perform these checks.

In general, the network bandwidth can be checked with tools such as iperf, and iperf calculates the maximum bandwidth the network can make. drxsim injects I/O load into DRX from local drxsim and receives remote relayed data from remote drxsim to calculate the throughput that DRX can process on the network. Therefore, DRX will perform best when iperf's result is equal to drxsim's or when drxsim's is greater than it by compression. Please refer to drxsim usage guide for more information on how to use it.

Based on the above, the factors of optimization are summarized as follows:

  • Replication I/O load
  • Network Bandwidth
  • DRX Buffering
  • Compression


Info

Latency and throughput optimization on the local I/O side should be handled with optimization on the replication side. For this, please refer to the following.


Performance Indicator

Performance for the DRX can be viewed through the CLI commands provided by the DRX.

BAB Performance : memusage

This command checks the usage rate and latency of the BAB.

  • Check the performance of BAB through BAB usage and latency.

    • You can check the usage rate of the BAB set by calculation of "2.4. Physical Memory Specifications".

    • You can see where bottlenecks have occurred by increasing or decreasing the latency value.

  • Output Sample

    • Capacity : Indicates the BAB size set in the resource, in Kilobytes.
    • Use (Size) : Indicates BAB usage rate and Size. The unit of size is Kilobytes.
    • BAB Latency is the current (cur), average (avg), maximum (max), and minimum (min) values ​​for the TX, RX, Compress, and Decompress items. The unit is microseconds.


      Code Block
      themeDJango
      titlememusage
      collapsetrue
      < general output >
[root@c73-p1 dgkim]# drx-ctl mem all
--------------------------------------------------------------------------------------
Name                 Capacity Use (Size)                                  Full Counts 
--------------------------------------------------------------------------------------
DR-1-3-c73-p2-c73-p1 716800   [>>>>>>>.....................] 23% (163277) 0           
                     10240    [............................]  0% (0)      -           
-----------------------------------------------------------------------------------
Name        ---
DR-2-3-c73-p2-c73-p1 512000   [............................]  0% (0)      0   Capacity Use (Size)      
                     10240      Full Counts 
-----[............................]  0% (0)      -           
---------------------------------------------------------------------------------
DR-1-3-c73-p2-c73-
 Active bab counts : 2

< verbose output >
[root@c73-p1 716800   [>>>>>>>.....................] 23% (163277) 0           
                     10240    [............................]  0% (0)      -           
-------dgkim]# drx-ctl mem all -v
------------------------------------------------------------------------------------------------------------
DR-2-3-c73-p2-c73-p1 512000   [............................]  0% (0)      0     
Name                 Capacity Begin     End       Use (Size)                            10240    [............................]  0% (0)      -           
  Full Counts 
---------------------------------------------------------------------------------------
 Active bab counts : 2

< verbose output >
[root@c73-p1 dgkim]# drx-ctl mem all -v
----------------
DR-1-3-c73-p2--------------------------------------------------------------------------------------
Namec73-p1 716800   340236152 504137496 [>>>>>>>.....................] 22% (160060) 0           
     Capacity Begin     End       Use (Size)  10240    0         0                   Full Counts[............................]  0% (0)      -           
----------------------------------------------------------------------------------------------------------
DR-12-3-c73-p2-c73-p1 512000 716800  64273922 340236152 64273922 504137496 [>>>>>>>............................] 22% 0% (1600600)      0           
                     10240    0         0         [............................]  0% (0)      -           
----------------------------------------------------------------------------------------------------------
DR-2-3-c73-p2-c73-p1 512000   64273922  64273922  [............................]  0% (0)
 Active bab counts : 2
----------------------------------------------------------------------------------------------------------
name          0       Latency    cur             avg         10240    0max         0    min     [............................]  0% (0)     
-           
----------------------------------------------------------------------------------------------------------
DR-1-3-c73-p2-c73-p1 TX ActiveBAB   bab counts 00:00: 2
----------------------------------------------------------------------------------------------------------
name       00.000000 00:00:00.443516 00:00:06.765142 00:00:00.000101 
                     RX BAB     00:00:00.000000 00:00:00.000000 00:00:00.000000 00:00:00.000000 
        Latency    cur         Compress    avg  00:00:00.000088 00:00:00.000087 00:00:00.001439 00:00:00.000001 
         max             min             
Decompress 00:00:00.000000 00:00:00.000000 00:00:00.000000 00:00:00.000000 
--------------------------------------------------------------------------------------------------------------
DR-12-3-c73-p2-c73-p1 TX BAB     00:00:00.000000 00:00:00.443516001627 00:00:0600.765142004173 00:00:00.000101000261 
                     RX BAB     00:00:00.000000 00:00:00.000000 00:00:00.000000 00:00:00.000000 
                     Compress   00:00:00.000088000000 00:00:00.000087000113 00:00:00.001439001440 00:00:00.000001 
                     Decompress 00:00:00.000000 00:00:00.000000 00:00:00.000000 00:00:00.000000 
----------------------------------------------------------------------------------------------------------
DR-2-3-c73-p2-c73-p1 TX BAB     00:00:00.000000 00:00:00.001627 00:00:00.004173 00:00:00.000261 
                     RX BAB     00:00:00.000000 00:00:00.000000 00:00:00.000000 00:00:00.000000 
                     Compress   00:00:00.000000 00:00:00.000113 00:00:00.001440 00:00:00.000001 
                     Decompress 00:00:00.000000 00:00:00.000000 00:00:00.000000 00:00:00.000000  Active bab counts : 2
[root@c73-p1 dgkim]#


Compression performance statistics.: statistics

This command checks the DRX statistics information.

  • To view compression statistics, use the filter option in Output mode. 
    • The compression ratio represents the average value and the compression efficiency of the data to be transmitted can be checked.
    • Compression plug-in performance can be checked by the average compression time (cp-avg) and the decompression average time (dc-avg).

  • Sample command.

    Code Block
    drx-ctl st all -v --filter=name,type,ratio,cur,best,worst,cp-avg,cp-cur,cp-min,cp-max,dc-avg,dc-cur,dc-min,dc-max


  • Output Sample
    • The unit of size is Kilobytes.

    • The compression ratio is calculated as (original size / compressed size), so it is round down to 2 decimal places for readability.

    • The unit of compress(cp), decompress(de), encypto(ec), decrypto(dc) is microseconds.


      Code Block
      themeDJango
      titlestatistics
      collapsetrue
      < verbose output >
[root@c69-1 bsr.d]# drx-ctl st all -v
----------------------------------------------------------------------------------------------------------
 Active bab counts : 2
[root@c73-p1 dgkim]#

7.2.2 Compression performance statistics.: statistics

This command checks the DRX statistics information.

  • To view compression statistics, use the filter option in Output mode. 
    • The compression ratio represents the average value and the compression efficiency of the data to be transmitted can be checked.
    • Compression plug-in performance can be checked by the average compression time (cp-avg) and the decompression average time (dc-avg).

  • Sample command.

    Code Block
    drx-ctl st all -v --filter=name,type,ratio,cur,best,worst,cp-avg,cp-cur,cp-min,cp-max,dc-avg,dc-cur,dc-min,dc-max
    Output Sample
    • The unit of size is Kilobytes.

    • The compression ratio is calculated as (original size / compressed size), so it is round down to 2 decimal places for readability.

    • The unit of compress(cp), decompress(de), encypto(ec), decrypto(dc) is microseconds.
      Code Block
      themeDJango
      titlestatistics
      collapsetrue
      < verbose output >
[root@c69-1 drbd.d]# drx-ctl st all -v
--------------------------------------------------------------------------------------------
name           type tx  rx t-raw t-net r-raw r-net ratio cur  best  worst cp-avg   cp-cur cp-min   cp-max   ec------------avg ec-cur ec-min ec-max dc-avg dc-cur dc-min dc-max de-avg   de-cur de-min   de-max   
-----------------------------------------------------------------------------------------------------------------------------
name           type tx  rx t-raw t-net r-raw r-net ratio cur  best  worst cp-avg   cp-cur cp-min   cp-max   ec-avg ec-cur ec-min ec-max dc-avg dc-cur dc-min dc-max de-avg   de-cur de-min   de-max   
-------------------------------------------------------------
r0-c69-2---------------------------------------------------------------------------------------------------------------------------------------
r0-c69-2-c69-1 data 184 0  13660 6821c69-1 data 184 0  13660 6821  0     1     1.93  0.00 18.07 1.00  0.000098 0      0.000001 0.013818 0      0      0      0      0      0      0     1 0    1.93  0.00000189 18.07 1.000   0.000098 0      0.000001 0.013818 0      0      0      0      0      0      0      0      0.000189 0      0.000001 0.002338 
r0-c69-2-c69-1 meta 0   0  
--------------------------------------------------------------------------------------------------------------------0.002338 
r0-c69-2-c69-1 meta 0   0  
----------------------------------------------------------------------------------
 All connections : 2, Established : 2
[root@c69-1 drbd.d]#


< use the filter option output>
[root@c69-1 drbd.d]# drx-ctl st all -v --filter=name,type,ratio,cur,best,worst,cp-avg,cp-cur,cp-min,cp-max,dc-avg,dc-cur,dc-min,dc-max
-----------------------------------------------------------------------------------------------------------------------
 All connections : 2, Established : 2
[root@c69-
name           type ratio cur  best  worst cp-avg   cp-cur   cp-min   cp-max   dc-avg dc-cur dc-min dc-max 
-----1 bsr.d]#


< use the filter option output>
[root@c69-1 bsr.d]# drx-ctl st all -v --filter=name,type,ratio,cur,best,worst,cp-avg,cp-cur,cp-min,cp-max,dc-avg,dc-cur,dc-min,dc-max
------------------------------------------------------------------------------------------------------
r0-c69-2-c69-1 data 1.91  1.94 18.07 1.00  0.000088 0.000139 0.000001 0.013818 0-----
name           type 0ratio cur  best  worst 0cp-avg   cp-cur   0cp-min   cp-max   dc-avg r0dc-c69-2-c69-1 meta 
cur dc-min dc-max 
-----------------------------------------------------------------------------------------------------------
 All connections : 2, Established : 2
[root@c69-1 drbd.d]#

      Abbreviations

      AbbreviationDescriptionAbbreviationDescriptionAbbreviationDescriptiontxTX speedratiocompress average ratioecencryptorxRX speedcurcompress current ratio dcdecryptot-netTX network total relayed bytesworstcompress worst ratiot-rawTX network total transferred bytesbestcompress best ratior-netRX network total relayed bytescpcompressr-rawRX network total transferred bytesdedecompress

7.3. Optimization procedure

DRX can optimize by adjusting the buffer size and compression options.

7.3.1 Buffer size

Size of the buffer is set via "memlimit" value in "proxy" section of the DRBD configuration file.

Code Block
proxy {
	memlimit 1G;
}

7.3.2 Compression type and level

DRX provides the following compression plug-ins.

...

Type

...

System

...

Level range

...

Remarks

...

The command to set the compression plug-in is as follows.

...

Code Block
themeDJango
titleset compress
collapsetrue
< Set compression plug-in to IQA >
[root@c73-qat-1 drbd.d]# drx-ctl set compress iqa 1 r0-c73-qat-2-c73-qat-1
r0-c73-qat-2-c73-qat-1: updated.
Result: 1 resource(s) updated.

[root@c73-qat-1 drbd.d]#

...

Code Block
themeDJango
titleset compress
collapsetrue
< Set compression plug-in to ZLIB >
[root@c73-qat-1 drbd.d]# drx-ctl set compress zlib 1 r0-c73-qat-2-c73-qat-1
r0-c73-qat-2-c73-qat-1: updated.
Result: 1 resource(s) updated.

[root@c73-qat-1 drbd.d]#

...

    • --------
r0-c69-2-c69-1 data 1.91  1.94 18.07 1.00  0.000088 0.000139 0.000001 0.013818 0      0      0      0      
r0-c69-2-c69-1 meta 
-----------------------------------------------------------------------------------------------------------
 All connections : 2, Established : 2
[root@c69-1 bsr.d]#


    • Abbreviations

      AbbreviationDescriptionAbbreviationDescriptionAbbreviationDescription
      txTX speedratiocompress average ratioecencrypto
      rxRX speedcurcompress current ratio dcdecrypto
      t-netTX network total relayed bytesworstcompress worst ratio

      t-rawTX network total transferred bytesbestcompress best ratio

      r-netRX network total relayed bytescpcompress

      r-rawRX network total transferred bytesdedecompress


Optimization procedure

DRX can optimize by adjusting the buffer size and compression options.

Buffer size

Size of the buffer is set via "memlimit" value in "proxy" section of the bsr configuration file.

Code Block
proxy {
	memlimit 1G;
}

Compression type and level

DRX provides the following compression plug-ins.


Type
System
Level range
Remarks
ZLIBSW1~9-
LZ4SW1~9-
NONE--OFF


The command to set the compression plug-in is as follows.

TypeCommandsExample output
ZLIB drx-ctl set compress zlib [level] [connect name] 


Code Block
themeDJango
titleset compress
collapsetrue
< Set compression plug-in to LZ4ZLIB >
[root@c73-qat-1 drbdbsr.d]# drx-ctl set compress lz4zlib 1 r0-c73-qat-2-c73-qat-1
r0-c73-qat-2-c73-qat-1: updated.
Result: 1 resource(s) updated.

[root@c73-qat-1 drbdbsr.d]#


NONELZ4drx-ctl set compress none 0 lz4 [level] [connect name] 


Code Block
themeDJango
titleset compress
collapsetrue
< Turn offSet compression plug-insin to LZ4 >
[root@c73-qat-1 drbdbsr.d]# drx-ctl set compress nonelz4 01 r0-c73-qat-2-c73-qat-1
r0-c73-qat-2-c73-qat-1: updated.
Result: 1 resource(s) updated.

[root@c73-qat-1 drbd.d]#

7.3.3 Number of compressing threads

It is possible to change the number of threads processing packets.

The number of threads is set to "process-thread-count" value in the configuration file. If you do not set it or input "-1", it will operate at half the number of CPU cores.

...

/wiki/spaces/QA/pages/925794470

...

-1 bsr.d]#


NONEdrx-ctl set compress none 0 [connect name] 


Code Block
themeDJango
title

...

set compress
collapsetrue
<

...

Turn off compression plug-ins >
[root@c73-

...

1

...

bsr.d]# drx-ctl set

...

compress none 0 r0-c73-2-c73-1
r0-c73-2-c73-1: updated.
Result: 1 resource(s) updated.

[root@c73-1 bsr.d]#

...



Number of compressing threads

It is possible to change the number of threads processing packets.

The number of threads is set to "process-thread-count" value in the configuration file. If you do not set it or input "-1", it will operate at half the number of CPU cores.

  • Commands


    =10
    Code Block
    themeDJango
    titleprocess-thread-count
    collapsetrue
    < Configuration >
[root@c73-qat-1 drbdbsr.d]# drx-ctl

7.4. Others

For DRX Appliance, you can perform optimizations on the local replication I/O reception intervals that work with DRBD. DRX is primarily designed to perform buffering in the application address space, so performance may vary depending on the receiving performance of the OS kernel of the appliance, which is directly linked to DRBD. Therefore, you can tune the receive buffer of this OS kernel socket to further enhance RX receive performance. To ensure that the OS kernel has enough receive buffers, it is necessary to apply the following to the OS kernel.

7.4.1 Linux Kernel Network Parameter

Info

Changing the value using sysctl should be aware that there is a risk that the system will crash.

If you set an unreasonable value to the system, it may stop immediately, so it should be done after enough testing.

...

Code Block
themeDJango
[root@c73-qat-1 drbd.d]# sysctl -a | grep net.core.[rw]mem
net.core.rmem_default = 212992
net.core.rmem_max = 212992
net.core.wmem_default = 212992
net.core.wmem_max = 212992
[root@c73-qat-1 drbd.d]#

...

Code Block
themeDJango
sysctl -w net.core.rmem_max="2097152"
sysctl -w net.core.wmem_max="2097152" 
Info

The set value is not an absolute value and the performance result may be different depending on server and network environment.

7.4.2 Virtual Memory Management

Virtual memory is the total amount of memory used by the process. Linux can adjust the virtual memory size used by the process through the environment variable MALLOC_ARENA_MAX. Windows is excluded because it can only change the total size of virtual memory.

Info

Setting environment variables can affect the entire system because it adjusts the memory pool of the process. 

There should be enough testing to proceed.

    the way to set MALLOC_ARENA_MAX value.
    Code Block
    $> export MALLOC_ARENA_MAX=1
    Info

    The value "1" in the example is not an absolute value and should be set differently depending on the server and environment.

    Considerations
    Changing the value of MALLOC_ARENA_MAX reduces virtual memory usage but may increase CPU utilization
  • set process-thread-count 10
Result: done.

[root@c73-1 bsr.d]# 


< Check > 
[root@c73-1 bsr.d]# drx-ctl show all
...
ip-verification=on
process-thread-count=10

[root@c73-1 bsr.d]#


Others

For DRX Appliance, you can perform optimizations on the local replication I/O reception intervals that work with bsr. DRX is primarily designed to perform buffering in the application address space, so performance may vary depending on the receiving performance of the OS kernel of the appliance, which is directly linked to bsr. Therefore, you can tune the receive buffer of this OS kernel socket to further enhance RX receive performance. To ensure that the OS kernel has enough receive buffers, it is necessary to apply the following to the OS kernel.

Linux Kernel Network Parameter

Info

Changing the value using sysctl should be aware that there is a risk that the system will crash.

If you set an unreasonable value to the system, it may stop immediately, so it should be done after enough testing.


  • Change TCP socket buffer size
    • Server used for testing : Server of /wiki/spaces/PX/pages/901775481
    • References link : https://www.ibm.com/support/knowledgecenter/en/SSZTET_10.1/smpi01/smpi01_tysfbpjp.html
    • Default : 208KBytes


      Code Block
      themeDJango
      [root@c73-1 bsr.d]# sysctl -a | grep net.core.[rw]mem
net.core.rmem_default = 212992
net.core.rmem_max = 212992
net.core.wmem_default = 212992
net.core.wmem_max = 212992
[root@c73-1 bsr.d]#


    • Modify command : Change the max entry to 2 MBytes.


      Code Block
      themeDJango
      sysctl -w net.core.rmem_max="2097152"
sysctl -w net.core.wmem_max="2097152" 


      Info

      The set value is not an absolute value and the performance result may be different depending on server and network environment.