Active Channel Instances

[jjpan888]

Hi,

We are using MQ 9.3.0.15 on Red Hat Enterprise Linux 9.5. CPH is running on the target queue manager, and the queue manager VM has 16G memory hosted on AWS.

I am testing if the qmgr can handle concurrent active channel instances up to our setup maxActiveChannels=300. This is our command:

./cph -to -1 -vo 3 -rt 1 -nt 500 -ss 5 -ms 2048 -wi 0 -rl 0 -tx -tc Responder -iq CPH1.REQUEST -oq CPH1.REPLY -cr -dn 10 -jp 1414 -jc CPH1.SVRCONN -jb MQPR -jt mqc -jh MQPR.mq-nonprod.com -jl ECDHE_RSA_AES_256_GCM_SHA384 -jw ibmwebspheremquser1 -kr /tmp/ssl/user

/tmp/cph/cph -to 30 -vo 3 -rt 1 -nt 500 -ss 5 -ms 2048 -wt 100 -wi 0 -rl 0 -tx -co -tc Requester -iq CPH1.REQUEST -oq CPH1.REPLY -dn 10 -jp 1414 -jc CPH1.SVRCONN -jb MQPR -jt mqc -jh MQPR.mq-nonprod.com -jl ECDHE_RSA_AES_256_GCM_SHA384 -jw ibmwebspheremquser1 -kr /tmp/ssl/user

When it is running up to 90 active channel instances, the requester always crashes within 5 mins after it started because of not enough memory allocation errors from /var/log/messages:

Feb 19 11:27:57 MQPR kernel: __vm_enough_memory: pid: 62851, comm: cph, not enough memory for the allocation Feb 19 11:27:57 MQPR kernel: __vm_enough_memory: pid: 62851, comm: cph, not enough memory for the allocation Feb 19 11:28:00 MQPR kernel: __vm_enough_memory: pid: 4780, comm: OAperiodicevent, not enough memory for the allocation Feb 19 11:28:00 MQPR kernel: __vm_enough_memory: pid: 4780, comm: OAperiodicevent, not enough memory for the allocation

How to optimize cph to use memory more efficiently to do stress testing? Is above cph command adjustable by using different flags to test the maximum load without out of memory? Or any sample commands you used to do stress testing? Thanks.

[stmassey]

How much memory do you have available to the client? Can you add free -h and cat /proc/meminfo?

MQ (and cph) is usually pretty minimal in its allocation, but we do require a certain amount per thread and the OS allocates a default stack size. You could multiplex over a larger number of sockets (SHARECNV), but am not sure this will really buy you much.

If you are really intent on reducing the footprint, you could try reducing the thread stack size ulimit -s. Although this allocation is virtual, this may provide you with some headroom in your constrained environment.

[stmassey]

If you have MaxActiveChannels set to 300, then you will receive an error when one of your processes attempts to create the 301st connection to the QM.

[jjpan888]

We can't even achieve 100 active channel instances because of the memory error. :-(

My AWS VM has plenty of memory of 16g. When CPH requester crashes, the memory still got 80% available.

When cph is not running:

[root@MQPR]# free -h
               total        used        free      shared  buff/cache   available
Mem:            15Gi       1.8Gi        11Gi        67Mi       2.1Gi        13Gi
Swap:             0B          0B          0B

[root@MQPR]# cat /proc/meminfo
MemTotal:       15819776 kB
MemFree:        12217468 kB
MemAvailable:   13972668 kB
Buffers:            2920 kB
Cached:          1993756 kB
SwapCached:            0 kB
Active:          2144476 kB
Inactive:         761532 kB
Active(anon):     978436 kB
Inactive(anon):        0 kB
Active(file):    1166040 kB
Inactive(file):   761532 kB
Unevictable:           4 kB
Mlocked:               4 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Zswap:                 0 kB
Zswapped:              0 kB
Dirty:                28 kB
Writeback:             0 kB
AnonPages:        873476 kB
Mapped:           557776 kB
Shmem:             69104 kB
KReclaimable:     159316 kB
Slab:             239560 kB
SReclaimable:     159316 kB
SUnreclaim:        80244 kB
KernelStack:        9808 kB
PageTables:        15288 kB
SecPageTables:         0 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    12655820 kB
Committed_AS:    4177596 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      257888 kB
VmallocChunk:          0 kB
Percpu:             1080 kB
HardwareCorrupted:     0 kB
AnonHugePages:    432128 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
Unaccepted:            0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      156080 kB
DirectMap2M:     6885376 kB
DirectMap1G:     9437184 kB

[root@MQPR]#ulimit -u
65535
[root@MQPR]# ulimit -n
65535
[root@MQPR]# ulimit -s
8192

I've reduced the stack size from 8192 to 2048, and this the free -h
when cph is running:

total        used        free      shared  buff
/cache   available
Mem:            15Gi       2.2Gi        11Gi        98Mi
1.7Gi        12Gi
Swap:             0B          0B          0B

It works for 100 connections. I will try more connections...

[jjpan888]

When I am trying 200 connections, responder failed. Please note: cph is running on the same VM where queue manager is running.

Command line: ./cph -to -1 -vo 1 -rt 1 -nt 1000 -ss 10 -ms 2048 -wt 10 -rl 0 -tc Responder -iq CPH1.REQUEST -oq CPH1.REPLY -cr -dn 10 -jp 1414 -jc CPH1.SVRCONN -jb MQPR -jt mqc -jh MQPR.mq-nonprod.com -jl ECDHE_RSA_AES_256_GCM_SHA384 -jw ibmwebspheremquser1 -kr /tmp/ssl/user
Created Error message to pass to runtime_error()
Call to MQCONNX failed [Completion code: 2; Reason code: 2393]
Sending errorMsg to STDERR
Call to MQCONNX failed [Completion code: 2; Reason code: 2393]
[Responder710] Caught exception: Call to MQCONNX failed [Completion code: 2; Reason code: 2393]

[ControlThread] Caught exception: Responder710: State ERROR set.
terminate called after throwing an instance of 'std::exception'
  what():  std::exception
Aborted (core dumped)

This time is not caused by memory error. How to prevent cph from completely exiting just for one thread got an error?

[jjpan888]

To make things easier. I'm running cph on another 2 VMs. Queue manager is running on its own VM. Below is cph VM spec:

• 16g memory
• 4086 stack size
• 500 threads each for responder and requester

The 1st VM is running fine with 100 active channel connections, but the 2nd VM requester crashes every time with same error. It is very consistent.

This is not memory related, and cph VM has 80% memory available. 2102 means There are insufficient system resources to complete the call successfully.

[Requester837] START
Created Error message to pass to runtime_error()
Call to MQCONNX failed [Completion code: 2; Reason code: 2059]
Sending errorMsg to STDERR
Call to MQCONNX failed [Completion code: 2; Reason code: 2059]
[Requester837] Caught exception: Call to MQCONNX failed [Completion code: 2; Reason code: 2059]

[Requester837] STOP
MQGET Return code caused error or not recognized; mqrc:2009
MQGET Throwing exception; mqcc:2 ;Name: CPH.REPLY7
MQGET Return code caused error or not recognized; mqrc:2009
MQGET Throwing exception; mqcc:2 ;Name: CPH.REPLY7
Created Error message to pass to runtime_error()
Call to MQGET failed [Completion code: 2; Reason code: 2009] CorrelId:414D512041444D515052303520202020FF48B66701686540 QName:CPH.REPLY7
Sending errorMsg to STDERR
Call to MQGET failed [Completion code: 2; Reason code: 2009] CorrelId:414D512041444D515052303520202020FF48B66701686540 QName:CPH.REPLY7
[Requester836] Caught exception: Call to MQGET failed [Completion code: 2; Reason code: 2009] CorrelId:414D512041444D515052303520202020FF48B66701686540 QName:CPH.REPLY7

Created Error message to pass to runtime_error()
Call to MQCLOSE failed [Completion code: 2; Reason code: 2009]
Sending errorMsg to STDERR
Call to MQCLOSE failed [Completion code: 2; Reason code: 2009]
Created Error message to pass to runtime_error()
Call to MQCLOSE failed [Completion code: 2; Reason code: 2009]
Sending errorMsg to STDERR
Call to MQCLOSE failed [Completion code: 2; Reason code: 2009]
Created Error message to pass to runtime_error()
Call to MQDISC failed [Completion code: 2; Reason code: 2009]
Sending errorMsg to STDERR
Call to MQDISC failed [Completion code: 2; Reason code: 2009]
[Requester836] STOP
[ControlThread] Caught exception: Requester837: State ERROR set.
[Requester12] STOP
[Requester89] STOP
[Requester13] STOP

Queue manager error log:

AMQ9508E: Program cannot connect to the queue manager.

EXPLANATION:
The connection attempt to queue manager 'MQPR' failed with reason code
2102.
ACTION:
Ensure that the queue manager is available and operational.
----- cmqxrmsa.c : 6181 -------------------------------------------------------
}

AMQ9999E: Channel 'CPH.SVRCONN' to host '10.86.xxx.xx' ended abnormally.

EXPLANATION:
The channel program running under process ID 16132 for channel
'CPH.SVRCONN' ended abnormally. The host name is '10.86.xxx.xx'; in some
cases the host name cannot be determined and so is shown as '????'.
ACTION:
Look at previous error messages for the channel program in the error logs to
determine the cause of the failure. Note that this message can be excluded
completely or suppressed by tuning the "ExcludeMessage" or "SuppressMessage"
attributes under the "QMErrorLog" stanza in qm.ini. Further information can be
found in the System Administration Guide.

AMQ9772E: MQCTL failed with MQRC=2534.

EXPLANATION:
The indicated IBM MQ API call failed for the specified reason code.
ACTION:
Refer to the Application Programming Reference manual for information about
Reason Code 2534.

What is the recommended way to do stress testing? I'm only sending 1 small msg per sec. Is there a sleep flag we can add?

[stmassey]

How many threads are you using, you say 500, but in the output its Requester837 that is featured, which suggests you are attempting more than 500. With just responders (will not cause any messaging), how many responders can you run in a single process?

Do you really need so many threads? What message rate can your QM support if run with unlimited rate (-rt 0) and a smaller set of requester/responders (50)

[jjpan888]

Good spot. The test report was from when running -nt 1000, but when running -nt 500 in requester, I got the same error.

I have 2 VMs running CPH to avoid not enough memory. On each VM, I run 500 thread responder and 500 thread requester. cph is running fine on 1st VM, but cph requester crashes on 2nd VM.

I did try -rt 0 before, but queue manager's CPU got nearly 100%. Small set of requester/responder has no problem, all running fine.

[jjpan888]

The reason I need that many threads is that in current production(version 8), we have one application using 400 active channel instances, we'd like to test our new version 9 qmgr can handle that or not.

There is "sharing conversation" channel property, our setting is 10, which means 10 thread = 1 active channel connection. In MQ Explorer, right click channel and choose status, you will see the number of active channel connections. To achieve 400 active channel instances, we need 4000 thread based on sharing conversation property.

The quickest way is to have 4 VMs running CPH, and each VM run 500-thread responder and 500-thread requester. Up to now, the 1st VM works, 2nd requester crashes because qmgr doesn't have sufficient resource.

I tried to increase the -wi thread ramping time to 2 sec, get same error.

[stmassey]

-wi refers to the interval between starting each thread. You can set -rt to decimal values, so that -rt 0.01 will only send a message every 100seconds.