Hi, two of our DB server suffered very high CPU load today. we could not identify the exact reason for it. We run a main DB whose table content update frequently. Both of the two questionable servers maintein Snapshot of main DB, they refresh snapshot every 5 min. The OS of two stallite server is HP UX 11i. Normally, the two stalitte servers' CPU is just around 20%. Today, both of them run suffered very high CPU load (around 96% and above), application on stalite DB respond very slow with a lot of service timeout. we have to stop the application which make the two stallite DB CPU come back within 20min. Restarting application does not generate the same problem at all. Looking into sar ball stored, we just find there is high CPU and a lot of semops. But there is NO alert.log record and Archive was executed successfully. What's the possible reason for this ? sar record information: # sar -f sa05 -u | more HP-UX idb-dcb B.11.00 U 9000/800 05/05/08 00:02:00 %usr %sys %wio %idle 19:41:59 46 8 6 40 19:46:59 79 15 1 5 19:51:59 80 15 1 4 19:56:59 80 15 1 4 20:02:00 80 15 1 4 20:07:00 80 16 1 4 20:12:00 80 15 1 4 20:17:00 78 15 1 6 20:22:00 76 15 1 8 20:27:00 44 8 3 44 20:32:00 8 1 3 88 20:37:00 7 1 4 88 20:42:00 8 1 11 79 20:47:00 7 1 6 86 20:52:00 7 1 8 84 # sar -b -f sa05 | more HP-UX idb-dcb B.11.00 U 9000/800 05/05/08 00:02:00 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/ s 19:31:59 0 3141 100 2 2 32 65 45 19:36:59 0 3152 100 1 2 26 65 37 19:41:59 0 5233 100 2 3 32 165 67 19:46:59 0 8475 100 2 3 38 141 15 19:51:59 0 8814 100 2 3 35 131 53 19:56:59 0 8448 100 2 4 36 125 57 20:02:00 0 8413 100 2 4 37 119 59 20:07:00 1 8448 100 2 3 37 117 53 20:12:00 1 8463 100 3 4 32 109 51 20:17:00 1 8404 100 2 3 37 114 52 20:22:00 0 8279 100 2 3 37 115 49 20:27:00 0 5226 100 1 2 26 72 37 20:32:00 0 1157 100 2 2 28 31 30 20:37:00 0 1078 100 1 2 32 43 26 20:42:00 0 1026 100 2 4 30 276 32 # sar -c -f sa05 | more HP-UX idb-dcb B.11.00 U 9000/800 05/05/08 00:02:00 scall/s sread/s swrit/s fork/s exec/s rchar/s wchar/s 19:26:59 16598 7558 4519 0.38 0.36 11036224 7397 19:31:59 16984 7616 4553 0.50 0.48 11296768 11547 19:36:59 16616 7569 4522 0.38 0.36 11412110 8113 19:41:59 31015 12931 7664 0.54 0.51 14109634 13459 19:46:59 53605 20837 12369 0.59 0.55 12035902 13353 19:51:59 54295 21122 12326 0.53 0.50 498452 12947 19:56:59 53569 20748 12314 0.67 0.64 11791916 15763 20:02:00 53674 20657 12267 0.63 0.61 11718527 14961 20:07:00 53529 20728 12309 0.61 0.58 11805223 13598 20:12:00 53757 20775 12346 0.60 0.57 11921990 17148 20:17:00 52144 20625 12246 0.60 0.56 11452784 14296 20:22:00 50627 20276 12043 0.56 0.53 10370528 13101 20:27:00 29580 12678 7521 0.42 0.40 14077691 9100 20:32:00 6054 2516 1462 0.48 0.47 9361588 11318 20:37:00 5173 2211 1271 0.40 0.38 8727180 8103 20:42:00 5876 2413 1253 0.59 0.57 8267531 17158 20:47:00 5626 2203 1210 0.65 0.63 8142735 15047 # sar -w -f sa05 | more HP-UX idb-dcb B.11.00 U 9000/800 05/05/08 00:02:00 swpin/s bswin/s swpot/s bswot/s pswch/s 19:36:59 0.00 0.0 0.00 0.0 7057 19:41:59 0.00 0.0 0.00 0.0 16138 19:46:59 0.00 0.0 0.00 0.0 31574 19:51:59 0.00 0.0 0.00 0.0 31989 19:56:59 0.00 0.0 0.00 0.0 31973 20:02:00 0.00 0.0 0.00 0.0 31700 20:07:00 0.00 0.0 0.00 0.0 31929 20:12:00 0.00 0.0 0.00 0.0 31971 20:17:00 0.00 0.0 0.00 0.0 31420 20:22:00 0.00 0.0 0.00 0.0 30354 20:27:00 0.00 0.0 0.00 0.0 16026 20:32:00 0.00 0.0 0.00 0.0 2100 20:37:00 0.00 0.0 0.00 0.0 1831 20:42:00 0.00 0.0 0.00 0.0 2106 20:47:00 0.00 0.0 0.00 0.0 1824 20:52:00 0.00 0.0 0.00 0.0 1881 # sar -f sa05 -d | more HP-UX idb-dcb B.11.00 U 9000/800 05/05/08 00:02:00 device %busy avque r+w/s blks/s avwait avserv 20:02:00 c1t6d0 2.78 0.55 4 37 4.26 12.33 c2t6d0 1.88 0.54 3 32 4.55 10.07 c40t0d1 13.61 0.80 32 278 5.10 4.74 c40t0d2 13.56 2.23 41 348 5.32 4.06 c40t0d3 13.02 1.76 38 328 5.23 4.02 c40t0d4 13.25 1.04 35 302 5.16 4.44 c40t0d5 13.78 0.87 33 289 5.13 4.90 20:07:00 c1t6d0 2.49 0.81 4 43 5.75 12.55 c2t6d0 1.74 0.83 3 39 6.41 10.87 c40t0d1 13.10 1.07 30 262 5.07 4.82 c40t0d2 12.79 2.44 37 321 5.38 4.23 c40t0d3 13.04 2.05 37 316 5.33 4.25 c40t0d4 13.12 1.50 33 290 5.23 4.63 c40t0d5 13.57 1.03 33 287 5.08 4.87 20:12:00 c1t6d0 3.50 0.62 5 72 4.76 11.16 c2t6d0 2.08 0.75 3 39 5.74 10.99 c40t0d1 12.56 1.52 29 257 5.19 4.82 c40t0d2 12.33 2.89 36 310 5.48 4.15 c40t0d3 11.86 2.76 34 294 5.44 4.21 c40t0d4 12.30 2.08 31 269 5.38 4.58 c40t0d5 12.84 1.64 31 270 5.24 5.08 regards Joe
