So I have three identical Oracle systems running Solaris 10u11. They are a test (hostA), production (hostB) and a standby host (hostC) respectively. Recently I configured monitoring on the three systems and found it rather strange that one of the machines had significantly more swap memory available than the other two. I tried to clarify the difference and eventually found the cause.<\/p>\n
First I had a look at swap -s to see if the monitoring check was sane enough:<\/p>\n
\r\nhostA: total: 42284908k bytes allocated + 4044144k reserved = 46329052k used, 43631132k available\r\nhostB: total: 42296644k bytes allocated + 4042428k reserved = 46339072k used, 41737960k available\r\nhostC: total: 42182668k bytes allocated + 4140236k reserved = 46322904k used, 83757568k available\r\n<\/pre>\nFrom this you can see that hostC, the standby host, differs most from hostA and hostB.<\/p>\n
To see if the swap device is configured correctly I had a look to swap -l:<\/p>\n
\r\nhostA:\r\nswapfile dev swaplo blocks free\r\n\/dev\/zvol\/dsk\/rpool\/swap 181,1 8 62914552 62914552\r\n\r\nhostB:\r\nswapfile dev swaplo blocks free\r\n\/dev\/zvol\/dsk\/rpool\/swap 181,1 8 62914552 62914552\r\n\r\nhostC:\r\nswapfile dev swaplo blocks free\r\n\/dev\/zvol\/dsk\/rpool\/swap 181,1 8 62914552 62914552\r\n<\/pre>\nThe above information looks the same to me, so the same amount of swap has been allocated…<\/p>\n
My next thought is that the standby host has less processes running and therefor less memory and swap allocated (wrong thought though but the results are nice) so I decided to get the running processes memory allocation and summarized the results using awk:<\/p>\n
\r\nhostA:\r\nps -e -o vsz,comm|awk '{ printf \"%10d %s\\n\", $1, $2; total += $1}; END {printf \"------------\\n%10d %s\\n\", total\/1024\/1024, \"total (in GB)\"}'\r\n 0 COMMAND\r\n 0 sched\r\n.\r\n.\r\n 16964 \/lib\/svc\/bin\/svc.startd\r\n 2980 -bash\r\n------------\r\n 40 total (in GB)\r\n\r\nhostB:\r\nps -e -o vsz,comm|awk '{ printf \"%10d %s\\n\", $1, $2; total += $1}; END {printf \"------------\\n%10d %s\\n\", total\/1024\/1024, \"total (in GB)\"}'\r\n 0 COMMAND\r\n 0 sched\r\n.\r\n.\r\n 147284 \/usr\/java\/bin\/java\r\n 2972 -bash\r\n------------\r\n 40 total (in GB)\r\n\r\nhostC:\r\nps -e -o vsz,comm|awk '{ printf \"%10d %s\\n\", $1, $2; total += $1}; END {printf \"------------\\n%10d %s\\n\", total\/1024\/1024, \"total (in GB)\"}'\r\n 0 COMMAND\r\n 0 sched\r\n.\r\n.\r\n 10140 \/usr\/sfw\/sbin\/snmpd\r\n 2976 -bash\r\n------------\r\n 40 total (in GB)\r\n<\/pre>\nHmm, no clue here either. Time to google…. and fortunately I found out a hint there, just on a hyperlink stating something about Solaris memory and the ARC cache. Well, I know something about ZFS and caching so have a look there:<\/p>\n
\r\nhostA:\r\nbash-3.2# echo ::memstat|mdb -k\r\nPage Summary Pages MB %Tot\r\n------------ ---------------- ---------------- ----\r\nKernel 1181417 4614 4%\r\nZFS File Data 13050976 50980 39%\r\nAnon 10550269 41211 31%\r\nExec and libs 18994 74 0%\r\nPage cache 67089 262 0%\r\nFree (cachelist) 11043 43 0%\r\nFree (freelist) 8657767 33819 26%\r\n\r\nTotal 33537555 131006\r\nPhysical 32612782 127393\r\n\r\nhostB:\r\nbash-3.2# echo ::memstat|mdb -k\r\nPage Summary Pages MB %Tot\r\n------------ ---------------- ---------------- ----\r\nKernel 1171292 4575 3%\r\nZFS File Data 12592435 49189 38%\r\nAnon 10547964 41202 31%\r\nExec and libs 18886 73 0%\r\nPage cache 30956 120 0%\r\nFree (cachelist) 8649 33 0%\r\nFree (freelist) 9167373 35810 27%\r\n\r\nTotal 33537555 131006\r\nPhysical 32612782 127393\r\n\r\nhostC:\r\nbash-3.2# echo ::memstat|mdb -k\r\nPage Summary Pages MB %Tot\r\n------------ ---------------- ---------------- ----\r\nKernel 883664 3451 3%\r\nZFS File Data 2869233 11207 9%\r\nAnon 10543984 41187 31%\r\nExec and libs 17157 67 0%\r\nPage cache 1403306 5481 4%\r\nFree (cachelist) 2945078 11504 9%\r\nFree (freelist) 14875133 58105 44%\r\n\r\nTotal 33537555 131006\r\nPhysical 32612782 127393\r\n<\/pre>\nBoom there it is… As you can see the ZFS File Data on both hostA and hostB is taking approximately 50GB of memory (39%) while hostC only has 11GB (9%) ZFS File Data cache in use. That’s close enough for me to clarify the difference of 40GB of available swap memory.<\/p>\n
Have also a look at man swap (look for details on -s) and prstat (search for SWAP) for more information on how available swap is calculated.<\/p>\n","protected":false},"excerpt":{"rendered":"
So I have three identical Oracle systems running Solaris 10u11. They are a test (hostA), production (hostB) and a standby host (hostC) respectively. Recently I configured monitoring on the three systems and found it rather strange that one of the machines had significantly more swap memory available than the other two. I tried to clarify … [Read more…]<\/a><\/span><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[14],"tags":[13],"_links":{"self":[{"href":"https:\/\/www.lambiek.eu\/blog\/wp-json\/wp\/v2\/posts\/134"}],"collection":[{"href":"https:\/\/www.lambiek.eu\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.lambiek.eu\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.lambiek.eu\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.lambiek.eu\/blog\/wp-json\/wp\/v2\/comments?post=134"}],"version-history":[{"count":3,"href":"https:\/\/www.lambiek.eu\/blog\/wp-json\/wp\/v2\/posts\/134\/revisions"}],"predecessor-version":[{"id":190,"href":"https:\/\/www.lambiek.eu\/blog\/wp-json\/wp\/v2\/posts\/134\/revisions\/190"}],"wp:attachment":[{"href":"https:\/\/www.lambiek.eu\/blog\/wp-json\/wp\/v2\/media?parent=134"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.lambiek.eu\/blog\/wp-json\/wp\/v2\/categories?post=134"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.lambiek.eu\/blog\/wp-json\/wp\/v2\/tags?post=134"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}