Greenplum error canceling query because of high vmem usage

ERROR:  Canceling query because of high VMEM usage. Used: 4613MB, available 507MB, red zone: 4608MB (runaway_cleaner.c:189)  (seg19 slice5 192.168.65.11:6007 pid=55634) (cdbdisp.c:254)

[gpadmin@node6 TPC-DS]$ df -h
Filesystem           Size  Used Avail Use% Mounted on
/dev/mapper/cl-root   50G  7.3G   43G  15% /
devtmpfs             126G     0  126G   0% /dev
tmpfs                126G     0  126G   0% /dev/shm
tmpfs                126G  9.4M  126G   1% /run
tmpfs                126G     0  126G   0% /sys/fs/cgroup
/dev/sdc1           1014M  140M  875M  14% /boot
/dev/sdd             3.7T  307G  3.4T   9% /data/data9
/dev/sdl1            3.7T  307G  3.4T   9% /data/data6
/dev/sdj             3.7T  307G  3.4T   9% /data/data11
/dev/sdg1            3.7T  308G  3.4T   9% /data/data2
/dev/sdk1            3.7T  307G  3.4T   9% /data/data5
/dev/sdf             3.7T  307G  3.4T   9% /data/data10
/dev/sdi1            3.7T  307G  3.4T   9% /data/data4
/dev/sdm             3.7T  306G  3.4T   9% /data/data12
/dev/sdb             3.7T  307G  3.4T   9% /data/data8
/dev/sde1            3.7T  2.1T  1.6T  58% /data/data1
/dev/sda             3.7T  307G  3.4T   9% /data/data7
/dev/sdh1            3.7T  307G  3.4T   9% /data/data3
/dev/mapper/cl-home  392G   29G  364G   8% /home
tmpfs                 26G     0   26G   0% /run/user/0
tmpfs                 26G     0   26G   0% /run/user/1000

node6:
              total        used        free      shared  buff/cache   available
Mem:           251G        2.0G        243G        5.3G        5.5G        243G
Swap:          4.0G          0B        4.0G

node7:
              total        used        free      shared  buff/cache   available
Mem:           251G        2.0G        243G        5.3G        5.5G        243G
Swap:          2.2G          0B        2.2G

node11:
              total        used        free      shared  buff/cache   available
Mem:           125G        1.3G        118G        5.1G        5.3G        118G
Swap:          2.2G          0B        2.2G

REPO="TPC-DS"
REPO_URL="https://github.com/pivotalguru/TPC-DS"
ADMIN_USER="gpadmin"
INSTALL_DIR="/pivotalguru"
EXPLAIN_ANALYZE="false"
RANDOM_DISTRIBUTION="false"
MULTI_USER_COUNT="5"
GEN_DATA_SCALE="1024"
SINGLE_USER_ITERATIONS="1"
RUN_COMPILE_TPCDS="false"
RUN_GEN_DATA="true"
RUN_INIT="true"
RUN_DDL="true"
RUN_LOAD="true"
RUN_SQL="true"
RUN_SINGLE_USER_REPORT="true"
RUN_MULTI_USER="true"
RUN_MULTI_USER_REPORT="true"
RUN_SCORE="true"

20190802:18:49:05:011726 analyzedb:node6:gpadmin-[INFO]:-Done.
/pivotalguru/TPC-DS/05_sql/rollout.sh
source ~/.bashrc

psql -v ON_ERROR_STOP=1 -A -q -t -P pager=off -v EXPLAIN_ANALYZE="" -f /pivotalguru/TPC-DS/05_sql/101.tpcds.01.sql | wc -l
psql -v ON_ERROR_STOP=1 -A -q -t -P pager=off -v EXPLAIN_ANALYZE="" -f /pivotalguru/TPC-DS/05_sql/102.tpcds.02.sql | wc -l
psql:/pivotalguru/TPC-DS/05_sql/102.tpcds.02.sql:59: ERROR:  Canceling query because of high VMEM usage. Used: 4610MB, available 510MB, red zone: 4608MB (runaway_cleaner.c:189)  (seg32 slice2 192.168.65.7:6008 pid=24111) (cdbdisp.c:254)

ERROR:  Canceling query because of high VMEM usage. Used: 4613MB, available 507MB, red zone: 4608MB (runaway_cleaner.c:189)  (seg19 slice5 192.168.65.11:6007 pid=55634) (cdbdisp.c:254)

Elastic Search 2.3.3 heap used too high，jvm old gen percent 100%，not crash

Elasticsearch version:2.3.3
Plugins installed: []
JVM version:1.8_091
OS version:linux-3.10.101kernel
Description of the problem including expected versus actual behavior:
We deploy 12 es nodes in 3 machine. After we insert 1 billion record into es cluster and Meanwhile,we also send query request to es for search. After 3 day passed, es's jvm heap FGC frequently.the node's jvm gcutil looks like that: total 3.8GB for ES_HEAP, 2.6GB for jvm's old gen, and the old gen used is 100%.
We stop all write and read request,but the old gen used could not decrease. The old gen used is 100%.
We clear cache of es,but the old gen used only decrease to 96%.
Steps to reproduce:
use config see below
do insert and query
after data increase to 1 billion, heap used become 100%.
Provide logs (if relevant):
[2017-02-05 11:58:38,221][INFO ][monitor.jvm ] [xxx]
[gc][old][1332853][268813] duration [5s], collections 1/[5.1s],
total [5s]/[17.2h], memory [3.8gb]->[2.8gb]/[3.8gb], all_pools
{[young] [1gb]->[166.8mb]/[1gb]}{[survivor]
[130.7mb]->[0b]/[136.5mb]}{[old] [2.6gb]->[2.6gb]/[2.6gb]} [2017-02-05
12:04:39,450][INFO ][monitor.jvm ] [xxx] [gc][old][1333207][268917]
duration [6.2s], collections 1/[6.5s], total [6.2s]/[17.2h], memory
[3.8gb]->[2.8gb]/[3.8gb], all_pools {[young]
[1gb]->[165.2mb]/[1gb]}{[survivor] [111mb]->[0b]/[136.5mb]}{[old]
[2.6gb]->[2.6gb]/[2.6gb]}
You can look at the reference for more information.

Why Spark application runs much slower with lower MaxGCPauseMillis?

I am testing Spark-1.5.1 with different G1 configurations and observe that my application takes 2 min to complete with MaxGCPauseMillis = 200 (default) and 4 min with MaxGCPauseMillis = 1. The heap usage depicted below. We can see from the statistics below that the GC time of both configs is different by only 5 sec.
I am wondering why execution time increases this much?
Some statistics:
MaxGCPauseMillis = 200 - No. young GCs: 67; GC time of an executor: 9.8 sec
MaxGCPauseMillis = 1 - No. young GCs: 224; GC time of an executor: 14.7 sec
Red area is area is young generation, black is old generation. The application runs on 10 nodes with 1 executor and 6 GB heap each.
The application is a Word Count example:
val lines = sc.textFile(args(0), 1)
val words = lines.flatMap(l => SPACE.split(l))
val ones = words.map(w => (w,1))
val counts = ones.reduceByKey(_ + _)
//val output = counts.collect()
//output.foreach(t => println(t._1 + ": " + t._2))
counts.saveAsTextFile(args(1))

MaxGCPauseMillis is an hint to the JVM that the overall pause times caused by GC should not be more than specified value (in milliseconds). Recommended value is 200 milliseconds for most of the production grade system.
Anything lower may force GC to run more number of times than it is required and would impact the overall throughput of the application, which is exactly happening in your case.
The number of young GCs is 67 while we configure MaxGCPauseMillis=200 and number of Young GC's is almost 4 times (224) when we configure MaxGCPauseMillis=1.
Refer here for more detailed explanations.

Your intuition is wrong. Rather, theoretically, with a heap size chosen, throughput and latency (hinted by MaxGCPauseMillis in this case) have a counter effect. So when you lower MaxGCPauseMillis and hence latency, your throughput goes down too.

My Mongo query is too large and I’m reaching a memory issue

I'm reaching some sort of RAM limit when doing this query, here's the error:
The operation: #<Moped::Protocol::Query
#length=100
#request_id=962
#response_to=0
#op_code=2004
#flags=[]
#full_collection_name="test_db.cases"
#skip=1650
#limit=150
#selector={"$query"=>{}, "$orderby"=>{"created_at"=>1}}
#fields=nil>
failed with error 17144: "Runner error: Overflow sort stage buffered data usage of 33555783 bytes exceeds internal limit of 33554432 bytes"
See https://github.com/mongodb/mongo/blob/master/docs/errors.md
for details about this error.
There are two solutions I can think of:
1) up the buffer limit. this requires mongo 2.8 which is some unstable release that i'd have to install manually.
2) break apart the query? chunk it? this is what the query looks like:
upload_set = Case.all.order_by(:created_at.asc).skip(#set_skipper).limit(150).each_slice(5).to_a
#set_skipper grows by 150 every time the method is called.
Any help?

From http://docs.mongodb.org/manual/reference/limits/
Sorted Documents
MongoDB will only return sorted results on fields without an index if
the combined size of all documents in the sort operation, plus a small
overhead, is less than 32 megabytes.
Did you try using an index on created_at ? That should remove that limitation.

Why does windbg> !EEHeap -gc show a much smaller managed heap than VMMAP.exe?

I have a C# application whose memory usage increases overtime. I've taken periodic user mode dumps and after loading sos, run !EEHeap -gc to monitor the managed heap size. In windbg/sos I've seen it start ~14MB and grow up to 160MB, then shrink back to 15MB, but the applications "Private Bytes" never decreases significantly. I have identified the activity that cauases the increase in "Private Bytes", so I can control when the memory growth occurs.
I tried running Vmmap.exe and noticed it reports a managed heap of ~360MB, took a quick dump and using windbg/sos/eeheap -gc I only see 15MB.
Why am I seeing such different values?
Is the managed heap really what vmmap.exe reports?
How can I examine this area of the managed heap in windbg?

You can't break into a .NET application with WinDbg and then run VMMap at the same time. This will result in a hanging VMMap. You can also not do it in the opposite direction: start VMMap first, then break into WinDbg and then refresh the values in VMMap.
Therefore the values shown by VMMap are probably never equal, because the numbers are from a different point in time. Different points in time could also mean that the garbage collector has run. If the application is not changing so much, the values should be close.
In my tests, the committed part of the managed heap in VMMap is the sum of !eeheap -gc and !eeheap -loader, which sounds reasonable.
Given the output of !eeheap -gc, we get the start of the GC heap at generation 2 (11aa0000) and a size of only 3.6 MB.
Number of GC Heaps: 1
generation 0 starts at 0x0000000011d110f8
generation 1 starts at 0x0000000011cd1130
generation 2 starts at 0x0000000011aa1000
...
GC Heap Size 0x374a00(3623424)
!address gives the details:
...
+ 0`11aa0000 0`11ef2000 0`00452000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE <unknown>
0`11ef2000 0`21aa0000 0`0fbae000 MEM_PRIVATE MEM_RESERVE <unknown>
0`21aa0000 0`21ac2000 0`00022000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE <unknown>
0`21ac2000 0`29aa0000 0`07fde000 MEM_PRIVATE MEM_RESERVE <unknown>
+ 0`29aa0000 0`6ca20000 0`42f80000 MEM_FREE PAGE_NOACCESS Free
...
Although not documented, I believe that a new segment starts at 11aa0000, indicated by the + sign. The GC segment ends at 29aa0000, which is also the starting point of the next segment. Cross check: .NET memory should be reported as <unknown> in the last column - ok.
The total GC size (reserved + committed) is
?29aa0000-11aa0000
Evaluate expression: 402653184 = 00000000`18000000
which is 402 MB or 393.216 kB, which in my case is very close to 395.648 kB reported by VMMap.
If you have more GC heaps, the whole process needs more effort. Therefore I typically take the shortcut, which is ok if you know that you don't have anything else than .NET that calls VirtualAlloc(). Type !address -summary and then look at the first <unknown> entry:
--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free 144 7ff`d8a09000 ( 7.999 Tb) 99.99%
<unknown> 180 0`1a718000 ( 423.094 Mb) 67.17% 0.01%
...

Thank you very much for the detailed answer. Much appreciated.
I'm clear on windbg vs VMmap access/control of the program. Since I can cause the leak by an external action, I'm pretty sure that since I quiesce the activity, memory won't grow much between samples.
I had been relying on the last line of output from !eeheap -gc:
GC Heap Size: Size: 0xed7458 (15561816) bytes.
I think this number must be the amount of managed heap in use (with un-free'ed objects in it). I summed all the "size" bytes reported by "!eeheap -gc" for each SOH and LOH and it matches the above value.
I ran VMmap, took a snap shot and quit VMmap. Then I attached to the process with windbg. Your technique of using !address was most enlightening. I'm using a 12 processor server system, so there are SOH's and LOH's for each processor, i.e 12 to sum. Taking your lead, the output from "!eeheap -gc" has the segments for all of the heaps. I feed them all into "!address " and summed their sizes (plus the size reported by !eeheap -loader ). The result was 335,108K which is within the variation I'd expect to see within the time elapsed (within 600K).
The VMmap Managed Heap seems to be the total amount of all of the memory segments committed for use by the managed heap (I didn't check the Reserved numbers). So now I see why the total reported by "!eeheap -gc" is so much less than what VMmap shows.
Thanks!

Which is faster to process a 1TB file: a single machine or 5 networked machines?

Which is faster to process a 1TB file: a single machine or 5 networked
machines? ("To process" refers to finding the single UTF-16 character
with the most occurrences in that 1TB file). The rate of data
transfer is 1Gbit/sec, the entire 1TB file resides in 1 computer, and
each computer has a quad core CPU.
Below is my attempt at the question using an array of longs (with array size of 2^16) to keep track of the character count. This should fit into memory of a single machine, since 2^16 x 2^3 (size of long) = 2^19 = 0.5MB. Any help (links, comments, suggestions) would be much appreciated. I used the latency times cited by Jeff Dean, and I tried my best to use the best approximations that I knew of. The final answer is:
Single Machine: 5.8 hrs (due to slowness of reading from disk)
5 Networked Machines: 7.64 hrs (due to reading from disk and network)
1) Single Machine
a) Time to Read File from Disk --> 5.8 hrs
-If it takes 20ms to read 1MB seq from disk,
then to read 1TB from disk takes:
20ms/1MB x 1024MB/GB x 1024GB/TB = 20,972 secs
= 350 mins = 5.8 hrs
b) Time needed to fill array w/complete count data
--> 0 sec since it is computed while doing step 1a
-At 0.5 MB, the count array fits into L2 cache.
Since L2 cache takes only 7 ns to access,
the CPU can read & write to the count array
while waiting for the disk read.
Time: 0 sec since it is computed while doing step 1a
c) Iterate thru entire array to find max count --> 0.00625ms
-Since it takes 0.0125ms to read & write 1MB from
L2 cache and array size is 0.5MB, then the time
to iterate through the array is:
0.0125ms/MB x 0.5MB = 0.00625ms
d) Total Time
Total=a+b+c=~5.8 hrs (due to slowness of reading from disk)
2) 5 Networked Machines
a) Time to transfr 1TB over 1Gbit/s --> 6.48 hrs
1TB x 1024GB/TB x 8bits/B x 1s/Gbit
= 8,192s = 137m = 2.3hr
But since the original machine keeps a fifth of the data, it
only needs to send (4/5)ths of data, so the time required is:
2.3 hr x 4/5 = 1.84 hrs
*But to send the data, the data needs to be read, which
is (4/5)(answer 1a) = (4/5)(5.8 hrs) = 4.64 hrs
So total time = 1.84hrs + 4.64 hrs = 6.48 hrs
b) Time to fill array w/count data from original machine --> 1.16 hrs
-The original machine (that had the 1TB file) still needs to
read the remainder of the data in order to fill the array with
count data. So this requires (1/5)(answer 1a)=1.16 hrs.
The CPU time to read & write to the array is negligible, as
shown in 1b.
c) Time to fill other machine's array w/counts --> not counted
-As the file is being transferred, the count array can be
computed. This time is not counted.
d) Time required to receive 4 arrays --> (2^-6)s
-Each count array is 0.5MB
0.5MB x 4 arrays x 8bits/B x 1s/Gbit
= 2^20B/2 x 2^2 x 2^3 bits/B x 1s/2^30bits
= 2^25/2^31s = (2^-6)s
d) Time to merge arrays
--> 0 sec(since it can be merge while receiving)
e) Total time
Total=a+b+c+d+e =~ a+b =~ 6.48 hrs + 1.16 hrs = 7.64 hrs

This is not an answer but just a longer comment. You have miscalculated the size of the frequency array. 1 TiB file contains 550 Gsyms and because nothing is said about their expected freqency, you would need a count array of at least 64-bit integers (that is 8 bytes/element). The total size of this frequency array would be 2^16 * 8 = 2^19 bytes or just 512 KiB and not 4 GiB as you have miscalculated. It would only take ≈4.3 ms to send this data over 1 Gbps link (protocol headers take roughly 3% if you use TCP/IP over Ethernet with an MTU of 1500 bytes /less with jumbo frames but they are not widely supported/). Also this array size perfectly fits in the CPU cache.
You have grossly overestimated the time it would take to process the data and extract the frequency and you have also overlooked the fact that it can overlap disk reads. In fact it is so fast to update the frequency array, which resides in the CPU cache, that the computation time is negligible as most of it will overlap the slow disk reads. But you have underestimated the time it takes to read the data. Even with a multicore CPU you still have only one path to the hard drive and hence you would still need the full 5.8 hrs to read the data in the single machine case.
In fact, this is an exemple kind of data processing that neither benefits from parallel networked processing nor from having more than one CPU core. This is why supercomputers and other fast networked processing systems use distributed parallel file storages that can deliver many GB/s of aggregate read/write speeds.

You only need to send 0.8tb if your source machine is part of the 5.
It may not even make sense sending the data to other machines. Consider this:
In order to for the source machine to send the data it must first hit the disk in order to read the data into main memory before it send the data over the network. If the data is already in main memory and not being processed, you are wasting that opportunity.
So under the assumption that loading to CPU cache is much less expensive than disk to memory or data over network (which is true, unless you are dealing with alien hardware), then you are better off just doing it on the source machine, and the only place splitting up the task makes sense is if the "file" is somehow created/populated in a distributed way to start with.
So you should only count the disk read time of a 1Tb file, with a tiny bit of overhead for L1/L2 cache and CPU ops. The cache access pattern is optimal since it is sequential so you only cache miss once per piece of data.
The primary point here is that disk is the primary bottleneck which overshadows everything else.