My MFC application has some strange behavior ... as it runs, the "virtual
size" as reported by Process Manager slowly grows to a ridiculous size,
while the "working set" stays relatively constant. When I exit the
application, MFC's built-in memory leak detection says I'm not leaking
memory anywhere. However, I absolutely realize the limitations of the MFC
memory leak stuff.

The application does a lot of creating of threads and communication with an
in-process COM object, so I can see lots of places where memory could be
allocated where MFC wouldn't know about it. I am very careful to let my
threads finish normally (no killing threads off prematurely).

Is there a programmatic way to enumerate the pages that are contributing to
the "virtual size"? Some way I can have my program spit out a list of what
memory blocks are being used, where/when they were allocated (and by which
thread), that sort of information?

The company isn't thrilled about the idea of buying one of the commercial
products for detecting memory leaks, but if that really is the answer I can
push them on it.

Thanks,
Chris

See below...
On Thu, 27 Mar 2008 07:09:38 -0600, "Chris Shearer Cooper" <chris_web@sc3.net> wrote:

Sounds like correct behavior. Also sounds like you have badly fragmented memory. WIthout
knowing the sizes of objects you are allocating, it is hard to guess.

Working set size represents a policy; if you use more pages than your allocated working
set, it will be trimmed to fit into your quota, so it will remain relatively constant.
****
Have your realized the limitations of tools like the Process Manager? It can at best give
approximate answers. What particular limitations of the MFC memory leak stuff are you
referring to?

And has it occurred to you that it might not be MFC that is leaking memory?
****
Do not confuse utilization with memory size. Also, are you closing your thread handles
when the thread finishes? If not, it will hold onto the thread stack, and that will eat
you alive
****
HeapWalk would be a start, Also _heapwalk. Also, watch the handle count for thread
handles.
****
The cost of the commercial products is small compared to your time. But note that you
have not eliminated issues such as thread stacks or other kinds of fragmentation, and have
you considered using the free Application Verifier?
joe
****
email: newcomer@flounder.com
Web: http://www.flounder.com
MVP Tips: http://www.flounder.com/mvp_tips.htm

Thanks for the rapid response!

I believe all of the thread handles are being closed when the threads finish
(I've looked over that code a few times) but since I didn't write that code
I can't be 100% certain. How do I find out how many thread handles are
currently being used?

Yes, I am all too aware of the limitations of Process Manager (grin) but
what I'm dealing with, is a QA department that is getting concerned because
of what Process Manager is showing them. If I can "prove" that the virtual
memory as reported by Process Manager is not actually impacting system
performance, that will be good enough.

What I'm doing right now, is stepping through all the memory "chunks" using
VirtualQueryEx(). It appears that when I call GetProcessHeaps() (so I have
something to pass to HeapWalk()), I get a subset of the information that I
get back from VirtualQueryEx() (right?).

When Process Manager displays its value for "Virtual Memory" usage, is it
displaying the sum of the sizes of all of the MEM_PRIVATE | MEM_COMMIT
regions? Or does it also include MEM_PRIVATE | MEM_RESERVE regions? Or can
I not correlate VirtualQueryEx() results with Process Manager "Virtual
Memory" numbers?

BTW, the regions returned from VirtualQueryEx() show a lot of fragmentation
at that level - I see a lot of 4k regions of MEM_PRIVATE | MEM_COMMIT
alternating with larger regions of MEM_PRIVATE | MEM_RESERVE. I'm thinking
that since I'm seeing fragmentation at the memory region level, the
fragmentation happening inside of any one heap (which is what I would see
with HeapWalk()) is probably not my biggest issue right now.

Thanks,
Chris




"Joseph M. Newcomer" <newcomer@flounder.com> wrote in message
news:n0anu3hegsqglp1tiscf9bt2gnqi9labp7@4ax.com...

See below,....
On Thu, 27 Mar 2008 09:10:24 -0600, "Chris Shearer Cooper" <chris_web@sc3.net> wrote:

It depends. The pages certainly glut up the pagefile.sys file, but if they are not being
used, they don't impact the overall performance. The whole point of the working set is
to minimize the impact on the system. But there's no simple answer.

You need to understand where the storage is going. You will need to instrument it at
various times to see where the segments are coming from.
****
Yes, because not all storage which has been allocated is managed by the C runtime
****
I generally don't trust any of those numbers
****
Reserved address space is not allocated address space. Therefore, it consumes no actual
pages.
****
email: newcomer@flounder.com
Web: http://www.flounder.com
MVP Tips: http://www.flounder.com/mvp_tips.htm

IMHO this pattern, increasing VM usage with statis of WS, is consistent with
leaked memory in general. Think about it and it will become obvious that
most applications that slowly leaks memory will eventually end up with this
condition. The exception being applications that have interspersed leaked
and non-leaked and accessed blocks on the same pages.



This may not necessarily represent a leak in your code. If you have a large
dynamic range in your memory requirements, as is typical with COM, the heap
may be allocating large blocks of memory in anticipation of further
allocations and end up not using them. In this case, you application has
not leaked memory with respect to its interface, the heap, but the OS
reports that lots of memory is in use; hence working set optimizations. If
this is a short-lived process like a GUI or a tool, then this may be just
fine.



Just for our general edification, what it the ridiculous size that VM usage
grows to?



If this is a long-lived process like a service, then I would recommend that
you get a commercial tool and make sure that you aren't leaking memory or
handles. However expensive they seem, they are usually worth it in the long
run.





"Chris Shearer Cooper" <chris_web@sc3.net> wrote in message
news:13uneat1vbs807c@corp.supernews.com...

As indicated below, heap size will not shrink. It can only grow. So you can have tons of
released heap and your memory footprint will not magically shrink. You need to understand
your allocation *patterns* (see my introduction to heaps on my MVP Tips site)
joe

On Thu, 27 Mar 2008 18:20:14 -0400, "m" <m@b.c> wrote:

email: newcomer@flounder.com
Web: http://www.flounder.com
MVP Tips: http://www.flounder.com/mvp_tips.htm

Having written the world's fastest low-fragentation allocator, based on the work of Cuhck
Weinstrock;s PhD dissertation some years before, it look like most of the time you just
needed to keep short lists in the cache, for allocations like1,2,3,4,5,6,7,8,9,10. It was
self-adapting and the longer you used it the faster it got. But it was low-fragmentation,
not no-fragmentation, but could not impose any discipline on our programmsby which storage
compaction could be made possible. Environmnents like Java, C#, VB/CLR eliminate the need
to worry about such structures; they are rerenced counted and allinstances of the
reference are properly maintained,

If anyone had posessed the guts to do the job right back then, we would have had very
reasonable languages without storage problems except total exhaustion of all storage.

It is also mutable at the source level in that if 15 and 22 become interesting sizes, they
can get managed.

But you can't create more memory than a machine holds. By adding control bits to the
allocator, you used between 50% and 2% of the storage as "overhead" bytes.

But without smart allocation, you can easily get pessimal allocation performance without
much effort, as evidenced by the horrific storage allocator that was allowed to exist on
large machines. Worst-case performance, but nobody would go in and just FIX it.

The person who wants to store 60lbs in a 5lb sack needs to understand this.

I was once accused ot a serious storage leak on a VAX. My code was called, and it called
some Unix API to get the current logged-in user name. Each time it did this, the Unix API
went out and got a 64K buffer, filled it with user statistics, and then having extracted
the one field you needed, failed to free the storage. Since it was always 64K, a "more
efficient" technique (what we would call raw VIrtualAlloc) was used, so it left no
footprints of the memory dumper to find (we wrote one! Well, I wrote one). Also, any
module that calls new that does not define debug_new will not get tracked.

Memory ultilization usually takes weeks to identify and fix, On good weeks. I spent over
a month reducing storage usage in a compiler.
joe

On Thu, 27 Mar 2008 18:20:14 -0400, "m" <m@b.c> wrote:

email: newcomer@flounder.com
Web: http://www.flounder.com
MVP Tips: http://www.flounder.com/mvp_tips.htm

My suggestions: buy "Advanced Windows Debugging" and use the tools
presented therein to find COM leaks (for example AppVerifier) and heap
leaks (GFLAGS, debugging kernel, ...)

You will not be able to find heap leaks (GlobalAlloc, LocalAlloc,
HeapAlloc) using MFC memory tracing.

Christian

VM would get up to about 400M and level out for a while, then jump to 600M,
then 800M, one time it got up to 1.7G. At that point all kinds of bad
things start happening (grin).

The problem with all of the commercial tools that I've used, is that they
slow things down so horribly. Since it takes 30-60 minutes for the program
to start showing problems when it's running normally, that would mean hours
and hours before I would get results with the tool running. And yes, that's
worth it if that's the only way to figure out the problem!

However, based on a hint that Joseph gave, I manually went through the code
and looked to see if every thread handle was being properly closed. I was
able to find one that wasn't, and that seems to have solved the problem!

Chris

"m" <m@b.c> wrote in message news:ewwk7iFkIHA.3636@TK2MSFTNGP02.phx.gbl...

I had a similar problem in 1983. I started the job running before I left at night, and
the next morning the failure mode had been identified; it appeared to have run about nine
hours before hitting it. But it required zero effort on my part to deal with it.

Thread handles will hurt you, because the default thread heap is 1MB, so it eats up
massive chunks of memory.
joe

On Fri, 28 Mar 2008 07:54:00 -0600, "Chris Shearer Cooper" <chris_web@sc3.net> wrote:

email: newcomer@flounder.com
Web: http://www.flounder.com
MVP Tips: http://www.flounder.com/mvp_tips.htm

I'm glad you have fixed your problem.

"Chris Shearer Cooper" <chris_web@sc3.net> wrote in message
news:13uq1pe95dcee89@corp.supernews.com...

Sounds like thread stacks. Try to use a attached debugger or minidump to
test that.