PDA

View Full Version : Memory Allocation Issues



mirosh82
12-21-2006, 07:01 AM
Hello again everyone,

I am having some problems with large files and memory allocation issues. I am trying to read two columns of data from a 1GB file and analyze that data. I have increased my Virtual Memory Paging File size to 4096 MB (the maximum allowed by Windows). However, my program still quits on a "unique" statement after the initial DC_READ_FREE saying "unable to allocate memory: to make array". When I check on my virtual memory paging file size, I still have 2600 MB available. Any ideas?

Here is the snippet of code causing the problem:


stat = DC_READ_FREE(directory+input_files(i), mac, nodeID, /column, Delim=['|', ','], resize=[1,2], Ignore=['$BLANK_LINES'], Get_Columns=[2,7])

unique_mac = 0
corresponding_nodeID = 0

unique_mac = unique(mac)
corresponding_nodeID = strarr(N_ELEMENTS(unique_mac))

The same thing happens when dealing with smaller files as well (on the order of 185MB when using the WHEREIN function. I have verion 8.51b, so the wherein function is not the culprit...I just don't think that I am managing memory properly. Any ideas?
Thanks for speedy replies,
Mike

mirosh82
12-21-2006, 12:50 PM
To add to this problem, I discovered the Windows XP hotfix for SP2 (xpsp2res.dll) that is supposed to fix the way that Windows handles virtual memory page file fragmentation. I implemented the hotfix and ran the program again, and am still receiving the error even though I am 100% confident that I have not run out of memory (including virtual memory).

Here are the important code snippets:


mac=''
nodeID=''
stat=DC_READ_FREE(input_file, mac, nodeID, /column, Delim=['|', ','], $
resize=[1,2], Ignore=['$BLANK_LINES'], Get_columns=[2,7])

unique_mac=0
corresponding_nodeID=0

unique_mac = unique(mac)
corresponding_nodeID = strarr(N_ELEMENTS(unique_mac))

sorted_idx = SORT(mac)
mac = mac(sorted_idx)
nodeID = nodeID(sorted_idx)
sorted_idx=0

FOR j=0,N_ELEMENTS(unique_mac)-1 DO BEGIN


idx=WHEREIN(mac, [unique_mac(j)], /Sorted)
corresponding_nodeID(j)=nodeID(idx(0))
idx=0

ENDFOR


Even using the 185MB file I still get the error


unable to allocate memory: to make array.
Execution halted at WHEREIN <wherein.pro (56)>.

even though I am 100% positive that I have enough available virtual memory. As the WHEREIN functions are looped through, my memory usage is increased, which leads me to believe that WHEREIN has a memory leak somewhere (after all, I am resetting idx each time). When looking at wherein.pro on line 56 I see the line m0=sorted_eq(a,b). I thought this was fixed with 8.51b...but I guess not.

I am at a loss here...everything says that this should work, yet it does not.

mirosh82
12-22-2006, 06:10 AM
I think I have come to the realization that the problem is due to that sorted_eq line in the wherein.pro file. When I use the WHERE function, WHEREIN without the /sorted keyword, and wherein2 (a user created function similar to wherein), my processing is successful.

I think that this needs to be looked at as a very possible memory leak.

Thanks,
Mike

donb
12-22-2006, 09:36 AM
Hi Mike,

Thank you for bringing this memory issue to our attention. As it turns out your timing is perfect. We encountered this issue internally a short time ago and have addressed the issue in sorted_eq. The fixed code however has not been released. We'll post an annoncement when a patch is available on our ftp site.

In the meantime, depending on the size of the problem you're trying to solve, WHEREIN without the /Sorted keyword is the best work-around. If you would like to discuss other alternatives based on your specific problem set, please contact support@vni.com.

Best Regards,

Don B.