PDA

View Full Version : DC_Read_Free Errors for very large file



rcarne1
09-01-2009, 04:06 PM
I am increasingly making use of gridded global topography files.

Today I needed to input 80658001 lines of 3 values csv , 2 floating point and an integer.

Attempting S=DC_READ_FREE produces the following

% Error number 33183 is out of range.
% Unknown DC Procedure Name: No error message.

The value of S = 32353

I then tried creating three arrays of the needed great length, lon, lat, depth and read the file in a Do Loop. The readf executed, but, the values were jumbled.

Thinking the file must also be jumbled, I set up a very large string array and read with a simple readf statement. That worked and the file was not jumbled at all.

Then I tried STRSPLIT in a Do Loop to get my 3 values. The loop was not executed...no error message.

So, I know I must be up against some limit for the index of the loops. But I get either no error messages or numbers that are meaningless.

This is all WAVE 9.0 and Vista64

How do you read simple very large files?

Bob C

rwagner
09-02-2009, 09:19 AM
Bob,
Please post code snippets of your DC_READ line as well as the loop with STRSPLIT.
-Ryan

rcarne1
09-02-2009, 10:42 AM
Here is the DC_Read code

RR_file = "C:\WAVE_DATA\ETOPO1.xyz"

i=233280000L

lon = fltarr(i)
lat = fltarr(i)
Z = intarr(i)

s=dc_read_free(rr_file,lon,lat,z,col=1, delim=[","])
print, "Read Complete"

Yesterday it generated the error messages. Today it ran but took about an hour to execute. I avoided the RESIZE option on the assumption that this would be faster. The machine is only 1.6 GHz, but has been adequate for me up to now.

I guess I have a system issue of some sort.