PDA

View Full Version : DC_READ_FREE and large files



hcrisp
03-19-2015, 03:19 PM
I am using 64-bit PV-WAVE 10.00a on Windows 7. When I try to read in a very large 3.8GB .csv file using DC_READ_FREE, I get a strange error message. Is this a bug?



> n_req_col = 11
> delimit = ','
> req_cols = [1, 2, 3, 4, 5, 6, 8, 9, 10, 13, 14]
> data = LONARR(n_req_col, 2)
> status = DC_READ_FREE(infile, data, /COL, DELIM = delimit, $
GET_COLUMNS = req_cols, RESIZE = [1])
%%%Informational number 29425 is out of range.
%%%DC_READ_FREE: No informational message.

donb
03-20-2015, 08:24 AM
Hi hcrisp,

DC_READ_FREE is not behaving as one would expect - for the time being I'm stopping short of calling this 'a bug'.

Please contact support (support@roguewave.com), and if possible we'd like a copy of your 'infile' for testing purposes. The function has clearly encountered an issue of some sort, and is reporting an ambiguous error. At a minimum we'd like to ensure that the error message matches whatever condition has been encounter while reading your file.

A quick visual inspection of your code says you're trying to get the first two entries out of 11 selected columns, and one of those columns you're trying to read the entire column into memory. If that's NOT what you're trying to do, please reexamine the keywords used.

Cheers, Don B.

hcrisp
03-20-2015, 03:09 PM
I understand the RESIZE keyword to be the index of variables to be resized, not columns. The docs state, "An array of integers indicating the variables in var_list that can be resized based on the number of records detected in the input data file." I have only one variable, so it should be resized automatically, correct? As for the error, if you try with any old 3.8GB .csv file you should be able to reproduce it if it is due to the filesize. As I can't breakpoint inside DC_READ_FREE, I do not know where the error is coming from. Has DC_READ_FREE been tested on files greater than 2 GB?

donb
03-31-2015, 02:35 PM
Hi hcrisp,

Yes, the variable (or variables) will be resized. We have tried your commands on a > 2 GB file and it read successfully. If you are still experiencing an unsuccessful read I suggest you create a case with the PV-WAVE support team.

Cheers, Don B.

017hnoor
04-01-2015, 04:07 AM
The function has clearly encountered an issue of some sort, and is reporting an ambiguous error.

donb
04-02-2015, 08:38 AM
NOOR,

We completely agree - and despite our best efforts to replicate the ambiguous error, we couldn't. That's why I've asked hcrisp to contact PV-WAVE support. This is not going to be solved by a forum dialog; only by our development team having an example that creates the error message. Only then can we determine the cause and arrive at a fix.