Results 1 to 6 of 6

Thread: DC_READ_FREE and large files

  1. #1
    Senior Member
    Join Date
    Oct 2008
    Posts
    187

    DC_READ_FREE and large files

    I am using 64-bit PV-WAVE 10.00a on Windows 7. When I try to read in a very large 3.8GB .csv file using DC_READ_FREE, I get a strange error message. Is this a bug?

    Code:
    > n_req_col = 11
    > delimit = ','
    > req_cols = [1, 2, 3, 4, 5, 6, 8, 9, 10, 13, 14]
    > data  =  LONARR(n_req_col,  2)
    > status = DC_READ_FREE(infile, data, /COL, DELIM = delimit, $
             GET_COLUMNS = req_cols, RESIZE = [1])
    %%%Informational number 29425 is out of range.
    %%%DC_READ_FREE: No informational message.

  2. #2
    Grand Poobah donb's Avatar
    Join Date
    Aug 2005
    Location
    Colorado
    Posts
    147
    Hi hcrisp,

    DC_READ_FREE is not behaving as one would expect - for the time being I'm stopping short of calling this 'a bug'.

    Please contact support, and if possible we'd like a copy of your 'infile' for testing purposes. The function has clearly encountered an issue of some sort, and is reporting an ambiguous error. At a minimum we'd like to ensure that the error message matches whatever condition has been encounter while reading your file.

    A quick visual inspection of your code says you're trying to get the first two entries out of 11 selected columns, and one of those columns you're trying to read the entire column into memory. If that's NOT what you're trying to do, please reexamine the keywords used.

    Cheers, Don B.
    Don Boughey
    PV-WAVE Advocate

  3. #3
    Senior Member
    Join Date
    Oct 2008
    Posts
    187
    I understand the RESIZE keyword to be the index of variables to be resized, not columns. The docs state, "An array of integers indicating the variables in var_list that can be resized based on the number of records detected in the input data file." I have only one variable, so it should be resized automatically, correct? As for the error, if you try with any old 3.8GB .csv file you should be able to reproduce it if it is due to the filesize. As I can't breakpoint inside DC_READ_FREE, I do not know where the error is coming from. Has DC_READ_FREE been tested on files greater than 2 GB?

  4. #4
    Grand Poobah donb's Avatar
    Join Date
    Aug 2005
    Location
    Colorado
    Posts
    147
    Hi hcrisp,

    Yes, the variable (or variables) will be resized. We have tried your commands on a > 2 GB file and it read successfully. If you are still experiencing an unsuccessful read I suggest you create a case with the PV-WAVE support team.

    Cheers, Don B.
    Don Boughey
    PV-WAVE Advocate

  5. #5
    Banned
    Join Date
    Apr 2015
    Posts
    1
    The function has clearly encountered an issue of some sort, and is reporting an ambiguous error.

  6. #6
    Grand Poobah donb's Avatar
    Join Date
    Aug 2005
    Location
    Colorado
    Posts
    147
    NOOR,

    We completely agree - and despite our best efforts to replicate the ambiguous error, we couldn't. That's why I've asked hcrisp to contact PV-WAVE support. This is not going to be solved by a forum dialog; only by our development team having an example that creates the error message. Only then can we determine the cause and arrive at a fix.
    Don Boughey
    PV-WAVE Advocate

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •