Geoconvert crashed on 68gb dataset (or How im going to write code to get around this)

  • Just FYI trying to convert 68Gb of tiffs covering roughly 150 miles by 50 miles of the Colorado Front Range (or likely anywhere) will crash Geoconvert... Working on a script to chew through my 450 tiffs one at a time, creating the tmc and batchfile and running each one piece by piece..

    Also built a script to pull down the USGS 1m NAIP imagery state by state based on year... might try to merge scripts so that it downloads/processes for coordinates and converts each image in turn.

    The idea would be to just create the whole united states as i have the bandwidth to pull and convert.. heck if i do it right i dont even need to store the jp2s/tifs

    Edited once, last by Sycosys (July 17, 2017 at 7:38 PM).

  • You may try just level 9 with the 68 GB tiffs to see whether geoconvert gets through this level.

    Regarding the USGS data I have a task for you:

    We need a TFW file for each image, reading the pixel size and the coordinates from the related XML files.

    But this is the current way, you are obviously thinking about a solution for tomorrow.

    Amazing!

    Rodeo

  • Regarding the USGS data I have a task for you:

    We need a TFW file for each image, reading the pixel size and the coordinates from the related XML files

    I am generating world files after I reproject the jp2s from usgs.

    The jp2s are coming projected in a version of web mercator which has coordinates in meters (easting and Northing) from some defined location... im converting everything into WGS84 (lat/long). After that the world files are all in the right format and ready to go for each tif. The next bit i need to work out is either calculating or extracting the coordinate of the lower right hand corner so i can inject that along with the upper left coordinate into the TMC.

    I need to dive into GDAL a bit to work out some of the raster maths that will need doing.


    Thats not a bad idea about trying a single level. I have a feeling i need more Ram though..

    Edited once, last by Sycosys (July 17, 2017 at 9:43 PM).

  • GeoConvert should handle arbitrary amount of input images.

    If GeoConvert crashes, we either have a configuration problem or some other issue with the input images.

    It might help to get an overview of your input images and your config files.

  • GeoConvert should handle arbitrary amount of input images.

    If GeoConvert crashes, we either have a configuration problem or some other issue with the input images.

    It might help to get an overview of your input images and your config files.

    I should be more specific. It hangs. my ram usage spikes and eventually i run out and the whole system becomes unresponsive.

    ill do some more testing

  • Just worked out how to snag corner coordinates for each individual image. Muahaha all opensource tools..