Cultivation template file for scenProc

  • Chris,

    I am wondering if we can direct our cpu allocation to favor ScenPro processing. I see my cpu is using less than 10% of available computer power.

    Regards,

    Ray

    As far as I could observe, CPU usage has quite little to do with processing time in ScenProc. It's usually rather a matter of memory usage.

    Look at the available memory value at the bottom right corner of ScenProc window. As long as the SPC you launched doesn't make this value get too low, processing is pretty fast.

    You don't need this value to drop next to 0 to have troubles : from my experience I'd say as soon as the available memory drops below 1GB or so processing time tends to rise dramatically. I assume it might be the case that ScenProc starts using pagefile system, leading to longer processing time.

    My first advice is to always process separately buildings, vegetations and lights with 3 different sets of SPC files => 1 SPC file dedicated to buildings, 1 for vegetation and 1 for lights.

    It's also good to have separate TOC files for each, IMO. It saves you a lot of time and nerves when you need to check and troubleshoot your cultivation, not having to reprocess everything while you're concentrating on building issues.

    One method might be to create 1 SPC file for everything, then deactivate (by commenting #) everthing that has to do with vegetation and lights and save as buildings SPC, then proceed in a similar way for vegetation, en eventually for lights.

    My second advice is to reduce the source size to ensure fast processing, i.e. low memory usage. If you have to cover a large area, it's better dividing it into smaller area and process each area with a dedicated SPC.

    For instance, while cultivating Corsica, dividing the island into 10 parts and processing them in batches with ScenProc resulted in much faster processing compared to my initial attempts to cultivate the same surface as 1 part.

    Once again, it's the same SPC file, you just copy and adapt the file names for input and output, and create a batch file to launch them all in turn.

    My third advice is to use SplitGrid. I use it to generate 0.1° x 0.1° toc files named according to their corners coordinates. A simple Excel marco of my own generates with a clic all the TSC files calling each toc file.

    Cheers

    Antoine

    Config : i7 6900K - 20MB currently set at 3.20GHz, Cooling Noctua NH-U14S, Motherboard ASUS Rampage V Extreme U3.1, RAM HyperX Savage Black Edition 16GB DDR4 3000 MHz, Graphic Card Gigabyte GeForce GTX 1080 8GB, Power supply Corsair RM Series 850W, Windows 10 64 bit.

    Edited 2 times, last by Trespassers (September 28, 2018 at 11:22 AM).

  • Thanks Chris and Ken,

    I really appreciate all the time and effort you and Ken are doing to promote better customized scenery. It makes a tremendous difference for me.

    It seems a good time for really nice scenery using FSET and adding fsCP airports is a breeze. Adding cultivation just brings the two to life.

    I am working on the Mississippi gulf coast this week and this area has very dense cultivation and plenty of small airports. Lots of lowlands, swamps and waste land. The time consuming part is cleaning up the water coloring.

    The wetlands, shrubs, trees etc. from scenPro adds so much realism to the images, and it is so easy using your templates.

    Thanks again.

    Regards,

    Ray

  • I tried the "storage tanks" code and it failed.

    Did you check in JOSM what attribute you could use for segregation ?

    Unfortunately, OSM data are not always consistent in attributes usage depending on the place and who filled in the data... In some cases I noticed tanks that only have a building = yes attribute, nothing to segregate from a common residential house...

    Cheers

    Antoine

    Config : i7 6900K - 20MB currently set at 3.20GHz, Cooling Noctua NH-U14S, Motherboard ASUS Rampage V Extreme U3.1, RAM HyperX Savage Black Edition 16GB DDR4 3000 MHz, Graphic Card Gigabyte GeForce GTX 1080 8GB, Power supply Corsair RM Series 850W, Windows 10 64 bit.

  • In my screen shot post I should have mentioned that in addition to the nodes used to 'draw' the shape (that are declared as 'building= yes'), there are additional single point node lists that reference the 'center point' node of each of the storage tanks and do indeed declare them as:

    <tag k="man_made" v="storage_tank"/>

    The problem is the definition of 'building' in OSM is too broad and includes storage tanks . (And pretty much anything that was 'built' by man.) There's a lot of discussion but not a lot of agreement on usage. OSM is crowd sourced.

    scenProc parses and runs into the building shape object. The filters used in the script code tests the area ratio of the circular 'building' and generates a square. It's working as expected.
    In this particular instance there really is no way to 'filter' these tanks out (other than an 'exclusion_shape.kml'). Even if scenProc were aware of the tank 'center point' tag lists it would still not be able to relate the referenced virtual center points back to any particular 'building=yes' shape. it would have to calculate the centroid of each building shape and 'guess' it was close enough to to be a tank.

    --Rich

  • I tried the "storage tanks" code and it failed. It actually added even more houses in the "storage tanks" area

    Hi Ray. I hadn't had a look at the area before as I was just responding to the code supplied by Ken. When I had a look at your area unfortunately none of the storage tanks I looked at have the attribute "man_made" = "storage_tank". As a result the code supplied by Ken will not be effective here.

    When looking at your problem I found I have a similar problem for an oil refinery near my local airport. Luckily these tanks do have the "man_made" = "storage_tank" attribute in the OSM source data, so I'll include the fix in my next template release. At least this will remove tanks in some areas.

    All your tanks sit in an industrial area, so you could use that area to exclude any buildings (ie the tanks) in much the same way my template excludes airport buildings automatically. But unfortunately that would also exclude any buildings in the same area as well. It would also prevent buildings appearing in other industrial areas, which is probably not desirable.

    I think your best option is to manually create an exclude for the area with the tanks.

    You are limited to what detail is recorded in OSM and sometimes really odd things are classified as buildings. For instance static historical ships, like the USS Missouri in Hawaii, are classified as buildings in OSM. Luckily they have enough other detail in OSM that allows me to exclude them easily.

    Thanks, Chris

    Win 10 64-bit, 24GB RAM, i5-9400F @ 3.9, 6GB Nvidia RTX-2060

  • Had a quick look at other refineries in the USA and many do record the storage tank attribute in OSM, so hopefully you won't need to use an exclude everywhere.

    I have tested the change in the template and confirm that it does work where this attribute is recorded in the OSM data. I will include it in the next template update.

    I added it to "Filter Buildings" section as "AddAttributeIfInside|building="*"|man_made="storage_tank"|String;skip|yes" rather changing the "CreateAF2Building|building="yes"" lines. This way it works for all building categories rather than just those classified as "yes". This should cater for some inconsistencies in OSM data.

    Thanks, Chris

    Win 10 64-bit, 24GB RAM, i5-9400F @ 3.9, 6GB Nvidia RTX-2060

  • Have been working on a project to add houses where no building footprints exist in OSM data. It is still work in progress but I have managed to add housing using OSM data not related to buildings. The left image shows the cultivation using just OSM data, while the right image shows buildings added with the new housing process. The limitation is that the new houses don't occupy the footprint of the houses in the textures, but you won't find them in the middle of roads, rivers, plant areas or any area with existing OSM recorded buildings. Only tested in Australia so far and not sure of the impact of denser areas.

    Additional housing added without impacting existing building, plants or lights.

    Added housing follows street layout

    Houses everywhere now.

    Thanks, Chris

    Win 10 64-bit, 24GB RAM, i5-9400F @ 3.9, 6GB Nvidia RTX-2060

  • The limitation is that the new houses don't occupy the footprint of the houses in the textures, but you won't find them in the middle of roads, rivers, plant areas or any area with existing OSM recorded buildings.

    Great find! That would save so much time, I could live with that! (Just need more natural rooftop textures to go with that) ^^

  • Crispy136, I am eager to give this a try in my local area that is almost void of OSM data in all the small towns and villages. This has been one my recent wishes. Thanks a million for your creative work.

    Jake, I copied the building texture folder from South Florida and replaced the exisiting texture folder, that may have come from the UK area with a ton of red roofs. This replaced the red with blues, grays, white, etc. roofs.

    Regards,

    Ray

  • Sorry, its not ready yet. Have fine tuned the performance a little further since my last post, but still a few things to resolve. At present it is a separate template you have to run, creating a second toc file to be added to your airport tsc. It runs on the same map.osm file as the OSM sourced building, plant & light data though. Will probably stay as a separate template as I think it is going to be too difficult to integrate it into the existing template. My initial test of a very small area took 40 minutes, so I had to fine tune the template for better performance in order for it to run for a level 10 sized area. While the level 10 sized area in Australia shown in the screen shots above ran fine and completed in about 10 minutes, New Orleans ran for 2 hours before I gave up and cancelled the job. After the fine tuning the template a little more, the Australian test area running time dropped to 5 minutes and New Orleans completed in 10 minutes. CPU usage originally peaked at 15% in earlier tests but now peaks at 70% so scenProc is working harder. I'll also see if I can apply these performance enhancements to the main template.

    In the meantime here are some screen shots of testing done in New Orleans. Once again the left shot is OSM building data only, while the right has the generated buildings from the new template. A few of the original buildings may look like they are missing or different, but this is not the case. FS2 randomly assigns building colours at startup so a building with a white roof may be less visible with gray roof the next time you load FS2.

    Looking towards Louis Armstrong airport where there is little OSM building data.


    And here a little further to the east and integrated into an area that already had a lot of OSM building data. Notice OSM buildings on the right are not compromised, even though they were included in the area the scenProc script was generating.

    Thanks, Chris

    Win 10 64-bit, 24GB RAM, i5-9400F @ 3.9, 6GB Nvidia RTX-2060

  • This looks very promising. You gotta get rid of all those red roofs. You can use the South Florida texture folder a replacement folder to kill the London reds and have gray, blue, white, orange, etc. roofs.

    Regards,

    Ray

  • The OSM for New Orleans metro area is really screwy. Somewhere along a North South line the houses just go to zero out towards the airport. East New Orleans is great, West New Orleans is like a ghost town for OSM.

    So your new trick will be a big boost in the visuals.

    Regards,

    Ray


  • This looks very promising. You gotta get rid of all those red roofs. You can use the South Florida texture folder a replacement folder to kill the London reds and have gray, blue, white, orange, etc. roofs.

    Don't have South Florida, but I did see comments somewhere on which textures to get rid of.

    Thanks, Chris

    Win 10 64-bit, 24GB RAM, i5-9400F @ 3.9, 6GB Nvidia RTX-2060

  • Have released my housing fill template to create houses where there are no housing footprints in OSM. The script is run separately from my other template which will continue to be developed. This means you will need to insert references to the 2 TOC files into your airport TSC file as indicated in the Honolulu (PHNL) TSC below.

    The template uses residential roads to decide where to place the houses. This has some limitations. For instance it is not possible to have it align the houses with the photo imagery of the houses, and you will find some small areas with no houses. Houses will not appear on other road types as these proved difficult to use successfully. The script is also limited to only houses, no larger buildings are generated .

    This template is designed so it won't create overlaps of buildings, or plants generated by my other template and has no impact on street lighting. The template is not 100% perfect so you may find the odd house where it shouldn't be, but houses are filtered out of most areas where they shouldn't be.

    As the template uses residential roads, and these sometimes appear in rural areas in OSM, I found you can often get lines of houses on a single street in the middle of a rural area. To combat this problem, I create an include polygon using Google Earth to define the urban areas of my cities and towns. You can run the script without an include but you will need to comment out a few lines of the template. The lines to comment out are documented in the script itself. The script makes no allowances for exclude polygons, as it is a lot easier to define an urban area to include than it is to define a rural area surrounding an urban area to exclude.

    You can have multiple polygons in your google earth include.kml file. And I also create the polygons for my whole city area, then use the same include for each level 10 area I am generating cultivation for, so there is no need to try and align the size of your include with the size of the OSM data area. When creating your include polygons in Google Earth it is best to set opacity to 40% so you can see the underlying urban areas as you create the polygon. This appears on the "style color" tab on the popup window that appears when creating or editing Google Earth polygons.

    Its a pretty quick process once you get the hang of it. Anyone wanting to know how to create Google Earth Polygons should look at the original "learning cultivation" video. The same method is used to create include polygons required here, as detailed in the video to create exclude polygons. https://www.aerofly.com/dokuwiki/doku.php/sdk:scenerydev4. Larger city areas may not require the use of an include polygon.

    Unlike my original template which is always attached to the first post in this thread, this one is attached to this post. Just rename it as .spc, create your include polygon (or comment out the noted lines) and you're good to go. As with my other template this one also works best when you are using a level 10 sized area from OSM. Run time was 5 minutes for my level 10 sized area in Honolulu and 40 minutes for my level 10 area for New Orleans which is mostly houses. The resulting TOC files are not large but it does require a lot of calculations to create them. My other similar sized test areas took 5 to 10 minutes. CPU usage peaks at 90%+ for some parts of the script but varies from step to step to as low as 15%.

    Example of a Google Earth include polygon. Not separate polygons for the satellite towns to the north and north east of the main city area.

    Example of houses created for Honolulu, before on the left and after on the right.

    Example of what can happen when you don't use an include polygon to define your urban areas.