Creating terrain heightmaps

    Quote from Spit40

    Love it! The UK has free elevation data too, but I don't know how easily contours are converted to heightmaps. I bet arnog could do it !

    https://www.ordnancesurvey.co.uk/opendatadownlo…cts.html#terr50

    Hi Phil

    The Ordnance Survey 50 m data that you mention ("OS Terrain 50") can be downloaded in ASCII (.asc) format, so it can be processed in pretty much the same way as I've processed the DEFRA 1m data in my tutorial above. The 1 m data has quite a lot of holes in it, so I was thinking of using the 50 m data as a base layer. I'm also currently investigating the Ordnance Survey 5 m data ("OS Terrain 5"). You have to fill in a form saying how you intend to use it etc - so I'm hoping they give me the go-ahead (without paying anything preferably!) If I can get hold of that it would probably be a better option than the 1 m data because, being from Ordnance Survey, I imagine it is complete and of good quality. The 1 metre data could then perhaps be used as a level 14 top layer in the areas where the scenery would benefit from it.

    Anyway, I tried the 50 m data out around Beachy Head and took 3 screen shots for comparison. The top pic is with the default mesh, the middle pic is with the OS 50 m mesh, and the bottom pic is with the DEFRA 1 m mesh. The 50 m mesh is actually a considerable improvement on the default mesh so I am hoping that, where there are holes in the 1 m data, the mesh boundary / fault line won't be so obvious with a 50 m base layer. That's the theory anyway!

    Edited 5 times, last by Ian C (February 8, 2019 at 11:35 PM).

    This is just a bit of extra detail to go with the above tutorial on heightmaps for Great Britain.

    Currently I'm working on the SW peninsula of England (roughly Cornwall) and I've attached a couple of Geotiffs to give you an idea of what to expect if you want to have a go yourself. The first one comes from the Ordnance Survey 50 metre data (OS Terrain 50) and is complete with no missing data, as you can see. I use this for level 11. The second was compiled from DEFRA 1 metre data. In the latter there are an annoying number of black patches - i.e. areas with no data. If geoconverted, these would produce a flat landscape at sea level - so they need to be avoided. Fortunately the coast is complete. I use this data, where available, for level 14 tiles.

    To get the 50 metre heightmap a total of 33 .asc files were merged. The size of the Geotiff is about 10 MB.

    The 1 metre heightmap is the result of merging roughly 3000 .asc files, and it had to be done in two stages in batches of about 100. Its size is a stonking 18.5 GB. Although it is interesting here for illustrative purposes, it is actually useless from the point of view of geoconversion. Geoconvert doesn't seem to be able to cope with a single TIF of this size. It hangs for ever on 0 tiles - "for ever" being about 10 mins before I gave up! About a quarter of this area is do-able I would say. Then you have the "interesting" task of picking out and deleting all the tiles that coincide with black areas. This can be done by getting the lon-lat co-ordinates of the black patches from QGIS and mapping them onto a grid showing the Aerofly tiles - but it's quite time-consuming. (If anybody can think of a better way I'd be very interested*.) Level 14 tiles are only about 1.5 km square so there's a good chance of them fitting in between the black areas. Very few level 12 & 13 tiles are usable, so I've pretty much given up on them.

    *Actually, it's just occurred to me that I could somehow or other superimpose a semi-transparent image of the geotiff on an image of the Aeroscenery map with all the level 14 tiles selected. I'll try that tomorrow using GIMP.

    [Frustratingly, the Ordnance Survey also has complete 5 metre data, but they won't let you try it out for free unless you intend eventually to use it commercially (and pay a hefty fee). My application to use it for a hobby was rejected.]

    Edited 10 times, last by Ian C (February 21, 2019 at 1:19 AM).

    Just a heads up that Orbx is doing the entire UK for Aerofly as we speak........ https://orbxsystems.com/forum/topic/16…for-aerofly-fs/

    Devons rig

    Intel Core i5-13600K - Core i5 13th Gen 14-Core (6P+8E) @ 5.5Ghz / G.SKILL Trident Z5 RGB Series 32GB RAM DDR5 6000 / GIGABYTE GeForce RTX 4070Ti GAMING OC 12G / Sound Blaster Z / Oculus Quest 2 VR Headset / Klipsch® Promedia 2.1 Computer Speakers / ASUS ROG SWIFT PG279Q ‑ 27" IPS LED Monitor ‑ QHD / 6x Samsung SSD/NVME's various sizes / Windows 11 Pro 64-bit / GIGABYTE Z790 AORUS ELITE AX LGA 1700 ATX Motherboard DDR5

    Quote from HiFlyer

    Just a heads up that Orbx is doing the entire UK for Aerofly as we speak........ https://orbxsystems.com/forum/topic/16…for-aerofly-fs/

    Interesting ..... and thanks

    I know that I would still probably want to use my own ortho images because I always geoconvert to level 15 - and I doubt that ORBX will do that because at level 15 we're talking about more than 1 TB of data for GB.

    However, I would definitely still buy the product in order to use the cultivation - so maybe I won't invest too much effort in that for the time being except for using improved textures on the default buildings, which is fairly easy.

    The terrain mesh is then the big question. Will they use the Ordnance Survey 5 m mesh? The price for me is prohibitive, but I imagine it wouldn't be a problem for ORBX. That's the real dilemma for me, because I only have access to 50 metre and 1 metre data - and the latter is very time-consuming to use.

    Conclusion: (1) I'll continue to make high res level 15 scenery. (2) I'll go easy on 3D models for the time being, though I'll continue with the one(s) I have on the drawing board. (3) I'll probably also continue with my work on the 1 metre mesh because I'm impatient to see what the scenery looks like with a better mesh.

    Edited 6 times, last by Ian C (February 25, 2019 at 12:31 AM).

    Hey Ian, great work!!! I never tested this myself, but I wonder if you can use those empty voids to create masks for the terrain mesh. I know that Geoconvert generates terrain masks, and assuming they work like masks for orthophotos, you might be able to do this in a way that just shows the lower-level mesh in places where your higher-res stuff is missing.

    The trick is, I don't know what geoconvert looks for to create masks with the terrain data. It's not full black, or full white -- does geotiff support transparency? Maybe if you leave those empty spots transparent instead of black, it'll mask them for you...?

    This of course creates the issue of partial terrain tiles and masks, but that might be ok if your original dataset isn't going to ever fill in those voids.

    Quote from qwerty42

    Hey Ian, great work!!! I never tested this myself, but I wonder if you can use those empty voids to create masks for the terrain mesh. I know that Geoconvert generates terrain masks, and assuming they work like masks for orthophotos, you might be able to do this in a way that just shows the lower-level mesh in places where your higher-res stuff is missing.

    The trick is, I don't know what geoconvert looks for to create masks with the terrain data. It's not full black, or full white -- does geotiff support transparency? Maybe if you leave those empty spots transparent instead of black, it'll mask them for you...?

    This of course creates the issue of partial terrain tiles and masks, but that might be ok if your original dataset isn't going to ever fill in those voids.

    Hi qwerty - and thanks!

    I've been puzzling about this for ages and asking myself the same questions. As far as I can see, the problem with the "holes" in the 1 metre data is that there is actually data in the "holes" - but it's the wrong data. i.e. The holes are basically flat areas at sea level. It might be useful if the holes could be converted into missing data in the true sense of the term. I don't know whether there is a way of doing that. (I may be mistaken but I think there is some way in QGIS of setting the black areas to a certain height - but that doesn't really solve our problem.) Geoconvert doesn't currently produce any masked tiles where the holes are. I assume that this is because it doesn't see them as missing data.

    But then there is the question of what effect it would have if there were masked TTH files in those places. Masked TTC files seem to have the effect of hiding whatever tiles are below them in level - i.e. masked level 15 tiles hide level 14 tiles and below. (To be more precise they hide what's below where the mask is - i.e. where there is no ortho image.) This produces black patches in the scenery. By analogy you would think that masked TTH files would have the same effect - i.e. they would produce patches where you couldn't see the lower level TTH tiles. This would presumably produce flat areas at sea level as well - so there wouldn't be any advantage.

    Another thing that puzzles me is the fact that the masked TTC files come in pairs - i.e. a tile + a separate mask - whereas there is only one masked TTH file. This makes me wonder whether the TTC and TTH masks really behave in the same way.

    I find masks very confusing - and I'm not sure that I fully understand them!

    Edited once, last by Ian C (February 22, 2019 at 11:29 AM).

    Hmmm... AFAIK, the range from full black to full white in the geotiff directly maps to elevation. But it's not trivial to edit them, because they are 16 bit grayscale images rather than the 8 bit that most people are familiar with, and even using something like Photoshop they aren't easy to work with while preserving the 16 bit data.

    My idea, which might be completely wrong, is that if geotiff supports transparency or something like it, and you already know where the 'holes' in the data are, the fix might be to remove those areas entirely from the geotiff. I haven't tested anything like this at all though. I just know that geoconvert is creating masks for terrain, and I think it's doing it correctly--it makes them where you don't have full data coverage within a complete tile.

    Regarding masks, I think you've got it backwards, actually. For orthophotos, if you have a masked level 15 tile, then the next lowest-level that exists will show through. In other words, if you have a full level 14 tile and a partially masked level 15 above it, the level 14 tile will show through the masked areas. You only get black regions if you have incomplete tiles and *no* mask for the incomplete area.

    The masks only create issues when you try to create adjoining scenery at the same level, because the mask blocks out the area it covers until it's removed, even if you replace the partial-coverage scenery tiles with full coverage.

    So going back to the terrain tiles, assuming the masks are working correctly, then for areas that were masked in your incomplete level 15 tiles, you'd just have level 14 terrain data showing through in those places. The trick to solve, is how do we modify these geotiffs so instead of the missing areas showing up as 0 elevation, they show up as something that geoconvert recognizes it needs to mask...

    Edited once, last by qwerty42 (February 22, 2019 at 3:31 PM).

    Quote from qwerty42

    The masks only create issues when you try to create adjoining scenery at the same level.

    Interesting what you say about masks - and I think I've partially (!) got my head around it. However some bits are still puzzling me:

    "The masks only create issues when you try to create adjoining scenery at the same level, because the mask blocks out the area it covers until it's removed, even if you replace the partial-coverage scenery tiles with full coverage."

    Suppose you geoconverted 2 adjacent areas and the boundary bisected level 15 tiles .....

    You say: "You only get black regions if you have incomplete tiles and *no* mask for the incomplete area."

    What confuses me is that I seem to remember getting black areas in my early attempts at scenery creation - when I had loads of masks for the incomplete tiles at the boundaries of my area. Or maybe I'm mis-remembering that. As you say, if you can see through masked areas then masked tiles shouldn't create any black areas (for TTC) or sea-level areas (for TTH). The worst you should get is blurry scenery at that point - or a low res mesh in the case of masked TTH files.

    The fact that the TTH files don't have separate masks confuses me too.

    Edited 11 times, last by Ian C (February 23, 2019 at 12:57 AM).

    Quote from Ian C

    Suppose you geoconverted 2 adjacent areas and the boundary bisected level 15 tiles .....

    If your source imagery has data all the way up to the boundary, then it should work just fine. Geoconvert doesn't care if the source images are split across multiple source files, just so long as all of it is there.

    But, as an example--if you have two source images that don't quite meet (maybe for some reason there is a 100m gap between the photo data), then a mask file will be created, if masks are enabled. If they aren't enabled, it will skip generating that tile since it doesn't have complete imagery to produce it.

    Likewise, if your source imagery ends halfway across a tile, and masks are enabled, geoconvert will create a mask file to let the lower levels show through where your source imagery ends, and you'll still end up with half a tile of orthophoto imagery which displays correctly in the sim.

    The orthophoto masks are actually very simple: they are just an extra hidden image layer that tells a converted orthophoto tile where it needs to be transparent. If you're familiar with using alpha channels for transparency in computer graphics, they work a lot like that.

    They become confusing because there is so much else going on that it's hard to keep track of all of it (source imagery resolution aka zoom level, geoconvert levels, and the fact that it's not immediately obvious which level you're currently viewing when flying in the sim at various altitudes). And if you've built a bunch of scenery that generated masks, those masks will force any newly-generated scenery of the same level to still be transparent in the same places unless the masks are removed. That can become a very tedious process, so that's why some of us started suggesting that everyone avoids creating mask files altogether.

    If you already have scenery that was converted with masks enabled and those masks were produced, it means you have some tiles that only had partial image coverage. If those masks then get deleted without removing their corresponding image tiles as well, that's when you'll start seeing the black areas, because there is no data there and the lower levels can't show through it without the mask.

    OK - thanks qwerty - that's helped a lot.

    I think the problem was that I decided early on to banish masks from my photoscenery so I stopped trying to understand them after that! The other thing that confused me was that I imagined that maybe something strange was happening to them during the geoconversion process so that they no longer behaved like normal images.

    As for the TTH files I assume that the mask and image are inseparable for some reason?

    UPDATE: I've started using raster > extraction > clipper in QGIS to crop rectangles from my Geotiff, thereby avoiding the holes in the data. Each cropped Geotiff needs its own separate TFW file so it's quite time-consuming - but it does work. It also enables me to geoconvert all levels 12-14. Previously I was only geoconverting level 14, which was giving poor results in some areas - even though I had a base layer of level 11 derived from 50 metre data.

    Edited 4 times, last by Ian C (February 25, 2019 at 6:50 PM).

    Quote from Ian C

    UPDATE: I've started using raster > extraction > clipper in QGIS to crop rectangles from my Geotiff, thereby avoiding the holes in the data. Each cropped Geotiff needs its own separate TFW file so it's quite time-consuming - but it does work. It also enables me to geoconvert all levels 12-14. Previously I was only geoconverting level 14, which was giving poor results in some areas - even though I had a base layer of level 11 derived from 50 metre data.

    That sounds like progress! Nice job! Regarding the mask files for terrain, I just had a closer look at my stuff here and didn't realize before that the terrain height masks don't have a corresponding non-mask file. I was confused by your question about that before because I thought they came in pairs like the orthophoto tiles. So... I have no idea how or if the elevation masks actually do work :D Based on the limited testing I did before when I converted a whole bunch of elevation, I remember concluding somehow that it did look like they were allowing partial terrain tiles to be built, but I could have been wrong about that. One of these days when I have some spare time I'll dig into it some more, unless you beat me to it :)

    Quote from qwerty42

    That sounds like progress! Nice job! Regarding the mask files for terrain, I just had a closer look at my stuff here and didn't realize before that the terrain height masks don't have a corresponding non-mask file. I was confused by your question about that before because I thought they came in pairs like the orthophoto tiles. So... I have no idea how or if the elevation masks actually do work :D Based on the limited testing I did before when I converted a whole bunch of elevation, I remember concluding somehow that it did look like they were allowing partial terrain tiles to be built, but I could have been wrong about that. One of these days when I have some spare time I'll dig into it some more, unless you beat me to it :)

    With my new method I'm producing loads of masked TTH tiles. They don't seem to be producing any problems at the moment - but I need to fly around a bit more just to check. Also most of the cropped Geotiffs are overlapping, which geoconvert seems to take in its stride. I didn't realise you could do that before!

    If only there was some way to automatically generate the TFW files (like the INF to TFW converter) - that's the weak point in the process because it's so time-consuming if you have lots of cropped bits of Geotiff. AeroScenery just uses the AID files as far as I'm aware, so how does it create them? That's something I need to experiment with. If you could merge the cropped bits without creating large no-data areas that would also help - but I don't think (?) you can do that. In any case I'm not sure how the TFW file would work with an odd-shaped (non-rectangular) Geotiff.

    The other thing I've noticed is that it's possible to make the black (no data) areas transparent in the QGIS canvas using layer > properties > transparency - but unfortunately it doesn't seem to be possible to export the Geotiff and retain the transparency. As you say, that would be the best way of solving the problem if it were possible.

    UPDATE: I've managed to get round the problem of creating multiple TFW files for all the cropped bits of geotiff by using "raster > conversion > translate" in QGIS. Check "batch mode" and select the input and output directories. Check "creation options", click "+" and type "tfw" under "name" and "yes" under "value". I'll update my tutorial with the full details.

    Edited 6 times, last by Ian C (February 25, 2019 at 5:52 PM).

    Ian,

    yes, the aid files are the coordinate files directly used by aerofly geoconvert.

    TFW were necessary during preprocessing, either because they existed already or they could be created from the INF files of FSET.

    INF to TFW converter was written before IPACS made AID public.

    Quote from Rodeo

    Ian,

    yes, the aid files are the coordinate files directly used by aerofly geoconvert.

    TFW were necessary during preprocessing, either because they existed already or they could be created from the INF files of FSET.

    INF to TFW converter was written before IPACS made AID public.

    OK thanks Rodeo - that's clarified it!

    Hello peoples

    I have some questions found in TLDR at the bottom of this post for any who might assist me in solving a problem Im having.

    Though first I want to say Wow what at great game and give you some background on me, my setup and my problem

    Wow this game is awesome, looks great and is very simple to get into and more importantly mod.

    I say this as a non pilot gamer, whose cup of tea in sim games is more MS Combat Flight Sim3 than MS Fight Sim whats its name.

    I needed a game to try out my new head tracker and HOTAS setup up with.

    And frankly I found out all the other flight sims had meh graphics unless you traded your childrens education and then they were still meh compared to this one after doing some research on the web.

    So after trying Aerofly I liked it, well the areas that were done.

    The Default undone stuff is meh like those other sims

    Anyway not to complain as I understand the situation concerning this.

    So I bought the DLC addon's to spruce up the game and reduce the undone areas for what I though was reasonable cost.

    And it was good.

    So I fly around for a while and then got bored as its not combat game but thats not bad or a complaint and as a gamer I got games for that.

    So I would play the game form time to time depending on whim.

    One time I loaded my home town Sydney. What a disappointment in comparison to the other "Done" areas

    Well this of course was my start to Modding the game. Wow, the results and not every difficult thanks to the great tools made by the developers and community

    First I did the Orthogrphic textures in Google and Bing to compare and it blew my mind at the different it made

    Next tried my hand at cultivation, still working on it to bring it to standard but have put that aside for messing with elevation meshes.

    And this is area of modding were my questions lay so hope I posted in the the right place.


    Background on the problem:

    I made an elevation map following the video tutorial at the beginning of the thread.

    The source geotiff came from http://srtm.csi.cgiar.org/srtmdata/ and its quite a large area which I though good as it far exceeded my usual flying area reducing masking problems.

    So I downloaded a 5 x 5 degree tile called srtm_67_19.zip

    [Blocked Image: https://i.imgur.com/ceUM2pt.jpg]

    OverViewtemp.jpg

    I used the contents of this file and made the elevation images successfully.

    But when I started the game got those 20000ft things sticking out of the ground.

    Since the file is a geotiff and knowing nuthing about nothing did some research and discovered Qgis

    I could see the map real good saw no apparent problem so I thought I'd try another program, opened the thing in gimp and had a look

    I could hardly see a thing as the contrast was no where near as good as in Qgis but I could see the outline of the coast, highest peaks and a bunch of white dots.

    These white dots after some research were nodata pixels and not visible in Qgis whilst visible in Gimp as white dots and visible in game as those 20000ft things.

    At this time my thinking was I dont have a clue, Lets wing it.

    You know what they say, a little information ......

    Any way before I learnt Gimp couldn't edit with geotiff without losing geo data, I used Gimp to edit a copy and it worked rather good with some problems and work

    Firstly I'm an MS paint Man, so I'm learning Gimp real fast and got the colour dropper/picker and picked a dark pixel next to the white nodata pixel, then used that colour on the white no data pixel.

    Now I like my home town Sydeny so I did this by hand pixel by pixel and untill nodata was driven outa a town. Lucky for Sydney their wasn't that many dots.

    For other parts of the map I frequent less, unfortunately/ fortunately their was much more dots. I like those parts less so used a bigger brush, MS Paint style.

    And then ran geoconvert, installed it and it was good.

    Except now I had a new problem which was now the Ocean was now 20000ft I think, not sure exactly but I still had them 20000ft things near the coast

    At this stage I'm like busting to get this thing to work already so having a wild guess the ocean is causing my problems.

    So I go into gimp grab a beach pixel, should be close enough to sea level, right! And use this colour to fill the ocean.

    Reconvert and works great with a small loss of detail compared to the original geotiff but no 2000ft things and heaps better then the default.

    TLDR:

    So two questions

    Is their an easier way to deal with the no data issue and how do you deal with masks for example the Sydney Coast/Ocean interface does that need a mask.

    Can masks be used on Nodata pixels so that rather than 20000ft spikes the default terrain mesh shows underneath which probably much less noticeable than a 20,000ft spike.

    Its not a big deal I'm happy with my current results so are, just wondering if their a better way

    I did some screens shoots to compare.

    Default mesh

    [Blocked Image: https://i.imgur.com/Mj6CwxV.jpg]

    SRMT 30m Mesh

    [Blocked Image: https://i.imgur.com/8n1RQ8w.jpg]

    SRMT 30m Mesh after spike removal with Gimp

    [Blocked Image: https://i.imgur.com/PdO7sbI.jpg]

    The improvement over the default mesh is obvious though some detail maybe lost when the geotiff is edited in Gimp

    Or it could just be a slightly different camera angle, should of used the developers camera I guess

    Now to learn to cultivation guid

    Great game