OSM and Microsoft building footprints

  • Hi,


    there exist areas in the USA, where literally no buildings show up in the OSM data.


    Fortunately Microsoft published building footprints for the whole contry.

    US Building Footprints


    Microsoft's files by state are way too large for most systems to process.
    I did find some data splitted by US Census tract boundaries. These files can be handled by most GIS software. See the different colors representing census tract areas.



    Merging them with OSM data reveals the dramatic increase of buildings.

    Pure OSM:


    OSM with US building footprints:


    Another example:



    There is some overlay and even some inaccuracy of the buildings.

    But for our purpose the inaccuracy is negligible.

    I assume, best way is to remove all OSM buildings and keep the Microsoft buildings.

    But next, we have to mass modify the attributes of these buildings.



    Currently I work on the whole process manually, but it would be a significant improvement to find a good workflow.

    What do you think about it?


    Regards

    Rodeo

  • Is any of the inaccuracy due to not taking into account the earths curvature?

    Devons rig

    Intel Core i7 8700K @ 5.0GHz / 32.0GB G.SKILL TridentZ Series Dual-Channel Ram / ZOTAC GAMING GeForce® RTX 2080 Ti Triple Fan / Sound Blaster Z / Oculus Rift VR Headset / Klipsch® Promedia 2.1 Computer Speakers / ASUS ROG SWIFT PG279Q ‑ 27" IPS LED Monitor ‑ QHD / 2x Samsung SSD 850 EVO 500GB / Windows 10 Pro 64-bit /Gigabyte Z370 AORUS Gaming 5 Motherboard

  • I'm going to do something with this data for AeroScenery at some point, sadly not enough hours in the day at the moment.


    My plan was to write something to split it into lots of 0.1 x 0.1 degree files while converting the data format to OSM.

    From there they can be brought in to scenProc easily.


    When you say

    Quote

    But next, we have to mass modify the attributes of these buildings.

    can you let me now what needs to be done there?

  • If there are attributes it MS buildings, then it's cool and I would reckon they don't need to be modified, just used.

    If there is no attribute in MS buildings, then it's just like most buildings in OSM...


    Nick, if you can translate these files into shapefiles instead of OSM, I think it's more powerful and better suited for opening/checking/editing attributes in QGIS or similar and work with ScenProc.


    I see some tools already exist for GeoJSON to SHP convertion, maybe you can pick some existing code without having to reinvent the wheel...

    https://mygeodata.cloud/converter/geojson-to-shp


    Cheers

    Antoine

    Config : i7 6900K - 20MB currently set at 4.00GHz, Cooling Noctua NH-U14S, Motherboard ASUS Rampage V Extreme U3.1, RAM HyperX Savage Black Edition 16GB DDR4 3000 MHz, Graphic Card Gigabyte GeForce GTX 1080 8GB, Power supply Corsair RM Series 850W, Windows 10 64 bit.

  • there are no attributes with the MS buildings. I can imagine to add attributes as we use in aerofly.

    Housetypes residential, industrial,

    rooftypes gable or flat,

    number of levels.

    I see. I haven't really looked at the MS data yet.


    Someone has written a Python script to get a chunk of OSM data and conflate MS footprint data with it.


    https://github.com/maxerickson/osm_ms_buildings


    This could be useful for getting the data down to a manageable size, with the result being one OSM file with OSM buildings, and one OSM file with MS only buildings.


    I guess some attributes can be worked out by scenProc from the height?

  • Is it possible to add osm height and type data (where available) to the Ms Buildings? I know in some programs (Outerra comes to mind) blending disparate datasets is well on its way to becoming a high art.


    EDIT: Looks like I cross posted with Nickhod.

    Devons rig

    Intel Core i7 8700K @ 5.0GHz / 32.0GB G.SKILL TridentZ Series Dual-Channel Ram / ZOTAC GAMING GeForce® RTX 2080 Ti Triple Fan / Sound Blaster Z / Oculus Rift VR Headset / Klipsch® Promedia 2.1 Computer Speakers / ASUS ROG SWIFT PG279Q ‑ 27" IPS LED Monitor ‑ QHD / 2x Samsung SSD 850 EVO 500GB / Windows 10 Pro 64-bit /Gigabyte Z370 AORUS Gaming 5 Motherboard

  • Is it possible to add osm height and type data (where available) to the Ms Buildings? I know in some programs (Outerra comes to mind) blending disparate datasets is well on its way to becoming a high art.

    As far as I can tell, the MS data already has height information for each building.


    (At least the small North and South Dakota data sets I tried did).


    I think the challenge was the overlap with OSM data, and the size of the MS data files, which that Python script should solve.

  • I started using SQLite + SQLiteStudio and dumping all the data in tables.

    Then I add all the attributes for High Rise, Residential, Roads and so forth (real fast)

    I use select statements to extract (break city down into pieces) import into QGIS and export then as a shapefile.

    Then I open the shapefile in JOSM, check it for inconsistencies, save it as a OSM file and run through sceneProc.

    Its a long process, but it's fast.

    Windows 10 64bit | iCore7-7770K| ASUS STRIX Z270E |KHX2400C15D4 - 32GB | Gigabyte Ge-Force GTX 1080Ti| SSD Intel 520 Series | 40" Philips BDM4065UC/75 4K - Oculus Rift

  • As far as I can tell, the MS data already has height information for each building....

    Really? That would be fantastic. Perhaps the datasets I used lost their attributes during previous splitting processes.


    But I just opened Microsoft Hawaii.json, these are the first lines:

    {

    "type":"FeatureCollection",

    "features":

    [

    {"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[-159.6574339277891,21.979007865692839],[-159.65747968724611,21.97908357656307],[-159.65755416870371,21.979044870723161],[-159.65750841359531,21.978969157572461],[-159.6574339277891,21.979007865692839]]]}},


    Unfortunately no height information in the complete file. But maybe there are other databases to be combined with the buildings.

  • Really? That would be fantastic. Perhaps the datasets I used lost their attributes during previous splitting processes.

    Yep, you're right. I followed a link to the previous version of this data which only had 9.8 million buildings. It did seem to have height data though. The newer, large dataset version doesn't.

    https://wiki.openstreetmap.org…t_Building_Footprint_Data

  • I started using SQLite + SQLiteStudio and dumping all the data in tables.

    Then I add all the attributes for High Rise, Residential, Roads and so forth (real fast)

    I use select statements to extract (break city down into pieces) import into QGIS and export then as a shapefile.

    Then I open the shapefile in JOSM, check it for inconsistencies, save it as a OSM file and run through sceneProc.

    Its a long process, but it's fast.

    Nice workflow.


    Why the JOSM and OSM step ? Why not simply checking the shapefile in QGIS and process it directly with ScenProc ? It would save you some steps...


    Cheers

    Antoine

    Config : i7 6900K - 20MB currently set at 4.00GHz, Cooling Noctua NH-U14S, Motherboard ASUS Rampage V Extreme U3.1, RAM HyperX Savage Black Edition 16GB DDR4 3000 MHz, Graphic Card Gigabyte GeForce GTX 1080 8GB, Power supply Corsair RM Series 850W, Windows 10 64 bit.

  • It allows me to edit the file & save it.

    I would need to add an exporter plugin to re-save as a shapefile.

    I think either way, it's the same amount of steps :-)

    Windows 10 64bit | iCore7-7770K| ASUS STRIX Z270E |KHX2400C15D4 - 32GB | Gigabyte Ge-Force GTX 1080Ti| SSD Intel 520 Series | 40" Philips BDM4065UC/75 4K - Oculus Rift

  • It allows me to edit the file & save it.

    I would need to add an exporter plugin to re-save as a shapefile.

    I think either way, it's the same amount of steps :-)

    I don't quite get it : with the preceding step you saved it as a shapefile... in QGIS you can edit your shapefile

    Config : i7 6900K - 20MB currently set at 4.00GHz, Cooling Noctua NH-U14S, Motherboard ASUS Rampage V Extreme U3.1, RAM HyperX Savage Black Edition 16GB DDR4 3000 MHz, Graphic Card Gigabyte GeForce GTX 1080 8GB, Power supply Corsair RM Series 850W, Windows 10 64 bit.

  • I don't quite get it : with the preceding step you saved it as a shapefile... in QGIS you can edit your shapefile

    I like editing in JOSM :-)

    QGIS is good for import - convert - export in most formats and converting centroids to polygons :-)

    Windows 10 64bit | iCore7-7770K| ASUS STRIX Z270E |KHX2400C15D4 - 32GB | Gigabyte Ge-Force GTX 1080Ti| SSD Intel 520 Series | 40" Philips BDM4065UC/75 4K - Oculus Rift