A second square, and the area types it needed

Everything else on this site was measured on one square: N37W123, San Francisco Bay. A single measurement is not a finding, and the area type mapping that came out of it was really a San Francisco mapping wearing a general name.

So the toolchain was run again on a square chosen to disagree with the first one as much as possible. N49E007: the Pfalz, the Hunsrück and the Nahe valley, in Rheinland-Pfalz. No coastline. No city. Farmland and forest where San Francisco had suburbs and ocean, and vineyards, which San Francisco had almost none of. Then a third, S23E017 in Namibia, to find out what happens where OpenStreetMap has almost nothing to say.

Three things came out of it. The mapping needed real work to travel. The timings from the first square were measuring something other than what this site claimed. And the first version of the second build was quietly wrong, in a way that took a picture rather than a log to notice.

The mapping was tuned to one landscape

Both squares were cut from Geofabrik’s OpenStreetMap extracts and counted the same way. The classes barely overlap in rank:

The largest single class in the Pfalz is meadow, thirty-two times more common there than in San Francisco, and the first mapping folded it in with municipal lawns. Vineyards went from a rounding error to the fifth largest class and were being drawn as ploughed field.

Except that counting polygons is the wrong measure

A count tells you how many times somebody drew a shape, not how much ground it covers. Rasterising each layer at 1500 × 1500 over the square and measuring the pixels it claims gives a different ranking; and a much better idea of what the scenery will look like:

scrub is the clearest example of the gap between the two measures. It is the second most numerous class in the Pfalz (21,519 polygons) and covers 1.8% of the ground. forest is drawn fewer times and is 41% of the square. If you are deciding which layers are worth decoding, count area, not features.

TerraGear has 96 area types. The first mapping used ten

The list is in src/BuildTiles/Main/default_priorities.txt, and it already contains almost everything the Pfalz needed:

Vineyard        Orchard         Heath           Cemetery
GolfCourse      Town            SubUrban        Industrial
DryCrop         IrrCrop         MixedCrop       ComplexCrop
CropGrass       CropWood        Grassland       Scrub

Nothing had to be invented. Vineyard, Orchard, Heath and Cemetery were sitting there unused, and Town distinguishes a village from a city centre where the first mapping had one Urban for both. The second build uses fourteen layers instead of ten:

ogr-decode --area-type Vineyard  --max-segment 500 \
  $WORK/Vineyard clipped/landuse.shp landuse \
  --where "fclass = 'vineyard'"

ogr-decode --area-type Orchard   ... --where "fclass IN ('orchard','allotments')"
ogr-decode --area-type Heath     ... --where "fclass = 'heath'"
ogr-decode --area-type Cemetery  ... --where "fclass = 'cemetery'"
ogr-decode --area-type Town      ... --where "fclass = 'residential'"
ogr-decode --area-type Grassland ... --where "fclass IN ('meadow','grass','park','recreation_ground','village_green')"
ogr-decode --area-type DryCrop   ... --where "fclass IN ('farmland','farmyard')"

All fourteen decoded without complaint, and all four of the new ones survive into the finished scenery; read back out of the triangle-face properties of the .btg.gz files, the way checking output describes:

Town        29 groups     Cemetery    27 groups
DryCrop     29 groups     Orchard     27 groups
Grassland   29 groups     Heath       18 groups
Scrub       28 groups     Vineyard    11 groups

Eleven groups for Vineyard rather than the twenty-nine the common classes get, because the vineyards are real and clustered: they are on the Nahe slopes in the east of the square and nowhere else in it.

A hole in the data, how to spot one, and what it costs

The first version of this build used rheinland-pfalz-latest-free.shp.zip alone, and a quarter of the square came out with no land cover at all. Not moorland, not a decode failure: the extract contains Rheinland-Pfalz, and the south-west corner of N49E007 is in Saarland and France.

The trap is that this is invisible from the metadata. The layer’s declared extent is 6.04°–8.57°E, 48.92°–50.95°N, which covers the square completely. The bounding box crosses the border; the content stops at it. Probing four points, counting features within 0.02° of each:

7.1E 49.2N  (Saarland / France)        3 features
7.9E 49.2N  (Pfalz)                  251
7.1E 49.8N  (Hunsrück)               205
7.9E 49.8N  (Nahe)                   407

Telling a missing download from thin mapping

28.6% of the land had no land-cover feature on it. That figure alone means nothing; San Francisco’s square is 34.6% bare, and nobody thinks northern California is unmapped. The tell is where the gap sits:

Sparse mapping is everywhere. A missing download has a corner. Merging the four neighbouring extracts (Saarland, Lorraine, Baden-Württemberg and Hessen) took the square from 28.6% bare to 9.4%, and flattened the distribution to something that looks like San Francisco’s: real unmapped ground, spread about evenly. That is the same test working in both directions, which is the reason to trust it.

for r in rlp saarland lorraine bw hessen; do
  ogr2ogr -f "ESRI Shapefile" $APPEND -clipsrc 7 49 8 50 \
    merged/landuse.shp $r/gis_osm_landuse_a_free_1.shp
  APPEND=-append
done

85,299 land-use polygons became 108,505. The schemas match across Geofabrik regions, which is what makes a plain -append safe.

The part that is not just tidiness

An incomplete extract does not only leave a visible blank. It biases every proportion downward, because the missing quarter counts as ground with nothing on it. Both columns below are the same square, on the same day, from the same commands:

This page carried the 41.2% figure before the merge, next to San Francisco’s 12%, as though the comparison were sound. It was understated by eight percentage points. Rasterise your layers and look at the picture before you trust a number off them; one pass over the shapefiles, and it would have been invisible in any log.

The cost of being right was 66 seconds and 19 MB. The figures on the rest of this page are the merged ones.

Side by side

The Pfalz square has more terrain detail, more land-cover layers and half as many vertices again, and still built in forty seconds less. It also has fewer materials, which is not a contradiction: most of San Francisco’s 85 were airport paint, and KSFO has a larger vocabulary of markings than seven small German fields put together.

Do not read that 246 as “this square is easier”. The tiles that make up the square took 179 s here against 36.6 s in San Francisco; nearly five times the work, which is what you would expect from a square that is land all the way across and needs 70% more terrain nodes. The whole of the difference in the totals is in the next section, and it is not about either landscape.

The node counts are the part that behaves as you would expect. terrafit keeps detail where the ground has any: San Francisco is half open water and needs almost nothing there, while the Pfalz is folded hills from edge to edge, so it keeps 70% more nodes from the same 12,967,201 input samples.

Tiles the build makes and then throws away

A one-degree box does not produce 32 tiles. It produces 36. The extra four sit in the row immediately north of the region, they have no elevation under them because hgtchop was given one tile, and they write nothing: every file in every output directory here belongs to the bucket inside the box.

The cause is in SimGear rather than TerraGear. sgGetBuckets() runs its latitude loop to max + SG_BUCKET_SPAN and its longitude loop to <= max, so a boundary that lands exactly on a tile edge picks up a whole row and column beyond it. Those lie outside the box: it touches them along a line of zero area.

-    for (lat = min.getLatitudeDeg(); lat < max.getLatitudeDeg()+SG_BUCKET_SPAN; ...)
-        for (lon = min.getLongitudeDeg(); lon <= max.getLongitudeDeg(); lon += span)
+    for (lat = lat_min; lat < lat_max || lat == lat_min; lat += SG_BUCKET_SPAN)
+        for (lon = lon_min; lon < lon_max || lon == lon_min; lon += span)

Patched and rebuilt, all three squares construct exactly 32 tiles and produce byte-for-byte the same output: 53 files for San Francisco, 39 for the others. The == min terms keep a degenerate box, where both corners fall in one tile, returning that tile.

A measurement of mine that was wrong

This page previously said those four tiles took 87% of the San Francisco build, and gave per-row timings for all three squares. That was wrong, and the fix is what proved it: removing the row changed the wall clock from 286 s to 291 s on San Francisco and from 15.07 s to 15.08 s on Namibia. Nothing.

The error was in how the timings were taken. They came from the gaps between consecutive Construct in … lines in the build log, which would be per-tile costs if the build were serial. It runs on ten threads, and those lines are emitted as each thread picks up its next tile, so the gap between two of them is the wait for a different thread, not the cost of a tile.

What survives is smaller and still worth fixing: the build does work it throws away, and a one-line change stops it. It is a correctness point, not a performance one, and the report now says so.

It loads

Two things that went wrong getting that picture

Neither is region-specific, and both cost more time than the build did.

tg-construct‘s output layout is not the one FlightGear accepts. It writes a flat bucket tree, and pointing --fg-scenery at that gives you open ocean. In the log, every file is found and then declined:

Found scenery file Path ".../e000n40/e007n49/3072754.stg" in scenery path /w/output-rlp.
Scenery paths without type subdirectories are no longer supported, please move thse files

It needs Terrain/ and Objects/ subdirectories above the buckets. Splitting them by kind does not work either: the tile .stg files reference their airports by bare filename and expect them in the same directory, so moving ETAR.btg.gz into Objects/ turns every airport into read error. What works is the whole tree copied under Terrain/:

mkdir -p scenery/Terrain scenery/Objects
cp -r output/* scenery/Terrain/
fgfs --fg-scenery=$PWD/scenery ...

And a check that lied. The screenshot script decided a frame was rendered when it contained more than 500 colours. FlightGear’s sky alone has thousands, so it reported success on a frame that was ocean to the horizon with no scenery loaded at all. It now crops the bottom 40% and counts colours in the ground, which is the thing being tested. Worth remembering if you automate a visual check of anything: measure the part you care about, not the frame it arrives in.

What carried over unchanged

• Pre-clip every layer. Same as before, for the same reason: ogr-decode --spat filters features, it does not clip geometry. Clipping fourteen layers with ogr2ogr took 7 seconds.
• --threads=4 for genapts. Seven airports, no crash, and no reason to think the chop.idx race is region-specific.
• Mixed BTG versions. 39 files, versions 7 and 10 in the same build, exactly as on the first square.
• The gzip hang. --input=apt.dat.gz was still spinning when it was killed at 20 seconds, on a run building a single airport.

A third square, where the map is nearly empty

Both squares so far are temperate, densely mapped and full of human land use. The obvious question about a fourteen-layer mapping is what it does where none of those things are true, so: S23E017; Windhoek, the Khomas Highland and the edge of the Kalahari.

The mapping degrades quietly, which is the good outcome

Two of the fourteen layers match nothing at all here. ogr-decode writes zero files and exits 0; tg-construct loads the empty directory without comment; the build completes.

Vineyard     Vineyard         exit=0    0s     0 files
Heath        Heath            exit=0    0s     4 files
Orchard      Orchard          exit=0    0s     4 files
Forest       EvergreenForest  exit=0    0s    15 files

Worth knowing, because it means a mapping written for one region is safe to carry to another; you get less, not something wrong. It also means nothing warns you that two of your layers did nothing, so the zero in that column is the only signal you will get.

What sparse data costs, which is almost nothing

Three squares of identical size on one machine, and the spread is 19× in time and 7× in memory. Area is not what you are paying for. You are paying for the number of shapes in your data, which is a property of how well somebody mapped the place; not of the place.

The BTG row is a detail that falls out for free: this square’s tiles never exceed what a 16-bit count can address, so nothing triggers version 10. A build writes whichever version each tile needs.

And the tiles outside the box, a third time

Namibia builds 36 tiles for a 32-tile request, like the other two, and like the other two the extra four write nothing. The patch above brings it to 32 with identical output. It is the same defect, and on a square this sparse it is equally cheap: 15.07 s before, 15.08 s after.

What this square does not settle

Three squares on one machine is three data points, not a curve. All three used SRTM 1-arc-second and the same worldwide apt.dat, all three were built in the same container by the same person on the same day, and none of them is a rainforest, a mountain range, or anywhere the elevation data itself is poor.

The fourteen-layer mapping is still a Pfalz mapping. Namibia showed it fails safely where classes are missing, which is not the same as showing it is right for somewhere it has not seen: a square with rice paddy, or mangrove, or irrigated desert agriculture would need area types none of these three needed, and TerraGear has 96 of them for a reason.

The scripts for both builds, and the raw per-stage logs, are published with the build record.