Well wouldn't you know. My latest test gave me the same problem - very blocky edges where alpha layer masks coastline. Config was:
- Some FSET with coastline
- Some of the tiles converted to TIFF, with alpha and 100% hardness eraser with hard edge
- Converted with default TMC that had Level 9 [write_images_with_mask][true] and other levels [write_images_with_mask][false]
Next I'm going to reconvert with all layers set to [write_images_with_mask][true] and switch on images_raw to see what that reveals.
UPDATE: I set it to true and converted again. I've solved the problem of blurry scenery at low altitude, but the blocky edges are just like yours still. I think I see why though. With no feathering/blending then a 1pix/metre resolution mask will produce this kind of jaggyness. I was converting down to level 13 but will try 14 now to see if that improves things. However I don't really see a solution until IPACS enable smooth edge alpha layers to allow the FSET scenery to blend smoothly with the high level background. The other option would be to draw your own smooth coast so that the FSET scenery was smoothly blended at the coast with a border colour that matches the IPACS default water. More trouble than its worth I think. Let's put up with jaggy coastline until IPACS develop the tools further.
We really need a way to capture best practice when this is done. I'm sure plenty in the forum have done this right but I don't know where to find out. I'm all for learning new stuff and discovering/sharing new techniques but I hate reinventing the wheel.