Troubles with TIFF: No Data Values with Floating Point


Update – We’ve moved to generating the TIFF DEMs with LAStools. The NoData value is now stored in TIFF tag ID 42113 instead of as an IEEE  not a number as described below, the same as is used by GDAL. Packages that use GDAL should have no problems. If you’re still having problems, you can either try some of the suggestions below or you might try using gdal_translate to make a copy of the file using the GeoTIFF profile. This will put the nodata information in a PAM .aux.xml (see the GDAL GeoTIFF page). For example:
gdal_translate -of GTiff -co PROFILE=GeoTIFF input.tif output.tif

 

The TIFF format has been around for quite some time. The current specification (TIFF Revision 6.0) was published in 1992. It has been a highly useful format for lots of disciplines, including remote sensing. The format is very flexible and uses information “tags” in the file to describe the file contents. You can even add new user defined tags to add information that nobody thought of in 1992. However, unless a tag becomes incorporated in the specification as a standard, common software is unlikely to understand it. This is where I run into trouble trying to use the TIFF format for lidar derived DEMs, because there is no standard tag for a “no data” value.

Digital Coast and Elevation TIFFs

Our Digital Coast data system will generate DEMs from the lidar points and a very common output choice is a 32-bit floating point TIFF. However, it is the “nature of the beast” that there will be areas in your rectangular DEM tile where there are no lidar data to provide an elevation value. This is particularly true in the coastal areas because relatively few data sets have both topography and bathymetry data. Even those that do invariably have gaps somewhere and don’t extend globally. We still have to put a value in the cell though! So, what do you put?

I’ve seen a few different values out there to represent no data. If you’re using a file format that allows specifying what value was used, it doesn’t really matter as long as it won’t conflict with real data. Examples include -32767 or -9999. I believe the USGS NED uses zero, but that won’t work for us since we include data below zero. The only thing that made logical sense to us was to use one of the IEEE Not a Number (NaN) values. I assumed software packages would automatically pick up that the cell was not valid and wouldn’t use it when figuring out a scale to color the image. You know what they say about assumptions!

Making It Work in ArcGIS

For some software packages, my assumption worked out (e.g. Global Mapper). However, for the package that most of our constituents use, Esri ArcGIS, it didn’t. ArcGIS isn’t the only software that didn’t like my solution, but it’s the one I have to worry most about. When you first bring in a TIFF image with IEEE NaN values into ArcGIS, you’ll see an image that has just two colors/tones and crazy numbers on the scale bar (see figure 1). The range shown is essentially the maximum range for 32-bit IEEE 754 standard floating point numbers. So, what is the trick in ArcGIS to get it to work right?

A 32-bit floating point TIFF containing IEEE NaN values when first read into ArcGIS. The image appears to have only two colors because the NaN value skew the scaling.
Figure 1. A 32-bit floating point TIFF containing IEEE NaN values when first read into ArcGIS. The black areas have no data due to either building removal or water (Charleston Harbor).

It turns out that if you do something to make ArcGIS calculate the statistics of the image, it figures out that those aren’t really values and scales it properly. For instance, telling it to scale the image with a two standard deviation stretch (see figure 2). If you do that, figure 1 becomes figure 3 and all seems right with the world again. Well, at least closer to right. The no data values end up colored the same as the low end of the color ramp even though doing a query on the cells shows “no data.” If anyone knows how to fix that, I’d love to hear it. Or if you have a better answer for what to put in those TIFF cells.

Setting the stretch type from “None” to “Standard Deviations” will trigger the compute statistics question
Figure 2. Screen shot of computing statistics. Setting the stretch type from “None” to “Standard Deviations” will trigger the compute statistics question.
data after computing statistics and using a rainbow color ramp
Figure 3. The same data is shown after computing statistics and using a rainbow color ramp. Note the range of values in the legend now makes sense.

 

2 comments

  1. Thanks for this post, it was very informative. The problem of what value to use in No Data areas is a challenging one, especially since different GIS systems behave differently in this regard. Your approach seems to make sense. I tested some lidar data downloaded from the Digital Coast Data Access Viewer in the MapWindow GIS system, which OpenNSPECT uses. MapWindow recognized the NoData value as “no data” and behaved appropriately.

    I also tested the same data set with Esri ArcGIS and have a few observations that might be helpful to others.

    In addition to using the color ramp technique to generate statistics for TIFFs, you can directly generate them using the “Calculate Statistics” function in Arc Catalog: just right click on the target raster in Arc Catalog pane.
    The continued display of the NoData areas is odd and seems to be a function of the Stretched color ramps. If you use a classified color ramp, you can show the NoData areas with a different color, or not at all, as desired. This option appears in the Stretch GUI, but doesn’t seem to work. Oddly, if you export the TIFF to a grid format, it still doesn’t work, which makes me think it is a data value issue, not a file format issue.
    If you really want those background NoData values to not show up (Which I do!), here is a trick: a Con command to pick the areas that are valid, then the rest will be set to Esri’s NoData value::
    Con( “Filename.tif” > – 40,”Filename.tif”),

    where Filename.tif is my TIFF file. This works, regardless of the file format of the results, TIFF or Grid.

    It is interesting that you can not just use:

    SetNull(IsNull(“Filename.tif”),”Filename.tif”), or SetNull(Con( “Filename.tif” < -40, "Filename.tif"), "Filename.tif")

    The first of these essentially says, fill all the NoData places with NoData values. But it doesn't work. It appears that the NoData values in the TIFF are not recognized as NoData for the purposes of the IsNull command. So if you do this, you get the original data at all locations. The second of these says, for areas with values less than -40 (chosen since all the "real" data are above that), fill them with NoData values. This approach fails, but for a different reason: The Con command recognizes the current NoData value as NoData, therefore it doesn't perform any calculations for those areas. Therefore, nothing matches the SetNull criteria, so everything is set to the original data. Very odd.

    However, you can trick the system by using a Con command to pass on values where you have legitimate data values. Where you don't have values that meet the Con criteria, Esri fills in Esri NoData values, which then work as desired.

    All this indicates to me that Esri has at least two ways they evaluate NoData values: one approach is used in the Stretched color ramp, IsNull, (and other?) functions, and another one is used in the Classified color ramp and Con (and other?) functions. The value you provide via the DAV system for a NoData flag is treated differently in these, whereas Esri's NoData value is treated consistently. Interesting.

    Thanks again, I look forward to hearing more about TIFFs. I use them a lot, but they do have some odd behaviors.

    Like

Leave a Reply. Comments are moderated.

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s