Reflectance pre-processing: masking out bad weather data in GEE

Since reflectance data is generated from a passive energy source (the sun), data collected in cloudy sky conditions are not directly comparable to data collected in clear-sky conditions, as overhead clouds can obscure the incoming light source. AOP aims to collect data only in optimal (<10% cloud-cover) weather conditions, but cannot always do so due to logistical constraints. The flight operators record the weather conditions during each flight, and this information is passed through to the final data product at the level of the flight line (as cloud conditions can change throughout the day). Cloud conditions are reported as green (<10% cloud cover), yellow (10-50% cloud cover), or red (>50% cloud cover). The figure below shows some examples of what the cloud conditions look like at different flights collected in the three different weather classes (green, yellow, and red).

Note that there is an important distinction between airborne and satellite reflectance data. Satellite data is collected in all weather conditions, and the clouds are below the sensor, so algorithms can be generated to filter out cloudy pixels. With aerial data, we have more control over when the data are collected, to a degree. However, clouds may be present overhead, if it were deemed necessary to collect in sub-optimal weather conditions. AOP typically will only collect in "red" sky conditions if we are running out of time in a Domain and the weather isn't forecasted to improve. Since the clouds won't appear in the actual data, maintaining this record of cloud conditions is essential for properly understanding the data, and using it for change detection or other research applications. For a more direct comparison of reflectance values, we recommend only working with the clear-weather data. This lesson outlines how to do this in GEE.

Read in the AOP SDR 2019 Dataset at SOAP

We will start at our ending point of the last tutorial. For this exercise, we will read in data from the NEON site Soaproot Saddle (SOAP) collected in 2019:

// Filter image collection by date and site to pull out a single image
var soapSDR = ee.ImageCollection("projects/neon-prod-earthengine/assets/HSI_REFL/001")
  .filterDate('2019-01-01', '2019-12-31')
  .filterMetadata('NEON_SITE', 'equals', 'SOAP')
  .first();

Display the QA Bands

From the previous lesson, recall that the SDR images include 442 bands. Bands 0-425 are the data bands, which store the spectral reflectance values for each wavelength recorded by the NEON Imaging Spectrometer (NIS). The remaining bands (426-441) contain metadata and QA information that are important for understanding and properly interpreting the hyperspectral data. The data bands all follow the naming convetion B001, B002, ..., B426, and the QA bands start with something other than the letter "B", so we can use that information to extract the QA bands.

// Pull out and display only the qa bands (these all start with something other than B)
// '[^B].*' is a regular expression to pull out bands that don't start with B
var soapSDR_qa = soapSDR.select('[^B].*') 
print('QA Bands',soapSDR_qa)

Most of these QA bands are inputs to and outputs from the Atmospheric Correction (ATCOR). We will expand upon these further, and encourage you to read more details about these data in the NEON Imaging Spectrometer Radiance to Reflectance Algorithm Theoritical Basis Document. For the purposes of this exercise, we will focus on the Weather Quality Indicator band. Note that you can explore each of the QA bands, following similar steps below, adjusting the band names and values accordingly.

Read in the Weather_Quality_Indicator Band

The weather information, called Weather_Quality_Indicator is one of the most important pieces of QA information that is collected about the NIS data, as it has a direct impact on the reflectance values.

These next lines of code pull out the Weather_Quality_Indicator band, select the "green" weather data from that band, and apply a mask to keep only the clear-weather data, which is saved to the variable soapSDR_clear.

// Extract a single band Weather Quality QA layer
var soapWeather = soapSDR.select(['Weather_Quality_Indicator']);

// Select only the clear weather data (<10% cloud cover)
var soapClearWeather = soapWeather.eq(1); // 1 = 0-10% cloud cover

// Mask out all cloudy pixels from the SDR image
var soapSDR_clear = soapSDR.updateMask(soapClearWeather);

Plot the weather quality band data

For reference, we can plot the weather band data, using AOP's stop-light (red/yellow/green) color scheme, with the code below:

// center the map at the lat / lon of the site, set zoom to 12
Map.setCenter(-119.25, 37.06, 11);

// Define a palette for the weather - to match NEON AOP's weather color conventions
var gyrPalette = [
  '00ff00', // green (<10% cloud cover)
  'ffff00', // yellow (10-50% cloud cover)
  'ff0000' // red (>50% cloud cover)
];

// Display the weather band (cloud conditions) with the green-yellow-red palette
Map.addLayer(soapWeather,
             {min: 1, max: 3, palette: gyrPalette, opacity: 0.3},
             'SOAP 2019 Cloud Cover Map');

Plot the clear-weather reflectance data

Finally, we can plot a true-color image of only the clear-weather data, from soapSDR_clear that we created earlier:

// Create a 3-band cloud-free image 
var soapSDR_RGB = soapSDR_clear.select(['B053', 'B035', 'B019']);

// Display the SDR image
Map.addLayer(soapSDR_RGB, {min:103, max:1160}, 'SOAP 2019 Reflectance RGB');

Plot acquisition dates

We can apply the same concepts to explore another one of the QA bands, this time let's look at the Acquisition_Date. This may be useful if you are trying to find the dates that correspond to field data you've collected, or you want to scale up to satellite data, for example. To determine the minimum and maximum dates, you can use reduceRegion with the reducer ee.Reducer.minMax() as follows. Then use these values in the visualization parameters.

// Get the minimum and maximum values of the soapDates band
var minMaxValues = soapDates.reduceRegion({reducer: ee.Reducer.minMax(),maxPixels: 1e10})
print('min and max dates', minMaxValues);
	
// Extract acquisition dates and map, don't display layer by default
var soapDates = soapSDR.select(['Acquisition_Date']);
Map.addLayer(soapDates,
            {min:20190612, max:20190616, opacity: 0.5},
            'SOAP 2019 Acquisition Dates',0);

Recap

In this lesson you learned how to read in Weather Quality Information from the Surface Directional Reflectance (SDR) QA bands in GEE. You learned to mask data to keep only data collected in the cleares sky conditions (<10% cloud cover), and plot the three weather quality classes. You also learned how to find the other QA bands, and following a similar approach could explore each of these bands similarly. Filtering by the weather quality is an important first pre-processing step to working with NEON hyperspectral data, and is essential for interpreting the data and carrying out subsequent data analysis.

Get Lesson Code

AOP GEE SDR Weather Quality