## On the choice of retrieval variables in the inversion of remotely sensed atmospheric measurements |

Optics Express, Vol. 21, Issue 9, pp. 11465-11474 (2013)

http://dx.doi.org/10.1364/OE.21.011465

Acrobat PDF (743 KB)

### Abstract

In this paper we introduce new variables that can be used to retrieve the atmospheric continuum emission in the inversion of remote sensing measurements. This modification tackles the so-called *sloppy model* problem. We test this approach on an extensive set of real measurements from the Michelson Interferometer for Passive Atmospheric Sounding. The newly introduced variables permit to achieve a more stable inversion and a smaller value of the minimum of the cost function.

© 2013 OSA

## 1. Introduction

*I*(

*x*) is the intensity at spatial coordinate

*x*, and

*K*is a frequency-dependent constant. Equation (1) has the general solution

_{σ}*I*(

*x*) =

*c*exp(−

*K*). As a consequence the radiance reaching the instrument is modeled as a linear combination of exponential terms. Disregarding scattering, which is justified in the mid-to-far infrared spectral region in clear sky conditions, the radiance

_{σ}x*S*(

*σ*) at frequency

*σ*reaching the instrument for a fixed observation geometry is described by the radiative transfer equation [1

1. T. von Clarmann, M. Hpfner, B. Funke, M. López-Puertas, A. Dudhia, V. Jay, F. Schreier, M. Ridolfi, S. Ceccherini, B. J. Kerridge, J. Reburn, R. Siddans, and J.-M. Flaud, “Modelling of atmospheric mid-infrared radiative transfer: The AMIL2DA algorithm intercomparison experiment,” J. Quant. Spectrosc. Radiat. Transfer **78**(3–4), 381–407 (2003) [CrossRef] .

*s*is the coordinate along the line of sight,

*s*

_{obs}is the coordinate of the observing instrument,

*τ*(

*σ*,

*s*

_{1},

*s*

_{2}) is the spectral transmittance along the line of sight from position

*s*

_{1}to position

*s*

_{2},

*B*

_{0}(

*σ*) is the background radiance and

*B*(

*σ*,

*T*(

*s*)) is the source function which, under the assumption of local thermodynamic equilibrium, becomes the Planck function, which depends on the kinetic temperature

*T*(

*s*) and the frequency

*σ*. In the limb scanning geometry, the term

*B*

_{0}(

*σ*) is assumed to be zero. A common choice is to discretize the atmosphere with a series of spherical homogeneous layers. In this case, let

*l*= 1,...,

*N*be the layers crossed by a given line of sight, and

*l*[2

2. A. R. Curtis, “Discussion of a statistical model for water vapor absorption,” Q. J. R. Meteorol. Soc. **78**, 638–640 (1952) [CrossRef] .

3. W. L. Godson, “The evaluation of infrared radiative fluxes due to atmospheric water vapour,” Q. J. R. Meteorol. Soc. **79**, 367–379 (1953) [CrossRef] .

*τ*is the transmittance through layer

_{σ,l}*l*at frequency

*σ*. The transmittance in the layer

*l*has the following expression [4

4. M. Ridolfi, B. Carli, M. Carlotti, T. von Clarmann, B. M. Dinelli, A. Dudhia, J.-M. Flaud, M. Höpfner, P. E. Morris, P. Raspollini, G. Stiller, and R. J. Wells, “Optimized forward model and retrieval scheme for MIPAS near-real-time data processing,” Appl. Opt. **39**, 1323–1340 (2000) [CrossRef] .

5. P. Raspollini, C. Belotti, A. Burgess, B. Carli, M. Carlotti, S. Ceccherini, B. M. Dinelli, A. Dudhia, J.-M. Flaud, B. Funke, M. Höpfner, M. López-Puertas, V. Payne, C. Piccolo, J. J. Remedios, M. Ridolfi, and R. Spang, “MI-PAS level 2 operational analysis,” Atmos. Chem. Phys. **6**, 5605–5630 (2006) [CrossRef] .

*m*is the gas index,

*k*is the cross-section of that gas at frequency

_{σ,m}*σ*, which depends only on

*c*is the gas column in the layer along the line of sight, which depends on the gas Volume Mixing Ratio (VMR)

_{l,m}*x*(

_{m}*z*) in the following way: where

*z*is the altitude,

*η*(

*z*) is the number density and

*ds/dz*is the linear element along the line of sight. Let

*z*be the retrieval grid. We may use the values

_{i}*x*=

_{i,m}*x*(

_{m}*z*) as retrieval variables, and model the VMR with a piecewise linear curve with nodes

_{i}*z*. It follows that

_{i}*c*is a linear function of

_{l,m}*x*so that the entire spectrum will be a linear combination of exponential terms

_{i,m}*e*of the form exp(∑

_{j}

_{i}_{∈}

_{I}_{(j)}

*d*) for some constants

_{i}x_{i,m}*d*, and the sum is extended on subsets

_{i}*I*(

*j*) of indexes.

*χ*

^{2}is defined as

*ℓ*

_{2}norm of the differences between observed and simulated radiances (the so-called residuals) weighted with the inverse of the error covariance matrix of the measurements.

*x*have a physical meaning, their selection as retrieval variables makes the forward model a

_{i,m}*sloppy model*, as described in [6

6. M. K. Transtrum, B. B. Machta, and J. P. Sethna, “Geometry of nonlinear least squares with applications to sloppy models and optimization,” Phys. Rev. E **83**, 036701 (2011) [CrossRef] .

*χ*

^{2}values close to the minimum.

6. M. K. Transtrum, B. B. Machta, and J. P. Sethna, “Geometry of nonlinear least squares with applications to sloppy models and optimization,” Phys. Rev. E **83**, 036701 (2011) [CrossRef] .

*χ*

^{2}. In the theory Section 2 we present the theoretical background of the modification. In Section 3 we present the results of some test retrievals applied to MI-PAS (Michelson Interferometer for Passive Atmospheric Sounding) observations. Finally in Section 4 we draw the conclusions.

## 2. Theory

### 2.1. The continuum variables

*k*appearing in Eq. (4) are computed including only the contributions of transitions not farther than 25 cm

_{σ,m}^{−1}from

*σ*[4

4. M. Ridolfi, B. Carli, M. Carlotti, T. von Clarmann, B. M. Dinelli, A. Dudhia, J.-M. Flaud, M. Höpfner, P. E. Morris, P. Raspollini, G. Stiller, and R. J. Wells, “Optimized forward model and retrieval scheme for MIPAS near-real-time data processing,” Appl. Opt. **39**, 1323–1340 (2000) [CrossRef] .

5. P. Raspollini, C. Belotti, A. Burgess, B. Carli, M. Carlotti, S. Ceccherini, B. M. Dinelli, A. Dudhia, J.-M. Flaud, B. Funke, M. Höpfner, M. López-Puertas, V. Payne, C. Piccolo, J. J. Remedios, M. Ridolfi, and R. Spang, “MI-PAS level 2 operational analysis,” Atmos. Chem. Phys. **6**, 5605–5630 (2006) [CrossRef] .

^{−1}) [7

7. P. Raspollini, B. Carli, M. Carlotti, S. Ceccherini, A. Dehn, B.M. Dinelli, A. Dudhia, J.-M. Flaud, M. López-Puertas, F. Niro, J.J. Remedios, M. Ridolfi, H. Sembhi, L. Sgheri, and T. von Clarmann, “Ten years of MIPAS measurements with ESA Level 2 processor V6 – Part I: retrieval algorithm and diagnostics of the products,” Atmos. Meas. Tech. Discuss. **6**, 461–518 (2013) [CrossRef] .

^{−1}threshold is modeled with a frequency-independent term, the so-called

*continuum*, which is normally MW and altitude dependent. This term may also account for unmodeled contributions from aerosols or residual instrument calibration errors. In the forward model this term is usually implemented as an additional gas, with VMR equal to 1. As a consequence the columns of the continuum depend only on the observation geometry and the density of the layers. The retrieval variables model the continuum cross-section vertical profile. This profile can be approximated with a curve with nodes

*z*(the retrieval grid) and values

_{i}*k*=

_{i}*k*(

*z*). The discretization used for the radiative transfer calculation (3) may be finer than the retrieval grid. However, for each layer

_{i}*l*the continuum

*k*in that layer will depend on at most two values

_{l}*k*and

_{i}*k*

_{i}_{−1}with consecutive indeces. Thus we have where 0 ≤

*a*,

_{l,i}*b*≤ 1 are constants determined by the interpolation law, for instance a linear interpolation in pressure.

_{l,i}*C*

_{air}≥

*c*

_{l,air}for any

*l*and any line of sight. In our implementation we choose

*C*

_{air}= 10

^{25}cm

^{−2}. Then we define the new variables: so that Note that while the acceptable values for the

*k*are

_{i}*k*≥ 0, the range for the new variables is 0 ≤

_{i}*ξ*≤ 1. This is clearly an improvement because the sensitivity of the forward model to

_{i}*k*vanishes for large enough values of

_{i}*k*because of the exponential dependence. The new variables

_{i}*ξ*are polynomially connected with

_{i}*τ*, that represents the optical transparency of layer

_{l,c}*l*due to the continuum. Thus they are tightly linked to the measured spectra.

*k*continuum variables are calculated as where the summation is extended to all the layers depending on the value

_{i}*k*. Because of (6), for each

_{j}*l*there are only two non vanishing derivatives,

### 2.2. The VMR variables

*k*. On the other hand, the cross-sections

_{l}*c*.

_{l,m}*z*and values

_{i}*x*=

_{i}*x*(

*z*), then there are constants such that As in the continuum case we may define new variables

_{i}*ζ*= exp (−

_{i}*x*) for some gas-dependent constant

_{i}C_{m}*C*, so that:

_{m}## 3. Test of the method on MIPAS measurements

8. H. Fischer, M. Birk, C. Blom, B. Carli, M. Carlotti, T. von Clarmann, L. Delbouille, A. Dudhia, D. Ehhalt, M. Endemann, J. M. Flaud, R. Gessner, A. Kleinert, R. Koopman, J. Langen, M. López-Puertas, P. Mosner, H. Nett, H. Oelhaf, G. Perron, J. Remedios, M. Ridolfi, G. Stiller, and R. Zander, “MIPAS: an instrument for atmospheric and climate research,” Atmos. Chem. Phys. **8**, 2151–2188 (2008) [CrossRef] .

4. M. Ridolfi, B. Carli, M. Carlotti, T. von Clarmann, B. M. Dinelli, A. Dudhia, J.-M. Flaud, M. Höpfner, P. E. Morris, P. Raspollini, G. Stiller, and R. J. Wells, “Optimized forward model and retrieval scheme for MIPAS near-real-time data processing,” Appl. Opt. **39**, 1323–1340 (2000) [CrossRef] .

5. P. Raspollini, C. Belotti, A. Burgess, B. Carli, M. Carlotti, S. Ceccherini, B. M. Dinelli, A. Dudhia, J.-M. Flaud, B. Funke, M. Höpfner, M. López-Puertas, V. Payne, C. Piccolo, J. J. Remedios, M. Ridolfi, and R. Spang, “MI-PAS level 2 operational analysis,” Atmos. Chem. Phys. **6**, 5605–5630 (2006) [CrossRef] .

7. P. Raspollini, B. Carli, M. Carlotti, S. Ceccherini, A. Dehn, B.M. Dinelli, A. Dudhia, J.-M. Flaud, M. López-Puertas, F. Niro, J.J. Remedios, M. Ridolfi, H. Sembhi, L. Sgheri, and T. von Clarmann, “Ten years of MIPAS measurements with ESA Level 2 processor V6 – Part I: retrieval algorithm and diagnostics of the products,” Atmos. Meas. Tech. Discuss. **6**, 461–518 (2013) [CrossRef] .

7. P. Raspollini, B. Carli, M. Carlotti, S. Ceccherini, A. Dehn, B.M. Dinelli, A. Dudhia, J.-M. Flaud, M. López-Puertas, F. Niro, J.J. Remedios, M. Ridolfi, H. Sembhi, L. Sgheri, and T. von Clarmann, “Ten years of MIPAS measurements with ESA Level 2 processor V6 – Part I: retrieval algorithm and diagnostics of the products,” Atmos. Meas. Tech. Discuss. **6**, 461–518 (2013) [CrossRef] .

_{2}O, O

_{3}, HNO

_{3}, CH

_{4}, N

_{2}O, NO

_{2}, CFC-11, CFC-12, ClONO

_{2}and N

_{2}O

_{5}. The ESA operational retrieval algorithm V6.0 is based on the Gauss-Newton method, modified with the regularizing Levenberg-Marquardt (LM) technique (see [7

**6**, 461–518 (2013) [CrossRef] .

9. M. Lampton, “Damping-undamping strategies for the Levenberg-Marquardt nonlinear least- squares method,” Comput. Phys. **11**, 110–115 (1997) [CrossRef] .

- The Iterative Variable Strength (IVS) regularization technique [10, 11
10. M. Ridolfi and L. Sgheri, “A self-adapting and altitude-dependent regularization method for atmospheric profile retrievals,” Atmos. Chem. Phys.

**9**, 1883–1897 (2009) [CrossRef] .], which is already scheduled for implementation in the next release of the ESA processor.11. M. Ridolfi and L. Sgheri, “Iterative approach to self-adapting and altitude-dependent regularization for atmospheric profile retrievals,” Opt. Express ,

**19**, 26696–26709 (2011) [CrossRef] . - In addition to the nominal (NOM) version, we implemented the change of retrieval variables (as explained in Section 2), either for the continuum variables only (CONT version) or for the continuum and VMR variables (POLY version). The better conditioning of the inversion obtained in the CONT and POLY approaches permits to start the retrieval with a smaller LM damping parameter for the new retrieval variables. The advantages of this modification will be discussed in Section 3.1.

### 3.1. Tests on a set of 12 MIPAS orbits

^{17}(the inverse of the machine precision). Furthermore we observed that this conditioning was less sensitive to the LM damping parameter. Hence we were able to decrease by a factor of 10 the initial value of this parameter for the new retrieval variables. This reduction is possible also thanks to the reduced non-linearity of the forward model with respect to the new continuum variables. The smaller value of the LM damping parameter leads to larger steps in the GN sequence. As a consequence we are able to get closer to the true minimum of the cost function, with fewer iterations. We found that the results of the retrieval do not depend critically on the initial value of the LM parameter for the new variables. Thus a fine tuning of this parameter on a per-target basis would not lead to significant improvements.

_{2}VMR is retrieved only above 24 km, while the continuum parameters are fitted only below 30 km. Thus, only a small number of continuum parameters are retrieved, in an altitude range where the opacity of the atmosphere is small. Hence the sensitivity of the retrieval to continuum parameters is good also in the NOM approach. In the retrieval of N

_{2}O

_{5}, the selected MWs span a limited spectral range (approximately 27 cm

^{−1}), therefore a single continuum profile is retrieved to make the inversion better conditioned. Hence the effect of the CONT approach on these two species is small.

_{2}. The regularizing effect of the LM method in this case is weaker due to the smaller initial value of the LM damping parameter. This is confirmed by the 5% increase in

*t*-test. Given the relatively large sample size, all the improvements of the CONT_R with respect to the NOM_R method are significant with a probability threshold of 0.01, except for the aforementioned NO

_{2}and N

_{2}O

_{5}species.

**6**, 461–518 (2013) [CrossRef] .

*k*(

_{l,m}*σ*) of the retrieved gas depend on the frequency within each MW. On the other hand the cross-sections

*k*for the continuum are constant within each MW. The same argument also applies to the transparencies

_{l,c}*τ*and

_{σ,l,m}*τ*. In the NOM approach a sudden large increase of a continuum parameter may easily lead to a fully opaque layer

_{l,c}*τ*≈ 0 for all the frequencies within the involved MW. As a consequence, any emission from a layer beneath

_{l,c}*l*is blocked by the fully opaque layer. On the other hand, a large increase in the VMR parameter hardly leads to a full MW with

*τ*≈ 0, due to the dependence on

_{σ,l,m}*σ*. Also in the CONT approach we can get an opaque layer with

*τ*≈ 0. However, while in the NOM case we have

_{l,c}*k*→ ∞, in the CONT case

_{j}*ξ*→ 0, because the exponents in Eq. (9) are less than 1. Consequently, in this case the sensitivity to the continuum parameters is not lost in the CONT approach.

_{j}### 3.2. Tests on a whole month of orbits

_{2}for the CONT_R over the NOM_R approach.

_{2}O

_{5}, see bottom right panel of the figure.

## 4. Conclusions

*sloppy model*. In this case the numerical search for the minimum of the cost function may be difficult.

## Acknowledgments

## References and links

1. | T. von Clarmann, M. Hpfner, B. Funke, M. López-Puertas, A. Dudhia, V. Jay, F. Schreier, M. Ridolfi, S. Ceccherini, B. J. Kerridge, J. Reburn, R. Siddans, and J.-M. Flaud, “Modelling of atmospheric mid-infrared radiative transfer: The AMIL2DA algorithm intercomparison experiment,” J. Quant. Spectrosc. Radiat. Transfer |

2. | A. R. Curtis, “Discussion of a statistical model for water vapor absorption,” Q. J. R. Meteorol. Soc. |

3. | W. L. Godson, “The evaluation of infrared radiative fluxes due to atmospheric water vapour,” Q. J. R. Meteorol. Soc. |

4. | M. Ridolfi, B. Carli, M. Carlotti, T. von Clarmann, B. M. Dinelli, A. Dudhia, J.-M. Flaud, M. Höpfner, P. E. Morris, P. Raspollini, G. Stiller, and R. J. Wells, “Optimized forward model and retrieval scheme for MIPAS near-real-time data processing,” Appl. Opt. |

5. | P. Raspollini, C. Belotti, A. Burgess, B. Carli, M. Carlotti, S. Ceccherini, B. M. Dinelli, A. Dudhia, J.-M. Flaud, B. Funke, M. Höpfner, M. López-Puertas, V. Payne, C. Piccolo, J. J. Remedios, M. Ridolfi, and R. Spang, “MI-PAS level 2 operational analysis,” Atmos. Chem. Phys. |

6. | M. K. Transtrum, B. B. Machta, and J. P. Sethna, “Geometry of nonlinear least squares with applications to sloppy models and optimization,” Phys. Rev. E |

7. | P. Raspollini, B. Carli, M. Carlotti, S. Ceccherini, A. Dehn, B.M. Dinelli, A. Dudhia, J.-M. Flaud, M. López-Puertas, F. Niro, J.J. Remedios, M. Ridolfi, H. Sembhi, L. Sgheri, and T. von Clarmann, “Ten years of MIPAS measurements with ESA Level 2 processor V6 – Part I: retrieval algorithm and diagnostics of the products,” Atmos. Meas. Tech. Discuss. |

8. | H. Fischer, M. Birk, C. Blom, B. Carli, M. Carlotti, T. von Clarmann, L. Delbouille, A. Dudhia, D. Ehhalt, M. Endemann, J. M. Flaud, R. Gessner, A. Kleinert, R. Koopman, J. Langen, M. López-Puertas, P. Mosner, H. Nett, H. Oelhaf, G. Perron, J. Remedios, M. Ridolfi, G. Stiller, and R. Zander, “MIPAS: an instrument for atmospheric and climate research,” Atmos. Chem. Phys. |

9. | M. Lampton, “Damping-undamping strategies for the Levenberg-Marquardt nonlinear least- squares method,” Comput. Phys. |

10. | M. Ridolfi and L. Sgheri, “A self-adapting and altitude-dependent regularization method for atmospheric profile retrievals,” Atmos. Chem. Phys. |

11. | M. Ridolfi and L. Sgheri, “Iterative approach to self-adapting and altitude-dependent regularization for atmospheric profile retrievals,” Opt. Express , |

12. | P. R. Bevington and D. K. Robinson, |

13. | C. D. Rodgers, |

**OCIS Codes**

(000.3860) General : Mathematical methods in physics

(010.1280) Atmospheric and oceanic optics : Atmospheric composition

(100.3190) Image processing : Inverse problems

(280.4991) Remote sensing and sensors : Passive remote sensing

**ToC Category:**

Atmospheric and Oceanic Optics

**History**

Original Manuscript: February 8, 2013

Revised Manuscript: April 5, 2013

Manuscript Accepted: April 8, 2013

Published: May 3, 2013

**Citation**

Marco Ridolfi and Luca Sgheri, "On the choice of retrieval variables in the inversion of remotely sensed atmospheric measurements," Opt. Express **21**, 11465-11474 (2013)

http://www.opticsinfobase.org/oe/abstract.cfm?URI=oe-21-9-11465

Sort: Year | Journal | Reset

### References

- T. von Clarmann, M. Hpfner, B. Funke, M. López-Puertas, A. Dudhia, V. Jay, F. Schreier, M. Ridolfi, S. Ceccherini, B. J. Kerridge, J. Reburn, R. Siddans, and J.-M. Flaud, “Modelling of atmospheric mid-infrared radiative transfer: The AMIL2DA algorithm intercomparison experiment,” J. Quant. Spectrosc. Radiat. Transfer78(3–4), 381–407 (2003). [CrossRef]
- A. R. Curtis, “Discussion of a statistical model for water vapor absorption,” Q. J. R. Meteorol. Soc.78, 638–640 (1952). [CrossRef]
- W. L. Godson, “The evaluation of infrared radiative fluxes due to atmospheric water vapour,” Q. J. R. Meteorol. Soc.79, 367–379 (1953). [CrossRef]
- M. Ridolfi, B. Carli, M. Carlotti, T. von Clarmann, B. M. Dinelli, A. Dudhia, J.-M. Flaud, M. Höpfner, P. E. Morris, P. Raspollini, G. Stiller, and R. J. Wells, “Optimized forward model and retrieval scheme for MIPAS near-real-time data processing,” Appl. Opt.39, 1323–1340 (2000). [CrossRef]
- P. Raspollini, C. Belotti, A. Burgess, B. Carli, M. Carlotti, S. Ceccherini, B. M. Dinelli, A. Dudhia, J.-M. Flaud, B. Funke, M. Höpfner, M. López-Puertas, V. Payne, C. Piccolo, J. J. Remedios, M. Ridolfi, and R. Spang, “MI-PAS level 2 operational analysis,” Atmos. Chem. Phys.6, 5605–5630 (2006). [CrossRef]
- M. K. Transtrum, B. B. Machta, and J. P. Sethna, “Geometry of nonlinear least squares with applications to sloppy models and optimization,” Phys. Rev. E83, 036701 (2011). [CrossRef]
- P. Raspollini, B. Carli, M. Carlotti, S. Ceccherini, A. Dehn, B.M. Dinelli, A. Dudhia, J.-M. Flaud, M. López-Puertas, F. Niro, J.J. Remedios, M. Ridolfi, H. Sembhi, L. Sgheri, and T. von Clarmann, “Ten years of MIPAS measurements with ESA Level 2 processor V6 – Part I: retrieval algorithm and diagnostics of the products,” Atmos. Meas. Tech. Discuss.6, 461–518 (2013). [CrossRef]
- H. Fischer, M. Birk, C. Blom, B. Carli, M. Carlotti, T. von Clarmann, L. Delbouille, A. Dudhia, D. Ehhalt, M. Endemann, J. M. Flaud, R. Gessner, A. Kleinert, R. Koopman, J. Langen, M. López-Puertas, P. Mosner, H. Nett, H. Oelhaf, G. Perron, J. Remedios, M. Ridolfi, G. Stiller, and R. Zander, “MIPAS: an instrument for atmospheric and climate research,” Atmos. Chem. Phys.8, 2151–2188 (2008). [CrossRef]
- M. Lampton, “Damping-undamping strategies for the Levenberg-Marquardt nonlinear least- squares method,” Comput. Phys.11, 110–115 (1997). [CrossRef]
- M. Ridolfi and L. Sgheri, “A self-adapting and altitude-dependent regularization method for atmospheric profile retrievals,” Atmos. Chem. Phys.9, 1883–1897 (2009). [CrossRef]
- M. Ridolfi and L. Sgheri, “Iterative approach to self-adapting and altitude-dependent regularization for atmospheric profile retrievals,” Opt. Express, 19, 26696–26709 (2011). [CrossRef]
- P. R. Bevington and D. K. Robinson, Data reduction and error analysis for the physical sciences, 3rd ed. (McGraw–Hill, 2003).
- C. D. Rodgers, Inverse methods for atmospheric sounding: Theory and practice Atmospheric, Oceanic and Planetary Physics, (World Scientific, 2000).

## Cited By |
Alert me when this paper is cited |

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.

« Previous Article | Next Article »

OSA is a member of CrossRef.