OSA's Digital Library

Optics Express

Optics Express

  • Editor: C. Martijn de Sterke
  • Vol. 19, Iss. 27 — Dec. 19, 2011
  • pp: 26696–26709
« Show journal navigation

Iterative approach to self-adapting and altitude-dependent regularization for atmospheric profile retrievals

Marco Ridolfi and Luca Sgheri  »View Author Affiliations


Optics Express, Vol. 19, Issue 27, pp. 26696-26709 (2011)
http://dx.doi.org/10.1364/OE.19.026696


View Full Text Article

Acrobat PDF (823 KB)





Browse Journals / Lookup Meetings

Browse by Journal and Year


   


Lookup Conference Papers

Close Browse Journals / Lookup Meetings

Article Tools

Share
Citations

Abstract

In this paper we present the IVS (Iterative Variable Strength) method, an altitude-dependent, self-adapting Tikhonov regularization scheme for atmospheric profile retrievals. The method is based on a similar scheme we proposed in 2009. The new method does not need any specifically tuned minimization routine, hence it is more robust and faster. We test the self-consistency of the method using simulated observations of the Michelson Interferometer for Passive Atmospheric Sounding (MIPAS). We then compare the new method with both our previous scheme and the scalar method currently implemented in the MIPAS on-line processor, using both synthetic and real atmospheric limb measurements. The IVS method shows very good performances.

© 2011 OSA

1. Introduction

On March 1st, 2002 the European Space Agency (ESA) launched the ENVISAT satellite on a polar orbit. The satellite payload includes the Michelson Interferometer for Passive Atmospheric Sounding (MIPAS, [1

1. H. Fischer, M. Birk, C. Blom, B. Carli, M. Carlotti, T. von Clarmann, L. Delbouille, A. Dudhia, D. Ehhalt, M. Endemann, J. M. Flaud, R. Gessner, A. Kleinert, R. Koopman, J. Langen, M. López-Puertas, P. Mosner, H. Nett, H. Oelhaf, G. Perron, J. Remedios, M. Ridolfi, G. Stiller, and R. Zander, “Mipas: an instrument for atmospheric and climate research,” Atmos. Chem. Phys. 8, 2151–2188 (2008). [CrossRef]

]). MIPAS sounds the mid-infrared atmospheric limb-emission spectrum. This region contains the signatures of many atmospheric constituents. ESA set up a retrieval algorithm [2

2. M. Ridolfi, B. Carli, M. Carlotti, T. von Clarmann, B. M. Dinelli, A. Dudhia, J.-M. Flaud, M. Höpfner, P. E. Morris, P. Raspollini, G. Stiller, and R. J. Wells, “Optimized forward model and retrieval scheme for mipas near-real-time data processing,” Appl. Opt. 39, 1323–1340 (2000). [CrossRef]

, 3

3. P. Raspollini, C. Belotti, A. Burgess, B. Carli, M. Carlotti, S. Ceccherini, B. M. Dinelli, A. Dudhia, J.-M. Flaud, B. Funke, M. Höpfner, M. López-Puertas, V. Payne, C. Piccolo, J. J. Remedios, M. Ridolfi, and R. Spang, “Mipas level 2 operational analysis,” Atmos. Chem. Phys. 6, 5605–5630 (2006). [CrossRef]

] to determine pressure at tangent points and geolocated profiles of temperature and of Volume Mixing Ratios (VMR) of some atmospheric constituents, among which H2O, O3, HNO3, CH4, N2O and NO2.

The full resolution MIPAS measurements acquired until March 2004 were extensively validated by several research groups (see Atmos. Chem. Phys. 2006 special issue on MIPAS). Oscillations beyond the error bars were observed in several MIPAS profiles, particularly in CH4 and N2O VMR [4

4. S. Payan, C. Camy-Peyret, H. Oelhaf, G. Wetzel, G. Maucher, C. Keim, M. Pirre, N. Huret, A. Engel, M. C. Volk, H. Kuellmann, J. Kuttippurath, U. Cortesi, G. Bianchini, F. Mencaraglia, P. Raspollini, G. Redaelli, C. Vigouroux, M. De Mazière, S. Mikuteit, T. Blumenstock, V. Velazco, J. Notholt, E. Mahieu, P. Duchatelet, D. Smale, S. Wood, N. Jones, C. Piccolo, V. Payne, A. Bracher, N. Glatthor, G. Stiller, K. Grunow, P. Jeseck, Y. Te, and A. Butz, “Validation of version-4.61 methane and nitrous oxide observed by mipas,” Atmos. Chem. Phys. 9, 413–442 (2009). [CrossRef]

]. The optimized spectral resolution adopted by MIPAS since January 2005 (see for instance [5

5. P. Raspollini, B. Carli, S. Ceccherini, and M. Ridolfi, “The new measurement scenario of mipas,” in “ASSFTS 14 Workshop, Firenze,” (2009).

]) is characterized by a finer vertical sampling step of 1.5 km in the upper troposphere - lower stratosphere region. The field of view of the instrument is approximately 3 km in the vertical, so the atmosphere turns out to be oversampled. The retrieval grid of the ESA processor coincides with the tangent points of the limb measurements. Therefore the oversampling of the optimized resolution measurements amplifies the unphysical oscillations already present in the full resolution measurements.

The EC (error consistency) [6

6. S. Ceccherini, “Analytical determination of the regularization parameter in the retrieval of atmospheric vertical profiles,” Opt. Lett. 30, 2554–2556 (2005). [CrossRef] [PubMed]

] Tikhonov regularization scheme has been implemented in the ESA retrieval algorithm to damp the profile oscillations. The choice of the scalar, altitude-independent Tikhonov parameter, which determines the trade-off between the smoothing of the oscillations and the preservation of small-scale features, is rather conservative. This choice guarantees that small-scale profile features in the altitude domain are preserved. However, the performances of this method (as for any scalar regularization scheme) are degraded when the profile changes by several orders of magnitude in the retrieval altitude range. For this reason the regularization of water vapor profiles has been turned off in the operational ESA retrievals.

In [7

7. M. Ridolfi and L. Sgheri, “A self-adapting and altitude-dependent regularization method for atmospheric profile retrievals,” Atmos. Chem. Phys. 9, 1883–1897 (2009). [CrossRef]

] we proposed the VS (variable strength) regularization scheme, a self-adapting and altitude-dependent approach that detects whether the actual observations contain information about small-scale profile features, and determines the strength of the regularization accordingly. While representing an optimal solution, this method has two drawbacks. First, the method relies on an external minimization routine that needs to be carefully tuned to obtain optimal performances. Second, the computational requirements of the method, and specifically of the optimization routine, lead to an increase of the retrieval time of about 20%.

In this paper we propose the IVS (iterative variable strength) method, an alternative to the VS scheme. The IVS, while based on the same rationale of the VS, does not use any minimization routine. As a consequence the method is more robust, and the implementation is easier. The additional computational effort required amounts only to about 1.5% of the total retrieval time. This feature makes the IVS method suitable also for implementation in the near real-time processor of MIPAS data.

In Sections 2 and 3 we outline the theoretical background of the developed regularization scheme. In Section 4 we highlight the aspects specific to our implementation of the proposed method. In Sections 5 and 6 we present some tests illustrating the self-consistency of the method when synthetic measurements are used. In Section 7 we show results from real measurements. Finally in Section 8 we draw the conclusions.

For the readers’ convenience, we report here a list of acronyms used in the text to specify various techniques. This list supplements the definitions given in the text.

  • LS - The Least Squares solution, see equation (2).
  • OE - The Optimal Estimation solution, see equation (3).
  • LM - The Levenberg-Marquardt solution, see equation (3) and subsequent explanations.
  • EC - The solution regularized with the Error Consistency method, see [6

    6. S. Ceccherini, “Analytical determination of the regularization parameter in the retrieval of atmospheric vertical profiles,” Opt. Lett. 30, 2554–2556 (2005). [CrossRef] [PubMed]

    ].
  • VS - The solution regularized with the Variable Strength method, see Eq. (11) and [7

    7. M. Ridolfi and L. Sgheri, “A self-adapting and altitude-dependent regularization method for atmospheric profile retrievals,” Atmos. Chem. Phys. 9, 1883–1897 (2009). [CrossRef]

    ].
  • IVS - The solution regularized with the Iterative Variable Strength method, the new approach introduced in this paper, see Section 3.

2. Theory

Tikhonov regularization is often used to improve the conditioning of atmospheric profile inversion. Smoother profiles are obtained by penalizing the oscillating solutions in the inversion formula. Let y = f(x) be the forward problem, where y is the m–dimensional vector of the observations with error covariance matrix Sy, f is the forward model, function of the n–dimensional atmospheric state vector x, whose components represent the unknown profile at altitudes z = zj, j = 1,...,n. The Tikhonov solution is the state vector xt minimizing the following cost function:
ξ2=(yf(x))TSy1(yf(x))+(xsx)TLTΛL(xsx).
(1)

The first term of the right side of Eq. (1) is referred as χ2 and represents the cost function minimized in the least–squares (LS) approach. The vector xs is an a-priori estimate of the solution, L is a h×n matrix operator, usually approximating a linear combination of the ith order vertical derivatives (i = 0,1,2). The h×h matrix Λ is diagonal, positive semi-definite and drives the strength of the regularization. Note that normally hn. Assuming that (Lx)ji=02cidixdzi|z=z˜j, we may think of Λjj as the regularization strength at z = j, j = 1,...,h. The altitudes j are determined by the choice of L, and may not necessarily coincide with the retrieval grid zj, j = 1,...,n. Thus we may speak of a vertical profile of Λ.

The standard scalar Tikhonov regularization is obtained when Λ = λI. There are many different methods to determine λ, a good review may be found in [8

8. A. Doicu, T. Trautmann, and F. Schreier, Numerical regularization for atmospheric inverse problems (Springer, 2010). [CrossRef]

]. The variable regularization of atmospheric profiles has been proposed in [9

9. S. S. Kulawik, G. Osterman, D. B. A. Jones, and K. W. Bowman, “Calculation of altitude-dependent tikhonov constraints for tes nadir retrievals,” IEEE Trans. Geosci. Remote Sens. 44, 1334–1342 (2006). [CrossRef]

], where combinations of 0th, 1st, and 2nd order Tikhonov constraints were considered. The method was applied to nadir retrievals from the Tropospheric Emission Spectrometer (TES) in [10

10. K. W. Bowman, C. Rodgers, S. S. Kulawik, J. Worden, E. Sarkissian, G. Osterman, T. Steck, M. Lou, A. Eldering, M. Shephard, H. Worden, M. Lampel, S. Clough, P. Brown, C. Rinsland, M. Gunson, and R. Beer, “Tropospheric emission spectrometer: Retrieval method and error analysis,” IEEE Trans. Geosci. Remote Sens. 44, 1297–1307 (2006). [CrossRef]

]. In [11

11. J. Steinwagner and G. Schwarz, “Shape-dependent regularization for the retrieval of atmospheric state parameter profiles,” Appl. Opt. 45, 1000–1009 (2006). [CrossRef] [PubMed]

], Λ = λSh and Sh is a diagonal matrix containing the reciprocal of the a-priori estimation of the profile. The VS method was introduced and tested on MIPAS measurements in [7

7. M. Ridolfi and L. Sgheri, “A self-adapting and altitude-dependent regularization method for atmospheric profile retrievals,” Atmos. Chem. Phys. 9, 1883–1897 (2009). [CrossRef]

], where a comprehensive comparison of scalar and altitude-dependent regularization schemes may also be found.

In this paper we test an altitude-dependent regularization method determining a profile of Λ as the result of an iterative process. We choose to apply the regularization a-posteriori to obtain a better convergence rate, as reported in [12

12. S. Ceccherini, C. Belotti, B. Carli, P. Raspollini, and M. Ridolfi, “Technical note: Regularization performances with the error consistency method in the case of retrieved atmospheric profiles,” Atmos. Chem. Phys. 7, 1435–1440 (2007). [CrossRef]

] and [7

7. M. Ridolfi and L. Sgheri, “A self-adapting and altitude-dependent regularization method for atmospheric profile retrievals,” Atmos. Chem. Phys. 9, 1883–1897 (2009). [CrossRef]

]. In other words, we first find the minimum of the χ2 via a Gauss-Newton iterative method:
xp+1=xp+(KpTSy1Kp)1[KpTSy1(yf(xp))],
(2)
where p is the iteration count and Kp is the m × n Jacobian matrix of f in xp. Let k be the iteration count at convergence, and xLSxk+1 the LS solution. The covariance matrix of xLS is SLS=(KkTSy1Kk)1.

The Gauss-Newton scheme of Eq. (2) may fail if for some pk the matrix KpTSy1Kp is singular or too much ill–conditioned to be inverted with satisfactory accuracy. In this case, the optimal estimation (OE) and/or Levenberg–Marquardt (LM) modifications (see e.g. [13

13. C. D. Rodgers, Inverse Methods for Atmospheric Sounding: Theory and Practice (Atmospheric, Oceanic and Planetary Physics, World Scientific, 2000). [CrossRef]

]) can be used. For example, if OE is used, Eq. (2) becomes:
xp+1=xp+(KpTSy1Kp+Sa1)1[KpTSy1(yf(xp))+Sa1(xaxp)],
(3)
where xa is an a-priori estimate of the profile x with covariance matrix Sa. The LM solution with damping factor α can also be represented with Eq. (3), by setting xa = xp and Sa1=αI or Sa1=αdiag(KpTSy1Kp). We recall here that for any square matrix M, diag(M) is a matrix having the same elements as M on the main diagonal and 0 elsewhere. With the OE/LM modifications the matrix to be inverted is KpTSy1Kp+Sa1. Since we want to use Tikhonov regularization, the only purpose of Sa1 is to permit the inversion of KpTSy1Kp+Sa1 with reasonably small numerical errors. Therefore this term should be chosen positive definite, diagonal or diagonally dominant and kept as small as possible.

Let k be the iteration count at convergence, thus xOExk+1, the unregularized solution. Let AOE be the averaging kernel of xOE and SOE its measurement error covariance matrix. The ESA retrieval algorithm uses the LM solution, and AOE and SOE can be calculated with alternative algorithms of different sophistication as explained in [12

12. S. Ceccherini, C. Belotti, B. Carli, P. Raspollini, and M. Ridolfi, “Technical note: Regularization performances with the error consistency method in the case of retrieved atmospheric profiles,” Atmos. Chem. Phys. 7, 1435–1440 (2007). [CrossRef]

] and [14

14. S. Ceccherini and M. Ridolfi, “Technical note: Variance-covariance matrix and averaging kernels for the levenberg-marquardt solution of the retrieval of atmospheric vertical profiles,” Atmos. Chem. Phys. 10, 3131–3139 (2010). [CrossRef]

]. The theory explained hereafter does not depend on the method used to determine AOE and SOE.

We can compute the regularized solution xΛ as the Gauss-Newton iterate for the minimization of:
ζ2=(yf(x))TSy1(yf(x))++(xax)TSa1(xax)+(xsx)TLTΛL(xsx),
(4)
starting from xk. Thus xΛ is given by:
xΛ=xk+(KkTSy1Kk+Sa1+LTΛL)1.[KkTSy1(yf(xk))+Sa1(xaxk)+LTΛL(xsxk)].
(5)
Both xa and xs are estimates of the solution, however usually xa constrains the values of the profile, while xs constrains its derivatives. Therefore two different symbols are used. If we extract the term KkTSy1(yf(xk))+Sa1(xaxk) from Eq. (3) and plug it in Eq. (5) we obtain:
xΛ=(KkTSy1Kk+Sa1+LTΛL)1[(KkTSy1Kk+Sa1)xOE+LTΛLxs].
(6)
Let D=(KkTSy1Kk+Sa1+LTΛL)1(KkTSy1Kk+Sa1), then the averaging kernel AΛ and the measurement error covariance matrix SΛ of xΛ can be written as:
AΛ=DAOE,
(7)
SΛ=DSOEDT.
(8)
Since in practical cases it is always difficult to find reliable a-priori profile estimates, in this paper we always select xs = 0. With this choice Eq. (6) can be simplified as:
xΛ=DxOE.
(9)
Vertical resolution is a measure of the dispersion of the signal, usually calculated via the averaging kernel AΛ. There are many practical ways of estimating the vertical resolution, we use:
νi=j=1n(AΛ)ij(zj1zj+1)2|AΛ|ii
(10)
where zj, j = 1,..., n are the altitudes, and z0 = z1 +(z1z2), zn+1 = zn +(znzn−1). Formula (10) is a variation of the full width half height (FWHH) estimation proposed in [13

13. C. D. Rodgers, Inverse Methods for Atmospheric Sounding: Theory and Practice (Atmospheric, Oceanic and Planetary Physics, World Scientific, 2000). [CrossRef]

]. We used |AΛ|ii in order to penalize the negative lobes of the averaging kernel. Note that rows of the averaging kernel not peaking at the diagonal element are penalized by Eq. (10), which in this case provides an overestimate of the FWHH. When AΛ = I, Eq. (10) provides the vertical step Δzi = (zi−1zi+1)/2 of the retrieval grid.

3. Altitude-dependent regularization

In the VS method [7

7. M. Ridolfi and L. Sgheri, “A self-adapting and altitude-dependent regularization method for atmospheric profile retrievals,” Atmos. Chem. Phys. 9, 1883–1897 (2009). [CrossRef]

] the Λ-profile is determined as the minimizer of the following target function:
ψVS(Λ)=1xΛ¯j=1n(SΛ)jj+(χ2(xΛ)χ2(xOE)nwe2)+++1Δz¯j=1n[(νj(xΛ)wrΔzj)+]2
(11)
where the bar over a vector stands for the average of the vector elements, and a superscript + stands for the positive part of a function. The constants we and wr are tunable parameters. The first term of formula (11) represents the error of the regularized profile. The other terms are penalization terms which take effect if the regularized profile is not compatible with the observations (second term), or if the vertical resolution is degraded beyond a pre-defined margin (third term). Loosely speaking, the minimum of (11) is obtained for the largest possible Λ-profile that keeps small enough the penalization terms.

The same objectives of the VS method can be pursued with an iterative approach, avoiding the expensive minimization of ψVS. In the IVS approach we define a Λ-profile λ(z) on a vertical grid so fine that we can consider it a continuous function. We start with a large λ(0)(z) = λmax constant profile and decrease it iteratively until the following requirements are fulfilled:
(xΛ(l)xOE)TSOE1(xΛ(l)xOE)wen
(12)
νj(xΛ(l))Δzjwr,j=1,,n
(13)
where l is the iteration count and xΛ(l) is calculated from Eq. (9) with Λjj = λ(l)(j), j = 1,...,h. Condition (12) ensures that, on average, the regularized profile lies within a fraction we of the error bars of the unregularized profile. It pursues the same objective of the second term of ψVS(Λ), as shown by Eq. (13) of [7

7. M. Ridolfi and L. Sgheri, “A self-adapting and altitude-dependent regularization method for atmospheric profile retrievals,” Atmos. Chem. Phys. 9, 1883–1897 (2009). [CrossRef]

]. Condition (13) is equivalent to the third term of ψVS(Λ).

Fix a threshold λmin. Let J ⊂ {1,...,n} the set of indices of the altitudes zj for which λ(l)(zj) > λmin and:
|(xΛ(l))j(xOE)j|>we(SOE)jjor:
(14)
νj(xΛ(l))>wrΔzj.
(15)
If the requirements (12–13) are not met, J is not empty and we decrease λ(l)(z). The decreased profile λ(l+1)(z) is calculated as:
λ(l+1)(z)=[jJT(zzj,δj)]λ(l)(z),
(16)
where T is the triangular shaped function:
T(z,δ)={1,if|z|>δr+1rδ|z|,if|z|δ,
(17)
and 0 < r < 1, δj > 0 are constants. The parameter r drives the speed of the attenuation of the Λ-profile, in our implementation we used r = 0.99. Furthermore we set δj = 3Δzj on the basis of the following considerations. The xΛ profile is obtained from xOE via Eq. (9). For any standard choice of L, LTΛL is at most a pentadiagonal matrix, i.e. a matrix having non-zero elements (i, j) only for |ij| 2. Moreover the matrix KkTSy1Kk+Sa1 is diagonally dominant, therefore the influence of Λjj is mostly localized in the altitude range zj ∈ [zj – 3Δzj, zj + 3Δzj]. Note that λ(l+1)(z) < λ(l)(z) if and only if z ∈ (zjδj, zj + δj) for some jJ.

4. Implementation details

We implemented the IVS regularization method in the Optimized Retrieval Model (ORM, see [2

2. M. Ridolfi, B. Carli, M. Carlotti, T. von Clarmann, B. M. Dinelli, A. Dudhia, J.-M. Flaud, M. Höpfner, P. E. Morris, P. Raspollini, G. Stiller, and R. J. Wells, “Optimized forward model and retrieval scheme for mipas near-real-time data processing,” Appl. Opt. 39, 1323–1340 (2000). [CrossRef]

, 3

3. P. Raspollini, C. Belotti, A. Burgess, B. Carli, M. Carlotti, S. Ceccherini, B. M. Dinelli, A. Dudhia, J.-M. Flaud, B. Funke, M. Höpfner, M. López-Puertas, V. Payne, C. Piccolo, J. J. Remedios, M. Ridolfi, and R. Spang, “Mipas level 2 operational analysis,” Atmos. Chem. Phys. 6, 5605–5630 (2006). [CrossRef]

]), the scientific prototype of the retrieval algorithm used by ESA for near real–time inversion of MIPAS data. In this code the LM modification is applied to the Gauss-Newton minimization with Sa1=αdiag(KpTSy1Kp), and the EC and VS regularization methods are also implemented. The regularization method is selectable via a switch. In the IVS implementation we used h = n – 2, with the operator L defined so that:
(Lx)j1=2(xj+1xj)/(zj+1zj)(xjxj1)/(zjzj1)zj+1zj1,j=2,,n1.
(18)
Let j−1 = (zj−1 + 2zj + zj+1)/4, j = 2,...,n – 1. Thus the definition of L given by Eq. (18) corresponds to (Lx)jd2xdz2|z=z˜j, j = 1,...,h, i.e. to the second derivative operator L2. In the case of a constant Δzj we have j−1 = zj, j = 2,..., n − 1.

We chose the thresholds λmin = 10−2, and λmax = 10. A notable exception is the regularization of H2O profiles where we set λmax = 103, and implemented a damping scheme which rapidly reduces λ(0)(z) from λmax to λmin below the tropopause. The aim of this modification is to avoid introducing a systematic bias in the H2O profile in this altitude range. Any regularization in this region, even if within the error bars, would introduce a bias around or below the knee of the profile. This effect is due either to the large second derivative values necessary to model the profile knee around the tropopause (when L = L2) or to the large values of the profile and its first derivative below the tropopause (when L = L0 or L = L1).

Unlike the IVS case, the VS computation load depends heavily on the number of grid points used to represent the Λ-profile. For this reason, the Λ-profile of the VS method was defined in our tests on a coarse grid of 9 points and then interpolated to the grid j, j = 1,..., n. Finally we set (we, wr) = (1, 5) for all retrieval targets, according to the results obtained when testing the VS method on real observations [7

7. M. Ridolfi and L. Sgheri, “A self-adapting and altitude-dependent regularization method for atmospheric profile retrievals,” Atmos. Chem. Phys. 9, 1883–1897 (2009). [CrossRef]

]. The same values were used for both VS and IVS, since the parameters have the same physical meaning in both methods. Finally note that the EC method is implemented in the ORM with L = L1.

5. Self-consistency test with synthetic observations from a single limb scan

In this section we test the self–consistency of the IVS method and its capability to preserve sharp profile features measured by the instrument. We repeated the test O3 retrieval based on synthetic observations already used to assess the VS method [7

7. M. Ridolfi and L. Sgheri, “A self-adapting and altitude-dependent regularization method for atmospheric profile retrievals,” Atmos. Chem. Phys. 9, 1883–1897 (2009). [CrossRef]

]. To avoid the forward model error, these observations were generated by the forward model included in the ORM, using the reference atmosphere model of [15

15. J. J. Remedios, R. J. Leigh, A. M. Waterfall, D. P. Moore, H. Sembhi, I. Parkes, J. Greenhough, M. P. Chipperfield, and D. Hauglustaine, “Mipas reference atmospheres and comparisons to v4.61/v4.62 mipas level 2 geophysical data sets,” Atmos. Chem. Phys. Discuss. 7, 9973–10017 (2007). [CrossRef]

]. The O3 profile was modified with a sharp bump in the 18–24 km altitude range, reflecting the double–peak sometimes observed in the real O3 profiles in pre ozone hole conditions (see [16

16. A. V. Nemuc and R. L. Dezafra, “Ground based measurements of stratospheric ozone in antarctica,” Rom. Rep. Phys. 57, 445–452 (2005).

]).

Instrument configuration, including field of view, vertical scan pattern and spectral line–shape, was adjusted to the MIPAS optimized resolution scenario [17

17. A. Dudhia, “Mipas-related section of the web site of the Oxford university, department of physics,” http://www.atm.ox.ac.uk/group/mipas/ (2008).

]. Spectral measurement noise was added to synthetic observations. For altitudes ≤ 40 km the noise was chosen consistent with MIPAS specifications; for altitudes > 40 km the noise was amplified by a factor 20 in order to obtain amplified oscillations in the unregularized retrieved profile. The initial guess profile was obtained by multiplying the climatological profile by a factor of 1.3 and with no bump modification. In this test the LM term was kept as small as possible (α = 10−3) in order to limit its regularization effect. Because of the artificially amplified errors, the LM term is however necessary to guarantee the convergence of the minimization sequence.

Figure 1 illustrates the results of the test. Panel (a) shows the reference profile (solid gray), the initial guess profile (dashed black), and the LM solution (solid blue). The same panel shows also the solutions regularized with the VS (solid green), EC (solid orange) and IVS (solid purple) methods. Panel (b) shows the LM retrieval errors calculated as the square roots of the diagonal elements of the covariance matrix SOE (dashed blue). The same panel shows also the differences between the LM (solid blue), VS (solid green), EC (solid orange), IVS (solid purple) solutions and the reference profile, i.e. the actual errors.

Fig. 1 Simulated O3 retrieval with amplified noise above 40 km and artificial bump added from 18 to 24 km: (a) Reference, initial guess and retrieved profiles; (b) estimated LM retrieval error and actual differences between retrieved and reference profiles.

From Fig. 1 we see that all the considered regularization methods are able to preserve the sharp bump feature of the reference profile in the 18–24 km altitude range. Above 40 km however, the artificial oscillations induced by the large noise are better smoothed out by the VS and IVS methods. In fact, while all the presented methods are self-adaptive (i.e. they adjust the regularization strength on the basis of the error bars of the unregularized profile), the altitude-dependence of the VS and IVS methods permits to achieve a stronger regularization localized in the altitude range where the error bars of the unregularized profile are particularly large (above 40 km). From panel (b) we see that on average the VS and IVS profiles are consistent with the LM profile within a fraction we = 1 of the LM error bars as required by the algorithms. Note that the relatively large errors obtained in this test retrieval are mainly due to the artificial amplification of the measurement noise that we applied above 40 km. Therefore the results of this test, while useful to assess the consistency of the IVS method, should not be considered as representative of the real MIPAS performance.

6. Results of retrievals from a simulated full MIPAS orbit

In this section we show the results of the IVS method applied to a full orbit of synthetic MIPAS measurements. Since inspection of a large number of profiles is unpractical, we introduce a few scalar quantifiers to characterize the performance of the retrieval.

Any regularization scheme with strength Λ aims at a reduction of Ω̄2, at the expenses of an increase of χ¯R2. To measure the efficiency of a regularization scheme, we introduce the quantifier
E=Ω¯2(Λ=0)χ¯R2(Λ=0)Ω2(Λ)χ¯R2(Λ).
(20)
In this expression the quantities in the numerator refer to the unregularized solution, while those in the denominator refer to the regularized solution. Of course this expression is meaningful only for relatively small increases of χ¯R2. We use E only as an additional parameter to evaluate the quality of a regularization scheme.

Finally we consider the number of degrees of freedom (DoF) of the retrieval, calculated as the trace of the averaging kernel matrix [13

13. C. D. Rodgers, Inverse Methods for Atmospheric Sounding: Theory and Practice (Atmospheric, Oceanic and Planetary Physics, World Scientific, 2000). [CrossRef]

]. This quantifier represents the overall reduction of the vertical resolution as a consequence of the applied constraints. We divide this number by the number n of points of the retrieved profile, since this latter can vary from scan to scan due to cloud contamination. We then take the arithmetic mean DoF/n¯ on the orbit. Note that 1DoF/n¯ measures the fraction of the vertical resolution lost (with respect to the LS solution) because of external constraints. Therefore 1DoF/n¯ is also an estimator of the overall strength of the combined effect of the LM technique and the a-posteriori regularization.

Since in this test the true atmosphere is known, we can also characterize the error of the retrieved profiles with the mean ( Δx¯) and the standard deviation (σ) of the difference between the retrieved and the true profile at each retrieval grid point along the orbit.

In this section we still use synthetic MIPAS measurements generated assuming the climatological atmosphere of [15

15. J. J. Remedios, R. J. Leigh, A. M. Waterfall, D. P. Moore, H. Sembhi, I. Parkes, J. Greenhough, M. P. Chipperfield, and D. Hauglustaine, “Mipas reference atmospheres and comparisons to v4.61/v4.62 mipas level 2 geophysical data sets,” Atmos. Chem. Phys. Discuss. 7, 9973–10017 (2007). [CrossRef]

]. We however apply an interpolation within the tabulated latitude bands to avoid latitude intervals with constant atmosphere and large jumps at the edges. In this test retrieval errors due to interfering species and pressure and temperature error propagation in VMR are avoided by assuming the true atmospheric state except for the retrieval target. For the retrieval target the initial guess is set equal to the true profile multiplied by 1.3. We used the same convergence criteria as the ESA Level 2 Processor version 6.0. These consist in checking the inter-iteration variation of the state-vector and of χ2. The thresholds were adjusted to keep the convergence error smaller than 10% of the error due to the measurement noise. A more comprehensive description of the convergence criteria may be found in the technical note [19

19. M. Ridolfi and L. Sgheri, “Improvement of the orm convergence criteria: test of performance and implementation details,” http://www.fci.unibo.it/~ridolfi/hak/tnconvcrit.pdf (2011).

].

In Table 1 we report Δx¯ and σ for each target parameter and retrieval method. For reference, we also report the standard deviation of the LS profiles. Since the pure LS method does not always converge, we calculate the σ of the LS profiles (σLS) as the average of (SLS)jj over the altitudes zj and the scans of the orbit.

Table 1. Summary of retrieval errors from a full orbit of synthetic MIPAS measurements. Average ( Δx¯) and standard deviation (σ) of the differences between retrieved and true profile (K for temperature and ppmv for VMR). The standard deviation of the LS profile is estimated from the diagonal of the SLS matrix, and is reported for reference.

table-icon
View This Table
| View All Tables

To check the altitude behavior of the errors, we also broke down the differences between retrieved and true profiles into altitude bins centered around nominal MIPAS tangent altitudes. For each bin we calculated the mean and the standard deviation of the sample. The plot of Fig. 2 shows the altitude behavior of the standard deviations for H2O retrieval. The σ of the LS method (σLS) is calculated by binning (SLS)jj. Figure 3 shows the retrieved H2O profiles for a sample scan.

Fig. 2 Water vapor retrieval with LM, VS and IVS methods. Standard deviations of the difference between the retrieved and true profiles. The differences are binned around nominal MIPAS tangent altitudes.
Fig. 3 Water vapor retrieval with LM, VS and IVS methods. Initial guess, reference and regularized profiles for a typical scan of simulated orbit 15451.

In Table 2 we report, for each target species and retrieval method, the values of χ¯R2, Ω̄2, E and DoF/n¯, except for the reference (REF) profile, for which only the Ω̄2 is defined. The last row of the table contains the efficiency E averaged over the retrieval targets.

Table 2. Summary of retrieval performances for simulated MIPAS measurements of orbit 15451.

table-icon
View This Table
| View All Tables

From Table 2 we see that a good reduction of the Ω̄2 is achieved at the expenses of a marginal increase of the χ¯R2. Because of their altitude-dependent strength, the VS and IVS methods have a better efficiency in reducing the oscillations of the profiles. Despite the theoretical suboptimality, the IVS achieves comparable performances with respect to the VS method. The only marginal increase of the efficiency in the water vapor case is motivated by the fact that here, the Ω̄2 quantifier is dominated by the contributions of the steep part of the profile below the tropopause, where the profile is only marginally regularized because of the damping we applied to λ(0)(z). Nevertheless the profile oscillations in the stratosphere are well smoothed out by the VS and IVS methods as illustrated in Fig. 3, even if this effect is not properly factored by Ω̄2.

7. Retrieval from real MIPAS data

In order to verify that the good performance of the IVS method persists also when real MIPAS measurements are processed, in this section we report the results of a test based on real observations. We selected 12 orbits of MIPAS measurements from the year 2007. The measurements cover the four seasons to ensure that the test includes sufficient atmospheric variability. Table 3 shows the results of the test, with the same format of Table 2. In this case the reported quantities are averages on all the scans (1128 in total) of the considered orbits. Note that the H2O retrieval in the EC case has been carried out with the LM method. The difference in the H2O results with respect to the full LM case is mainly due to the method used to retrieve the temperature.

Table 3. Summary of retrieval performances for real MIPAS measurements of a sample of 12 orbits.

table-icon
View This Table
| View All Tables

We may compare the results of the full orbit retrieval from synthetic (Table 2) and real (Table 3) measurements. The numbers reported in Table 3 show clearly that, globally, the performance of the regularization is preserved also when real measurements are processed. We note that the Ω̄2 of the LM retrieved profiles is generally smaller in the synthetic case. This is due to the combination of two causes. First, synthetic measurements do not include systematic model errors which are present in the real observations. Second, the reference model atmosphere of the synthetic test retrieval is probably smoother than the actual atmosphere sounded by MIPAS in the selected orbits. The only exception is H2O. In this case the presence of clouds in the sample of real measurements makes the average bottom altitude of the profiles higher than in case of orbit 15451, used to model the synthetic measurements. As already mentioned, in the case of H2O the Ω̄2 quantifier is dominated by the contributions of the profile below the tropopause, hence the smaller value obtained with real measurements.

8. Conclusions

In this paper we present a new approach to the variable strength (VS) Tikhonov regularization that we already proposed in a earlier paper [7

7. M. Ridolfi and L. Sgheri, “A self-adapting and altitude-dependent regularization method for atmospheric profile retrievals,” Atmos. Chem. Phys. 9, 1883–1897 (2009). [CrossRef]

]. The new approach avoids the two drawbacks of the VS method: the dependence on an external specifically tuned minimization routine and the increase of computing time by about 20%.

The new method we propose, the iterative variable strength (IVS), has the same rationale of the VS method. The aim is to apply the strongest possible regularization that, within pre-defined margins, preserves the compatibility of the regularized profile with the observations (only a small increase of chi-square is permitted) and does not destroy the profile vertical resolution. Instead of minimizing a target function, the IVS method starts with a large (strong) Tikhonov constraint at all altitudes, then decreases it locally and iteratively, until the user requirements on chi-square and vertical resolution are met.

We prove the self-consistency of the proposed algorithm on the basis of synthetic limb-scanning observations. In particular we show that the IVS method is able to preserve sharp profile features, such as ozone double peaks, that might be unexpected and therefore not modeled in the initial guess of the retrieval. Based on the analysis of a statistically significant set of synthetic observations we also quantify the bias and the smoothing error introduced by the regularization. It turns out that the bias is one order of magnitude smaller than the noise error. Furthermore, in total, the smoothing and the noise error components of the regularized profile never exceed the noise error of the unregularized solution.

Compared to the VS, the IVS method does not rely on external minimization routines, therefore the method is more robust, its implementation is easier and the required computational effort is much smaller. For the tests presented in this paper the additional computing time required by the IVS method amounts to only 1.5% of the total retrieval time. The proposed method can be implemented in any Gauss-Newton-type algorithm for the retrieval of vertical distribution profiles. Currently the IVS algorithm is coded in a standard FORTRAN routine both in a stand-alone version and in a version integrated within the ORM code. The routine can be easily interfaced with any existing inversion software. The authors will freely supply the IVS routine to scientists who would like to test the algorithm in their inversion codes, for no-profit purposes.

Acknowledgments

We thank the IFAC-CNR institute in Firenze for making available computing and technical facilities through associate contract to M.R., in the frame of the research activity TA.P06.002. This study was supported by the ESA-ESRIN contract 21719/08/I-OL.

References and links

1.

H. Fischer, M. Birk, C. Blom, B. Carli, M. Carlotti, T. von Clarmann, L. Delbouille, A. Dudhia, D. Ehhalt, M. Endemann, J. M. Flaud, R. Gessner, A. Kleinert, R. Koopman, J. Langen, M. López-Puertas, P. Mosner, H. Nett, H. Oelhaf, G. Perron, J. Remedios, M. Ridolfi, G. Stiller, and R. Zander, “Mipas: an instrument for atmospheric and climate research,” Atmos. Chem. Phys. 8, 2151–2188 (2008). [CrossRef]

2.

M. Ridolfi, B. Carli, M. Carlotti, T. von Clarmann, B. M. Dinelli, A. Dudhia, J.-M. Flaud, M. Höpfner, P. E. Morris, P. Raspollini, G. Stiller, and R. J. Wells, “Optimized forward model and retrieval scheme for mipas near-real-time data processing,” Appl. Opt. 39, 1323–1340 (2000). [CrossRef]

3.

P. Raspollini, C. Belotti, A. Burgess, B. Carli, M. Carlotti, S. Ceccherini, B. M. Dinelli, A. Dudhia, J.-M. Flaud, B. Funke, M. Höpfner, M. López-Puertas, V. Payne, C. Piccolo, J. J. Remedios, M. Ridolfi, and R. Spang, “Mipas level 2 operational analysis,” Atmos. Chem. Phys. 6, 5605–5630 (2006). [CrossRef]

4.

S. Payan, C. Camy-Peyret, H. Oelhaf, G. Wetzel, G. Maucher, C. Keim, M. Pirre, N. Huret, A. Engel, M. C. Volk, H. Kuellmann, J. Kuttippurath, U. Cortesi, G. Bianchini, F. Mencaraglia, P. Raspollini, G. Redaelli, C. Vigouroux, M. De Mazière, S. Mikuteit, T. Blumenstock, V. Velazco, J. Notholt, E. Mahieu, P. Duchatelet, D. Smale, S. Wood, N. Jones, C. Piccolo, V. Payne, A. Bracher, N. Glatthor, G. Stiller, K. Grunow, P. Jeseck, Y. Te, and A. Butz, “Validation of version-4.61 methane and nitrous oxide observed by mipas,” Atmos. Chem. Phys. 9, 413–442 (2009). [CrossRef]

5.

P. Raspollini, B. Carli, S. Ceccherini, and M. Ridolfi, “The new measurement scenario of mipas,” in “ASSFTS 14 Workshop, Firenze,” (2009).

6.

S. Ceccherini, “Analytical determination of the regularization parameter in the retrieval of atmospheric vertical profiles,” Opt. Lett. 30, 2554–2556 (2005). [CrossRef] [PubMed]

7.

M. Ridolfi and L. Sgheri, “A self-adapting and altitude-dependent regularization method for atmospheric profile retrievals,” Atmos. Chem. Phys. 9, 1883–1897 (2009). [CrossRef]

8.

A. Doicu, T. Trautmann, and F. Schreier, Numerical regularization for atmospheric inverse problems (Springer, 2010). [CrossRef]

9.

S. S. Kulawik, G. Osterman, D. B. A. Jones, and K. W. Bowman, “Calculation of altitude-dependent tikhonov constraints for tes nadir retrievals,” IEEE Trans. Geosci. Remote Sens. 44, 1334–1342 (2006). [CrossRef]

10.

K. W. Bowman, C. Rodgers, S. S. Kulawik, J. Worden, E. Sarkissian, G. Osterman, T. Steck, M. Lou, A. Eldering, M. Shephard, H. Worden, M. Lampel, S. Clough, P. Brown, C. Rinsland, M. Gunson, and R. Beer, “Tropospheric emission spectrometer: Retrieval method and error analysis,” IEEE Trans. Geosci. Remote Sens. 44, 1297–1307 (2006). [CrossRef]

11.

J. Steinwagner and G. Schwarz, “Shape-dependent regularization for the retrieval of atmospheric state parameter profiles,” Appl. Opt. 45, 1000–1009 (2006). [CrossRef] [PubMed]

12.

S. Ceccherini, C. Belotti, B. Carli, P. Raspollini, and M. Ridolfi, “Technical note: Regularization performances with the error consistency method in the case of retrieved atmospheric profiles,” Atmos. Chem. Phys. 7, 1435–1440 (2007). [CrossRef]

13.

C. D. Rodgers, Inverse Methods for Atmospheric Sounding: Theory and Practice (Atmospheric, Oceanic and Planetary Physics, World Scientific, 2000). [CrossRef]

14.

S. Ceccherini and M. Ridolfi, “Technical note: Variance-covariance matrix and averaging kernels for the levenberg-marquardt solution of the retrieval of atmospheric vertical profiles,” Atmos. Chem. Phys. 10, 3131–3139 (2010). [CrossRef]

15.

J. J. Remedios, R. J. Leigh, A. M. Waterfall, D. P. Moore, H. Sembhi, I. Parkes, J. Greenhough, M. P. Chipperfield, and D. Hauglustaine, “Mipas reference atmospheres and comparisons to v4.61/v4.62 mipas level 2 geophysical data sets,” Atmos. Chem. Phys. Discuss. 7, 9973–10017 (2007). [CrossRef]

16.

A. V. Nemuc and R. L. Dezafra, “Ground based measurements of stratospheric ozone in antarctica,” Rom. Rep. Phys. 57, 445–452 (2005).

17.

A. Dudhia, “Mipas-related section of the web site of the Oxford university, department of physics,” http://www.atm.ox.ac.uk/group/mipas/ (2008).

18.

P. R. Bevington and D. K. Robinson, Data Reduction and Error Analysis for the Physical Sciences, 3rd ed. (McGraw–Hill, 2003).

19.

M. Ridolfi and L. Sgheri, “Improvement of the orm convergence criteria: test of performance and implementation details,” http://www.fci.unibo.it/~ridolfi/hak/tnconvcrit.pdf (2011).

OCIS Codes
(000.3860) General : Mathematical methods in physics
(010.1280) Atmospheric and oceanic optics : Atmospheric composition
(100.3190) Image processing : Inverse problems
(280.4991) Remote sensing and sensors : Passive remote sensing

ToC Category:
Atmospheric and Oceanic Optics

History
Original Manuscript: September 14, 2011
Revised Manuscript: November 3, 2011
Manuscript Accepted: November 3, 2011
Published: December 14, 2011

Citation
Marco Ridolfi and Luca Sgheri, "Iterative approach to self-adapting and altitude-dependent regularization for atmospheric profile retrievals," Opt. Express 19, 26696-26709 (2011)
http://www.opticsinfobase.org/oe/abstract.cfm?URI=oe-19-27-26696


Sort:  Author  |  Year  |  Journal  |  Reset  

References

  1. H. Fischer, M. Birk, C. Blom, B. Carli, M. Carlotti, T. von Clarmann, L. Delbouille, A. Dudhia, D. Ehhalt, M. Endemann, J. M. Flaud, R. Gessner, A. Kleinert, R. Koopman, J. Langen, M. López-Puertas, P. Mosner, H. Nett, H. Oelhaf, G. Perron, J. Remedios, M. Ridolfi, G. Stiller, and R. Zander, “Mipas: an instrument for atmospheric and climate research,” Atmos. Chem. Phys.8, 2151–2188 (2008). [CrossRef]
  2. M. Ridolfi, B. Carli, M. Carlotti, T. von Clarmann, B. M. Dinelli, A. Dudhia, J.-M. Flaud, M. Höpfner, P. E. Morris, P. Raspollini, G. Stiller, and R. J. Wells, “Optimized forward model and retrieval scheme for mipas near-real-time data processing,” Appl. Opt.39, 1323–1340 (2000). [CrossRef]
  3. P. Raspollini, C. Belotti, A. Burgess, B. Carli, M. Carlotti, S. Ceccherini, B. M. Dinelli, A. Dudhia, J.-M. Flaud, B. Funke, M. Höpfner, M. López-Puertas, V. Payne, C. Piccolo, J. J. Remedios, M. Ridolfi, and R. Spang, “Mipas level 2 operational analysis,” Atmos. Chem. Phys.6, 5605–5630 (2006). [CrossRef]
  4. S. Payan, C. Camy-Peyret, H. Oelhaf, G. Wetzel, G. Maucher, C. Keim, M. Pirre, N. Huret, A. Engel, M. C. Volk, H. Kuellmann, J. Kuttippurath, U. Cortesi, G. Bianchini, F. Mencaraglia, P. Raspollini, G. Redaelli, C. Vigouroux, M. De Mazière, S. Mikuteit, T. Blumenstock, V. Velazco, J. Notholt, E. Mahieu, P. Duchatelet, D. Smale, S. Wood, N. Jones, C. Piccolo, V. Payne, A. Bracher, N. Glatthor, G. Stiller, K. Grunow, P. Jeseck, Y. Te, and A. Butz, “Validation of version-4.61 methane and nitrous oxide observed by mipas,” Atmos. Chem. Phys.9, 413–442 (2009). [CrossRef]
  5. P. Raspollini, B. Carli, S. Ceccherini, and M. Ridolfi, “The new measurement scenario of mipas,” in “ASSFTS 14 Workshop, Firenze,” (2009).
  6. S. Ceccherini, “Analytical determination of the regularization parameter in the retrieval of atmospheric vertical profiles,” Opt. Lett.30, 2554–2556 (2005). [CrossRef] [PubMed]
  7. M. Ridolfi and L. Sgheri, “A self-adapting and altitude-dependent regularization method for atmospheric profile retrievals,” Atmos. Chem. Phys.9, 1883–1897 (2009). [CrossRef]
  8. A. Doicu, T. Trautmann, and F. Schreier, Numerical regularization for atmospheric inverse problems (Springer, 2010). [CrossRef]
  9. S. S. Kulawik, G. Osterman, D. B. A. Jones, and K. W. Bowman, “Calculation of altitude-dependent tikhonov constraints for tes nadir retrievals,” IEEE Trans. Geosci. Remote Sens.44, 1334–1342 (2006). [CrossRef]
  10. K. W. Bowman, C. Rodgers, S. S. Kulawik, J. Worden, E. Sarkissian, G. Osterman, T. Steck, M. Lou, A. Eldering, M. Shephard, H. Worden, M. Lampel, S. Clough, P. Brown, C. Rinsland, M. Gunson, and R. Beer, “Tropospheric emission spectrometer: Retrieval method and error analysis,” IEEE Trans. Geosci. Remote Sens.44, 1297–1307 (2006). [CrossRef]
  11. J. Steinwagner and G. Schwarz, “Shape-dependent regularization for the retrieval of atmospheric state parameter profiles,” Appl. Opt.45, 1000–1009 (2006). [CrossRef] [PubMed]
  12. S. Ceccherini, C. Belotti, B. Carli, P. Raspollini, and M. Ridolfi, “Technical note: Regularization performances with the error consistency method in the case of retrieved atmospheric profiles,” Atmos. Chem. Phys.7, 1435–1440 (2007). [CrossRef]
  13. C. D. Rodgers, Inverse Methods for Atmospheric Sounding: Theory and Practice (Atmospheric, Oceanic and Planetary Physics, World Scientific, 2000). [CrossRef]
  14. S. Ceccherini and M. Ridolfi, “Technical note: Variance-covariance matrix and averaging kernels for the levenberg-marquardt solution of the retrieval of atmospheric vertical profiles,” Atmos. Chem. Phys.10, 3131–3139 (2010). [CrossRef]
  15. J. J. Remedios, R. J. Leigh, A. M. Waterfall, D. P. Moore, H. Sembhi, I. Parkes, J. Greenhough, M. P. Chipperfield, and D. Hauglustaine, “Mipas reference atmospheres and comparisons to v4.61/v4.62 mipas level 2 geophysical data sets,” Atmos. Chem. Phys. Discuss.7, 9973–10017 (2007). [CrossRef]
  16. A. V. Nemuc and R. L. Dezafra, “Ground based measurements of stratospheric ozone in antarctica,” Rom. Rep. Phys.57, 445–452 (2005).
  17. A. Dudhia, “Mipas-related section of the web site of the Oxford university, department of physics,” http://www.atm.ox.ac.uk/group/mipas/ (2008).
  18. P. R. Bevington and D. K. Robinson, Data Reduction and Error Analysis for the Physical Sciences, 3rd ed. (McGraw–Hill, 2003).
  19. M. Ridolfi and L. Sgheri, “Improvement of the orm convergence criteria: test of performance and implementation details,” http://www.fci.unibo.it/~ridolfi/hak/tnconvcrit.pdf (2011).

Cited By

Alert me when this paper is cited

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.

Figures

Fig. 1 Fig. 2 Fig. 3
 

« Previous Article  |  Next Article »

OSA is a member of CrossRef.

CrossCheck Deposited