OSA's Digital Library

Virtual Journal for Biomedical Optics

Virtual Journal for Biomedical Optics


  • Editors: Andrew Dunn and Anthony Durkin
  • Vol. 8, Iss. 3 — Apr. 4, 2013

Spectral data mining for rapid measurement of organic matter in unsieved moist compost

Somsubhra Chakraborty, David C. Weindorf, Md. Nasim Ali, Bin Li, Yufeng Ge, and Jeremy L. Darilek  »View Author Affiliations

Applied Optics, Vol. 52, Issue 4, pp. B82-B92 (2013)

View Full Text Article

Enhanced HTML    Acrobat PDF (716 KB)

Browse Journals / Lookup Meetings

Browse by Journal and Year


Lookup Conference Papers

Close Browse Journals / Lookup Meetings

Article Tools



Fifty-five compost samples were collected and scanned as received by visible and near-IR (VisNIR, 350–2500 nm) diffuse reflectance spectroscopy. The raw reflectance and first-derivative spectra were used to predict log 10 -transformed organic matter (OM) using partial least squares (PLS) regression, penalized spline regression (PSR), and boosted regression trees (BRTs). Incorporating compost pH, moisture percentage, and electrical conductivity as auxiliary predictors along with reflectance, both PLS and PSR models showed comparable cross-validation r 2 and validation root-mean-square deviation (RMSD). The BRT–reflectance model exhibited best predictability (residual prediction deviation = 1.61 , cross-validation r 2 = 0.65 , and RMSD = 0.09 log 10 % ). These results proved that the VisNIR–BRT model, along with easy-to-measure auxiliary variables, has the potential to quantify compost OM with reasonable accuracy.

© 2013 Optical Society of America

OCIS Codes
(130.6010) Integrated optics : Sensors
(300.6340) Spectroscopy : Spectroscopy, infrared
(280.1415) Remote sensing and sensors : Biological sensing and sensors

Original Manuscript: July 10, 2012
Revised Manuscript: December 23, 2012
Manuscript Accepted: December 23, 2012
Published: January 30, 2013

Virtual Issues
Vol. 8, Iss. 3 Virtual Journal for Biomedical Optics

Somsubhra Chakraborty, David C. Weindorf, Md. Nasim Ali, Bin Li, Yufeng Ge, and Jeremy L. Darilek, "Spectral data mining for rapid measurement of organic matter in unsieved moist compost," Appl. Opt. 52, B82-B92 (2013)

Sort:  Author  |  Year  |  Journal  |  Reset  


  1. USDA-USCC, “Test methods for the evaluation of composts and composting” (CD ROM) (Composting Council Research and Education Foundation, 2002).
  2. A. Walkley and I. A. Black, “An examination of the Degtjareff method for determining organic carbon in soils: effect of variations in digestion conditions and of inorganic soil constituents,” Soil Sci. 63, 251–264 (1934). [CrossRef]
  3. D. W. Nelson and L. E. Sommers, “Total carbon, organic carbon and organic matter,” in Methods of Soil Analysis, Part 3: Chemical Methods, J. M. Bigham, ed. (ASA, 1996), pp. 961–1010.
  4. S. Chakraborty, D. C. Weindorf, C. L. S. Morgan, Y. Ge, J. M. Galbraith, and C. S. Kahlon, “Rapid identification of oil-contaminated soils using visible near-infrared diffuse reflectance spectroscopy,” J. Environ. Qual. 39, 1378–1387 (2010). [CrossRef]
  5. D. C. Weindorf, J. P. Muir, and C. Landeros-Sánchez, “Organic compost and manufactured fertilizers: economics and ecology,” in Integrating Agriculture, Conservation, and Ecotourism: Examples from the Field (Issues In Agroecology—Present Status and Future Prospectus 1), W. B. Campbell and O. S. Lopez, eds. (Springer, 2011), pp. 27–53.
  6. S. Chakraborty, D. C. Weindorf, Y. Zhu, C. L. S. Morgan, Y. Ge, and J. M. Galbraith, “Spectral reflectance variability from soil physicochemical properties in oil contaminated soils,” Geoderma 177–178, 80–89 (2012). [CrossRef]
  7. D. F. Malley, L. Yesmin, and R. G. Eilers, “Rapid analysis of hog manure and manure-amended soils using near-infrared spectroscopy,” Soil Sci. Soc. Am. J. 66, 1677–1686 (2002). [CrossRef]
  8. J. B. Reeves, G. W. McCarty, and J. J. Meisinger, “Near infrared reflectance spectroscopy for the analysis of agricultural soils,” J. Near Infrared Spectrosc. 7, 179–193 (1999). [CrossRef]
  9. W. S. Lee, J. F. Sanchez, R. S. Mylavarapu, and J. S. Choe, “Estimating chemical properties of Florida soils using spectral reflectance,” Trans. ASAE 46, 1443–1453 (2003).
  10. C. Chang, D. A. Laird, and C. R. Hurburgh, “Influence of soil moisture on near-infrared reflectance spectroscopic measurement of soil properties,” Soil Sci. 170, 244–255 (2005). [CrossRef]
  11. G. W. McCarty, J. B. Reeves, V. B. Reeves, R. F. Follett, and J. M. Kimble, “Mid-infrared and near-infrared diffuse reflectance spectroscopy for soil carbon measurement,” Soil Sci. Soc. Am. J. 66, 640–646 (2002). [CrossRef]
  12. K. D. Shepherd and M. G. Walsh, “Development of reflectance spectral libraries for characterization of soil properties,” Soil Sci. Soc. Am. J. 66, 988–998 (2002). [CrossRef]
  13. E. Ben-Dor and A. Banin, “Near-infrared analysis as a rapid method to simultaneously evaluate several soil properties,” Soil Sci. Soc. Am. J. 59, 364–372 (1995). [CrossRef]
  14. B. W. Dunn, H. G. Beecher, G. D. Batten, and S. Ciavarella, “The potential of near-infrared reflectance spectroscopy for soil analysis—a case study from the Riverine Plain of south-eastern Australia,” Aust. J. Exp. Agric. 42, 607–614 (2002). [CrossRef]
  15. K. Islam, B. Singh, and A. McBratney, “Simultaneous estimation of several soil properties by ultraviolet, visible, and near-infrared reflectance spectroscopy,” Aust. J. Soil Res. 41, 1101–1114 (2003). [CrossRef]
  16. K. A. Sudduth and J. W. Hummel, “Soil organic matter, CEC, and moisture sensing with a portable NIR spectrophotometer,” Trans. ASAE 36, 1571–1582 (1993).
  17. M. Kang, “Quantification of soil organic carbon using mid- and near-DRIFT spectroscopy,” Master’s thesis (Texas A&M University, 2002).
  18. S. Tanner, H. Shu, A. Frank, L. Wang, E. Zandi, M. Mumby, P. A. Pevzner, and V. Bafna, “Inspect: fast and accurate identification of post-translationally modified peptides from tandem mass spectra,” Anal. Chem. 77, 4626–4639 (2005). [CrossRef]
  19. J. W. T. Tung, “Determination of metal components in marine sediments using energy dispersive x-ray fluorescence (ED-XRF) spectrometry,” Ann. Chim. 94, 837–846 (2004). [CrossRef]
  20. D. M. Smith, J. J. Griffin, and E. D. Goldberg, “Spectrometric method for the quantitative determination of elemental carbon,” Anal. Chem. 47, 233–238 (1975). [CrossRef]
  21. R. C. Dalal and R. J. Henry, “Simultaneous determination of moisture, organic carbon and total nitrogen by near infrared reflectance spectrophotometry,” Soil Sci. Soc. Am. J. 50, 120–123 (1986). [CrossRef]
  22. M. J. Morra, M. H. Hall, and L. L. Freeborn, “Carbon and nitrogen analysis of soil fractions using near-infrared reflectance spectroscopy,” Soil Sci. Soc. Am. J. 55, 288–291 (1991). [CrossRef]
  23. L. Tremblay and J. Gagné, “Fast quantification of humic substances and organic matter by direct analysis of sediments using DRIFT spectroscopy,” Anal. Chem. 74, 2985–2993 (2002). [CrossRef]
  24. E. K. Kemsley, H. S. Tapp, A. J. Scarlett, S. J. Miles, R. Hammond, and R. H. Wilson, “Comparison of spectroscopic techniques for the determination of Kjeldahl and ammoniacal nitrogen content of farmyard manure,” J. Agric. Food Chem. 49, 603–609 (2001). [CrossRef]
  25. W. Saeys, A. M. Mouazen, and H. Ramon, “Potential for on-site and on-line analysis of hog manure using visual and near-infrared reflectance spectroscopy,” in Precision Livestock Farming’05, S. Cox, ed. (Wageningen Academic, 2005), pp. 131–138.
  26. J. B. Reeves and J. S. Van Kessel, “Determination of ammonium-N, moisture, total C and total N in dairy manures using a near infrared fibre-optic spectrometer,” J. Near Infrared Spectrosc. 8, 151–160 (2000). [CrossRef]
  27. S. L. Preece, C. L. S. Morgan, B. W. Auvermann, K. Wilke, and K. Heflin, “Determination of ash content in solid cattle manure with visible near-infrared diffuse reflectance spectroscopy,” Trans. ASABE 52, 609–614 (2009).
  28. S. L. P. Sakirkin, C. L. S. Morgan, and B. W. Auvermann, “Effects of sample processing on ash content determination in solid cattle manure with visible/near-infrared spectroscopy,” Trans. ASABE 53: 421–428 (2010).
  29. W. Ye, J. C. Lorimor, C. Hurburgh, H. Zhang, and J. Hattery, “Application of near-infrared reflectance spectroscopy for determination of nutrient contents in liquid and solid manures,” Trans. ASAE 48, 1911–1918 (2005).
  30. Z. Yang, L. Han, and X. Fan, “Rapidly estimating nutrient contents of fattening pig manure from floor scrapings by near infrared reflectance spectroscopy,” J. Near Infrared Spectrosc. 14, 261–268 (2006). [CrossRef]
  31. K. Suehara, Y. Nakano, and T. Yano, “Simultaneous measurement of the carbon and nitrogen content of compost using near-infrared spectroscopy,” J. Near Infrared Spectrosc. 9, 35–41 (2001). [CrossRef]
  32. D. F. Malley, C. McClure, P. D. Martin, K. Buckley, and W. P. McCaughe, “Compositional analysis of cattle manure during composting using a field-portable near-infrared spectrometer,” Commun. Soil Sci. Plant Anal. 36, 455–475 (2005). [CrossRef]
  33. E. Ben-Dor, Y. Inbar, and Y. Chen, “The reflectance spectra of organic matter in the visible near infrared and short wave infrared region (400–2500 nm) during a control decomposition process,” Remote Sens. Environ. 61, 1–15 (1997). [CrossRef]
  34. H. S. S. Sharma, M. Kilpatrick, G. Lyons, S. Sturgeon, J. Archer, S. Moore, L. Cheung, and K. Finegan, “Visible and near-infrared calibrations for quality assessment of fresh phase I and II mushroom (Agaricus bisporus) compost,” Appl. Spectrosc. 59, 1399–1405 (2005). [CrossRef]
  35. A. L. McWhirt, D. C. Weindorf, S. Chakraborty, and B. Li, “Visible near infrared diffuse reflectance spectroscopy (VisNIR DRS) for rapid measurement of organic matter in compost,” Waste Manag. Res. 30, 1049–1058 (2012). [CrossRef]
  36. Y. Zhu, D. C. Weindorf, S. Chakraborty, B. Haggard, S. Johnson, and N. Bakr, “Characterizing surface soil water with field portable diffuse reflectance spectroscopy,” J. Hydrol. 391, 133–140 (2010). [CrossRef]
  37. T. H. Demetriades-Shah, M. D. Steven, and J. A. Clark, “High-resolution derivative spectra in remote sensing,” Remote Sens. Environ. 33, 55–64 (1990). [CrossRef]
  38. R Development Core Team, “R: a language and environment for statistical computing,” http://www.cran.r-project.org (2008).
  39. D. J. Brown, K. D. Shepherd, M. G. Walsh, M. D. Mays, and T. G. Reinsch, “Global soil characterization with VNIR diffuse reflectance spectroscopy,” Geoderma 132, 273–290 (2006). [CrossRef]
  40. G. E. P. Box and D. R. Cox, “An analysis of transformations,” J. R. Stat. Soc. Ser. B 26, 211–252 (1964). [CrossRef]
  41. H. Steinhaus, “Sur la division des corp materiels en parties,” Bull. Acad. Pol. Sci. 4, 801–804 (1956) (in French).
  42. J. H. Ward, “Hierarchical grouping to optimize an objective function,” J. Am. Stat. Assoc. 48, 236–244.(1963). [CrossRef]
  43. T. H. Waiser, C. L. S. Morgan, D. J. Brown, and C. T. Hallmark, “In situ characterization of soil clay content with visible near-infrared diffuse reflectance spectroscopy,” Soil Sci. Soc. Am. J. 71, 389–396 (2007). [CrossRef]
  44. D. M. Haaland and E. V. Thomas, “Partial least-squares methods for spectral analyses. 1. Relation to other quantitative calibration methods and the extraction of qualitative information,” Anal. Chem. 60, 1193–1202 (1988). [CrossRef]
  45. P. H. C. Eilers and B. D. Marx, “Generalized linear additive smooth structures,” J. Comput. Graph. Statist. 11, 758–783 (2002). [CrossRef]
  46. J. H. Friedman, “Greedy function approximation: a gradient boosting machine,” Technical report (Department of Statistics, Stanford University, 1999).
  47. R. A. V. Rossel and T. Behrens, “Using data mining to model and interpret soil diffuse reflectance spectra,” Geoderma 158, 46–54 (2010). [CrossRef]
  48. G. A. Shonk, L. D. Gaultney, D. G. Schulze, and G. E. Van Scoyoc, “Spectroscopic sensing of soil organic matter content,” Trans. ASAE 34, 1978–1984 (1991).
  49. P. C. Williams, “Commercial near-infrared reflectance analyzers,” in Near-infrared Technology in the Agricultural and Food Industries, P. C. Williams and K. H. Norris, eds. (American Association of Cereal Chemists, 1987), pp. 107–136.
  50. B. Minasny and A. B. McBratney, “Regression rules as a tool for predicting soil properties from infrared reflectance spectroscopy,” Chemom. Intell. Lab. Syst. 94, 72–79 (2008). [CrossRef]
  51. L. Kooistra, R. Wehrens, R. S. E. W. Leuven, and L. M. C. Buydens, “Possibilities of VNIR spectroscopy for the assessment of soil contamination in river floodplains,” Anal. Chim. Acta 446, 97–105 (2001). [CrossRef]
  52. S. Shibusawa, S. W. I. Anom, S. Sato, A. Sasao, and S. Hirako, “Soil mapping using the real-time soil spectrophotometer,” in ECPA 2001, Third European Conference on Precision Agriculture, G. Grenier and S. Blackmore, eds. (Agro Montpellier, 2001), Vol. 1, pp. 497– 508.
  53. American Association for State Highway and Transportation Officials, “Standard specifications for compost for erosion/sediment control,” http://compostingcouncil.org/admin/wp-content/plugins/wp-pdfupload/pdf/32/AASHTO-Specifications.pdf .
  54. J. L. Bishop, M. D. Lane, M. D. Dyar, and A. J. Brown, “Reflectance and emission spectroscopy study of four groups of phyllosilicates: smectites, kaolinite–serpentines, chlorites and micas,” Clay Miner. 43, 35–54 (2008). [CrossRef]
  55. G. R. Hunt, “Spectral signatures of particulate minerals in visible and near IR,” Geophysics 42, 501–513 (1977). [CrossRef]
  56. R. N. Clark, “Spectroscopy of rocks and minerals, and principles of spectroscopy,” in Remote Sensing for the Earth Sciences: Manual of Remote Sensing, N. Rencz, ed. (Wiley, 1999), pp. 3–52.
  57. J. Elith, J. R. Leathwick, and T. Hastie, “A working guide to boosted regression trees,” J. Anim. Ecol. 77, 802–813 (2008). [CrossRef]
  58. F. J. Calderon, J. B. Reeves, H. P. Collins, and E. A. Paul, “Chemical differences in soil organic matter fractions determined by diffuse-reflectance mid-infrared spectroscopy,” Soil Sci. Soc. Am. J. 75, 568–579 (2011). [CrossRef]
  59. Y. Ge, C. L. S. Morgan, J. A. Thomasson, and T. Waiser, “A new perspective to near-infrared reflectance spectroscopy: a wavelet approach,” Trans. ASABE 50, 303–311 (2007).

Cited By

Alert me when this paper is cited

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.

« Previous Article  |  Next Article »

OSA is a member of CrossRef.

CrossCheck Deposited