OSA's Digital Library

Journal of Optical Communications and Networking

Journal of Optical Communications and Networking

  • Editors: K. Bergman and V. Chan
  • Vol. 3, Iss. 8 — Aug. 1, 2011
  • pp: A40–A48

Building Data Centers With Optically Connected Memory

Daniel Brunina, Caroline P. Lai, Ajay S. Garg, and Keren Bergman  »View Author Affiliations

Journal of Optical Communications and Networking, Vol. 3, Issue 8, pp. A40-A48 (2011)

View Full Text Article

Enhanced HTML    Acrobat PDF (764 KB)

Browse Journals / Lookup Meetings

Browse by Journal and Year


Lookup Conference Papers

Close Browse Journals / Lookup Meetings

Article Tools



Future data centers will require novel, scalable memory architectures capable of sustaining high bandwidths while still achieving low memory access latencies. Electronic interconnects cannot meet the challenges presented by the need for multi-terabit off-chip memory data paths. In this work, the electronic bus between main memory and its host processor is replaced with a circuit-switched optical interconnection network. We investigate the impact of our optically connected memory system on large-scale architectures and experimentally validate the protocol using field-programmable gate array based processor nodes and a custom-designed memory controller. The processor communicates all-optically with multiple synchronous dynamic random access memory nodes using 4 × 2.5-Gb/s wavelength-striped payloads, operating error free with bit-error rates less than 1 0 12 .

© 2011 OSA

OCIS Codes
(060.4510) Fiber optics and optical communications : Optical communications
(200.0200) Optics in computing : Optics in computing
(200.4650) Optics in computing : Optical interconnects

ToC Category:
Optics in the Data Center

Original Manuscript: February 1, 2011
Revised Manuscript: June 12, 2011
Manuscript Accepted: June 17, 2011
Published: July 14, 2011

Daniel Brunina, Caroline P. Lai, Ajay S. Garg, and Keren Bergman, "Building Data Centers With Optically Connected Memory," J. Opt. Commun. Netw. 3, A40-A48 (2011)

Sort:  Author  |  Year  |  Journal  |  Reset  


  1. W. A. Wulf and S. A. McKee, "Hitting the memory wall: implications of the obvious," ACM SIGARCH Comput. Archit. News 23, (1), 20‒24 (1995). [CrossRef]
  2. K. Bergman, S. Borkar, D. Campbell, W. Carlson, W. Dally, M. Denneau, P. Franzon, W. Harrod, J. Hiller, S. Karp, S. Keckler, D. Klein, R. Lucas, M. Richards, A. Scarpelli, S. Scott, A. Snavely, T. Sterling, R. S. Williams, K. Yelick, and P. Kogge, Exascale computing study: technology challenges in achieving exascale systems, [Online]. Available: http://www.cse.nd.edu/Reports/2008/TR-2008-13.pdf.
  3. B. Jacob, S. W. Ng, and D. T. Wang, Memory Systems: Cache, DRAM, Disk, Morgan Kaufmann, 2007.
  4. R. Ho, W. Mai, and M. A. Horowitz, "The future of wires," Proc. IEEE 89, (4), 490‒504 (2001). [CrossRef]
  5. DDR3 SDRAM Standard, JEDEC Solid State Technology Association, [Online]. Available: http://www.jedec.org/standards-documents/docs/jesd-79-3d.
  6. Product specification. 2 GB DDR3 SDRAM, Micron Technology Inc., [Online]. Available: http://download.micron.com/pdf/datasheets/modules/ddr3/jsf16c256x64h.pdf.
  7. International Technology Roadmap for Semiconductors (ITRS) 2009 Edition, The ITRS Technology Working Groups, [Online]. Available: http://www.itrs.net.
  8. Specification Addendum. Fully Buffered DIMM, Intel Corp., [Online]. Available: http://www.intel.com/technology/memory/FBDIMM/spec/Intel_FBD_Spec_Addendum_rev_p9.pdf.
  9. N. Magen, A. Kolodny, U. Weiser, and N. Shamir, "Interconnect-power dissipation in a microprocessor," SLIP’04: Proc. of the 2004 Int. Workshop on System Level Interconnect Prediction, 2004, ACM, pp. 7‒13.
  10. A. F. Benner, D. M. Kuchta, P. K. Pepeljugoski, R. A. Budd, G. Hougham, B. V. Fasano, K. Marston, H. Bagheri, E. J. Seminaro, X. Hui, D. Meadowcroft, M. H. Fields, L. McColloch, M. Robinson, F. W. Miller, R. Kaneshiro, R. Granger, D. Childers, and E. Childers, "Optics for high-performance servers and supercomputers," Opt. Fiber Commun. Conf. (OFC 2010), Mar. 2010, San Diego, CA, OTuH1.
  11. B. J. Offrein and P. Pepeljugoski, "Optics in supercomputers," Eur. Conf. Opt. Commun. (ECOC 2009), Sept. 2009, Vienna, Austria, 3.1.3.
  12. Y. Katayama and A. Okazaki, "Optical interconnect opportunities for future server memory systems," Proc. IEEE HPCA, 2007, pp. 46‒50.
  13. G. Hendry, E. Robinson, V. Gleyzer, J. Chan, L. P. Carloni, N. Bliss, and K. Bergman, "Circuit-switched memory access in photonic interconnection networks for high-performance embedded computing," Supercomputing (SC), Nov. 2010.
  14. A. Hadke, T. Benavides, S. J. B. Yoo, R. Amirtharajah, and V. Akella, "OCDIMM: scaling the DRAM memory wall using WDM based optical interconnects," Int. Symp. on High-Performance Interconnects, Aug. 2008.
  15. S. Beamer, C. Sun, Y. J. Kwon, A. Joshi, C. Batten, and V. Stojanovic, "Re-architecting DRAM memory systems with monolithically integrated silicon photonics," 37th Int. Symp. on Computer Architecture (ISCA-37), June 2010.
  16. Datasheet: Corning SMF-28e optical fiber product information, Corning Inc., [Online]. Available: http://www.princetel.com/datasheets/SMF28e.pdf.
  17. B. G. Lee, F. E. Doany, S. Assefa, W. M. J. Green, M. Yang, C. L. Schow, C. V. Jahnes, S. Zhang, J. Singer, V. I. Kopp, J. A. Kash, and Y. A. Vlasov, "20-µm-pitch eight-channel monolithic fiber array coupling 160 Gb/s/channel to silicon nanophotonic chip," Opt. Fiber Commun. Conf. (OFC 2011), Mar. 2010, San Diego, CA, PDPA4.
  18. W. A. Zortman, M. R. Watts, D. C. Trotter, R. W. Young, and A. L. Lentine, "Low-power high-speed silicon microdisk modulators," Conf. on Lasers and Electro-Optics, May 2010, CThJ4.
  19. H. D. Thacker, Y. Luo, J. Shi, I. Shubin, J. Lexau, X. Zheng, G. Li, J. Yao, J. Costa, T. Pinguet, A. Mekis, P. Dong, S. Liao, D. Feng, M. Asghari, R. Ho, K. Raj, J. G. Mitchell, A. V. Krishnamoorthy, and J. E. Cunningham, "Flip-chip integrated silicon photonic bridge chips for sub-picojoule per bit optical links," Electronic Components and Technology Conf. (ECTC), June 2010, pp. 240‒246.
  20. D. Brunina, C. P. Lai, A. S. Garg, and K. Bergman, "First experimental demonstration of optically-connected SDRAM across a transparent optical network test-bed," 23rd Annu. Meeting of the IEEE Photonics Society, Nov. 2010, Denver, CO, ThI 1.
  21. K. J. Barkerm, A. Benner, R. Hoare, A. Hoisie, A. K. Jones, D. J. Kerbyson, D. Li, R. Melhem, R. Rajamony, E. Schenfeld, S. Shao, C. Stunkel, and P. Walker, "On the feasibility of optical circuit switching for high performance computing systems," Supercomputing (SC), Nov. 2005.
  22. D. Brunina, C. P. Lai, A. S. Garg, and K. Bergman, "Wavelength-striped multicasting of optically-connected memory for large-scale computing systems," Opt. Fiber Commun. Conf. (OFC 2011), Mar. 2011, Los Angeles, CA, OWH4.
  23. S. Rixner, "Memory controller optimizations for web servers," 37th Int. Symp. on Microarchitecture, Dec. 2004, pp. 355‒366.
  24. S. Rixner, W. J. Dally, U. J. Kapasi, P. Mattson, and J. D. Owens, "Memory access scheduling," Proc. of the 27th Int. Symp. on Computer Architecture, 2000, pp. 128‒138.
  25. W. Lin, S. K. Reinhardt, and D. Burger, "Reducing DRAM latencies with an integrated memory hierarchy design," Proc. of the Int. Symp. on High-Performance Computer Architecture (HPCA), 2001, pp. 301‒312.
  26. A. Greenberg, J. R. Hamilton, N. Jain, S. Kandula, C. Kim, P. Lahiri, D. A. Maltz, P. Patel, and S. Sengupta, "VL2: a scalable and flexible data center network," Proc. of the ACM SIGCOMM 2009 Conf. on Data Communication (SIGCOMM’09), Aug. 2009, Barcelona, Spain, pp. 51‒62.
  27. C. P. Lai, D. Brunina, and K. Bergman, "Demonstration of 8 × 40-Gb/s wavelength-striped packet switching in a multi-terabit capacity optical network test-bed," 23rd Annu. Meeting of the IEEE Photonics Society, Nov. 2010, Denver, CO, ThQ 2.
  28. W. Carlson, T. El-Ghazawi, B. Numrich, and K. Yelick, "Programming in the partitioned global address space model," Supercomputing 2003, [Online]. Available: http://www.gwu.edu/upc/tutorials.html.
  29. O. Liboiron-Ladouceur, B. A. Small, and K. Bergman, "Physical layer scalability of WDM optical packet interconnection networks," J. Lightwave Technol. 24, (1), 262‒270 (2006). [CrossRef]
  30. “IEEE P802.3ba 40 Gb/s and 100 Gb/s Ethernet Task Force,” 21 June 2010.
  31. J. W. Cooley and J. W. Tukey, "An algorithm for the machine calculation of complex Fourier series," Math. Comput. 19, 297‒301 (1965). [CrossRef]
  32. N. Nedovic, A. Kristensson, S. Parikh, S. Reddy, W. Walker, S. McLeod, N. Tzartzanis, H. Tamura, K. Kanda, T. Yamamoto, S. Matsubara, M. Kibune, Y. Doi, S. Ide, Y. Tsunoda, T. Yamabana, T. Shibasaki, Y. Tomita, T. Hamada, M. Sugawara, J. Ogawa, T. Ikeuchi, and N. Kuwata, "A 2 × 22.3 Gb/s SFI5.2 SerDes in 65 nm CMOS," Compound Semiconductor Integrated Circuit Symp., 2009 (CISC 2009), 11–14 Oct. 2009, pp. 1‒4.
  33. L. Chen, K. Preston, S. Manipatruni, and M. Lipson, "Integrated GHz silicon photonic interconnect with micrometer-scale modulators and detectors," Opt. Express 17, (17), 15248‒15256 (2009). [CrossRef]

Cited By

Alert me when this paper is cited

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.

« Previous Article  |  Next Article »

OSA is a member of CrossRef.

CrossCheck Deposited