OSA's Digital Library

Journal of Optical Communications and Networking

Journal of Optical Communications and Networking

  • Editors: K. Bergman and O. Gerstel
  • Vol. 4, Iss. 9 — Sep. 1, 2012
  • pp: 651–662

AWGR-Based Optical Topologies for Scalable and Efficient Global Communications in Large-Scale Multi-Processor Systems

Xiaohui Ye, S. J. B. Yoo, and Venkatesh Akella  »View Author Affiliations

Journal of Optical Communications and Networking, Vol. 4, Issue 9, pp. 651-662 (2012)

View Full Text Article

Enhanced HTML    Acrobat PDF (1336 KB)

Browse Journals / Lookup Meetings

Browse by Journal and Year


Lookup Conference Papers

Close Browse Journals / Lookup Meetings

Article Tools



In large-scale multi-processor computing systems, global communications are typically supported by an auxiliary network (e.g., IBM Blue Gene) or with hardware support in the network (e.g., NEC Earth Simulator). We explore the potential for realizing efficient global communications that can scale beyond a million processors by harnessing the unique parallelism and wavelength routing properties of optical devices. Specifically, we use an arrayed waveguide grating router (AWGR) device as the basic building block in realizing scalable global communication. The AWGR is a passive switch fabric (wavelength router) that uses multiple wavelengths to interconnect outputs and inputs by following a specific cyclic wavelength routing (permutation) pattern. We analyze different network topologies using AWGR devices for barrier synchronization and propose techniques to pick parameters of the network for a given number of processors. We compare the performance and energy consumption for barrier synchronization with what is achievable with state-of-the-art electrical networks.

© 2012 OSA

OCIS Codes
(200.4650) Optics in computing : Optical interconnects
(060.4258) Fiber optics and optical communications : Networks, network topology

ToC Category:
Regular Papers

Original Manuscript: April 4, 2012
Revised Manuscript: June 17, 2012
Manuscript Accepted: July 11, 2012
Published: August 3, 2012

Xiaohui Ye, S. J. B. Yoo, and Venkatesh Akella, "AWGR-Based Optical Topologies for Scalable and Efficient Global Communications in Large-Scale Multi-Processor Systems," J. Opt. Commun. Netw. 4, 651-662 (2012)

Sort:  Author  |  Year  |  Journal  |  Reset  


  1. R. Karmani, N. Chen, A. Shali, and R. Johnson, “Barrier synchronization pattern,” 2009 [Online]. Available: http://parlab.eecs.berkeley.edu/wiki/_media/patterns/paraplop_g1_3.pdf.
  2. M. Blumrich, D. Chen, P. Coteus, A. Gara, M. Giampapa, P. Heidelberger, S. Singh, B. Steinmacher-Burow, T. Takken, and P. Vranas, “Design and analysis of the BlueGene/L torus inter-connection network,” IBM Research Report RC23025 (W0312–022), 2003.
  3. S. L. Scott, “Synchronization and communication in the T3E multiprocessor,” ACM SIGOPS Oper. Syst. Rev., vol. 30, pp. 26–36, 1996. [CrossRef]
  4. S. Habata, K. Umezawa, M. Yokokawa, and S. Kitawaki, “Hardware system of the Earth Simulator,” Parallel Comput., vol. 30, pp. 1287–1313, 2004. [CrossRef]
  5. R. Ramaswami, K. Sivarajan, and G. Sasaki, Optical Networks: A Practical Perspective. 3rd ed.Morgan Kaufmann, 2009.
  6. Y. Yoshikuni, “Semiconductor arrayed waveguide gratings for photonic integrated devices,” IEEE J. Sel. Top. Quantum Electron., vol. 8, pp. 1102–1114, 2002. [CrossRef]
  7. T. Suzuki and H. Tsuda, “Ultrasmall arrowhead arrayed-waveguide grating with V-shaped bend waveguides,” IEEE Photon. Technol. Lett., vol. 17, pp. 810–812, 2005. [CrossRef]
  8. F. Soares, J. H. Baek, N. Fontaine, X. Zhou, Y. Wang, R. Scott, J. Heritage, C. Junesand, S. Lourdudoss, K. Y. Liou, R. Hamm, W. Wang, B. Patel, S. Vatanapradit, L. Gruezke, W. T. Tsang, and S. J. B. Yoo, “Monolithically integrated InP wafer-scale 100-channel × 10-GHz AWG and Michelson interferometers for 1-THz-bandwidth optical arbitrary waveform generation,” in Optical Fiber Communication Conf. (OFC), 2010, OThS1.
  9. J. Kim, W. J. Dally, and D. Abts, “Flattened butterfly: a cost-efficient topology for high-radix networks,” in 34th Int. Symp. on Computer Architecture, 2007, pp. 126–137.
  10. J. Oh, M. Prvulovic, and A. Zajic, “TLSync: support for multiple fast barriers using on-chip transmission lines,” in Proc. of the 38th Int. Symp. on Computer Architecture, 2011, pp. 105–116.
  11. J. Sartori and R. Kumar, “Low-overhead, high-speed multi-core barrier synchronization,” in 5th Int. Conf. on High-Performance Embedded Architectures and Compilers (HiPEAC), 2010, pp. 18–34.
  12. D. Adams, “Cray T3D system architecture overview manual,” 1993 [Online]. Available: ftp://ftp.cray.com/product-info/mpp/T3D_Architecture_Over/T3D.overview.html.
  13. E. Anderson, J. Brooks, C. Grassl, and S. Scott, “Performance of the CRAY T3E multiprocessor,” in 1997 ACM/IEEE Conf. on Supercomputing, 1997, pp. 1–17.
  14. W. Cohen, D. Hyde, and R. Gaede, “An optical bus-based distributed dynamic barrier mechanism,” IEEE Trans. Comput., vol. 49, pp. 1354–1365, 2000. [CrossRef]
  15. N. Binkert, A. Davis, M. Lipastiy, R. Schreiber, and D. Van-trease, “Nanophotonic barriers,” in Workshop on Photonic Interconnects & Computer Architecture (in conjunction with MICRO 41), 2009, pp. 1–4.
  16. A. Louri and H. Sung, “An optical multi-mesh hypercube: a scalable optical interconnection network for massively parallel computing,” J. Lightwave Technol., vol. 12, pp. 704–716, 1994. [CrossRef]
  17. R. Rabenseifner, “Optimization of collective reduction operations,” Lect. Notes Comput. Sci., vol. 3036, pp. 1–9, 2004.
  18. R. Thakur, R. Rabenseifner, and W. Gropp, “Optimizing of collective communication operations in MPICH,” Int. J. High Perform. Comput. Appl., vol. 19, pp. 49–66, 2005. [CrossRef]
  19. M. Al-Fares, A. Loukissas, and A. Vahdat, “A scalable, commodity data center network architecture,” in Proc. of the ACM SIGCOMM 2008 Conf. on Data Communication, 2008, pp. 63–74.
  20. J. Kim, W. J. Dally, S. Scott, and D. Abts, “Technology-driven, highly-scalable dragonfly topology,” in 35th Int. Symp. on Computer Architecture, 2008, pp. 77–88.
  21. C. Guo, H. Wu, K. Tan, L. Shi, Y. Zhang, and S. Lu, “Dcell: a scalable and fault-tolerant network structure for data centers,” in Proc. of the ACM SIGCOMM 2008 Conf. on Data Communication, 2008, pp. 75–86.
  22. D. Li, C. Guo, H. Wu, K. Tan, Y. Zhang, and S. Lu, “FiConn: using backup port for server interconnection in data centers,” in Proc. of INFOCOM, 2009, pp. 2276–2285.
  23. K. Okamoto, T. Hasegawa, O. Ishida, A. Himeno, and Y. Ohmori, “32 × 32 arrayed-waveguide grating multiplexer with uniform loss and cyclic frequency characteristics,” Electron. Lett., vol. 33, pp. 1865–1866, 1997. [CrossRef]
  24. S. Bregni, A. Pattavina, and G. Vegetti, “Architectures and performance of AWG-based optical switching nodes for IP networks,” IEEE J. Sel. Areas Commun., vol. 21, no. 7, pp. 1113–1121, 2003. [CrossRef]
  25. W. D. Zhong and R. S. Tucker, “Wavelength routing-based photonic packet buffers and their applications in photonic packet switching systems,” J. Lightwave Technol., vol. 16, no. 10, pp. 1737–1745, 1998. [CrossRef]
  26. D. Banerjee, J. Frank, and B. Mukherjee, “Passive optical network architecture based on waveguide grating routers,” IEEE J. Sel. Areas Commun., vol. 16, no. 7, pp. 1040–1050, 1998. [CrossRef]
  27. X. Ye, P. Mejia, Y. Yin, R. Proietti, S. J. B. Yoo, and V. Akella, “DOS—A scalable optical switch for datacenters,” in Proc. of ACM/IEEE Symp. on Architectures for Networking and Communications Systems, 2010, pp. 1–12.
  28. N. Farrington, G. Porter, S. Radhakrishnan, H. H. Bazzaz, V. Subramanya, Y. Fainman, G. Papen, and A. Vahdat, “Helios: a hybrid electrical/optical switch architecture for modular data centers,” in Proc. of the ACM SIGCOMM 2010 Conf. on Data Communication, 2010, pp. 339–350.
  29. C. Minkenberg, F. Abel, P. Muller, R. Krishnamurthy, M. Gusat, P. Dill, I. Iliadis, R. Luijten, R. R. Hemenway, R. Grzybowski, and E. Schiattarella, “Designing a crossbar scheduler for HPC applications,” IEEE Micro, vol. 26, pp. 58–71, 2006. [CrossRef]
  30. J. M. Mellor-Crummey and M. L. Scott, “Algorithms for scalable synchronization on shared-memory multiprocessors,” ACM Trans. Comput. Syst., vol. 9, pp. 21–65, 1991. [CrossRef]
  31. B. Arimilli, R. Arimilli, V. Chung, S. Clark, W. Denzel, B. Drerup, T. Hoefler, J. Joyner, J. Lewis, J. Li, N. Ni, and R. Rajamony, “The PERCS high-performance interconnect,” in Proc. of the 2010 18th IEEE Symp. on High Performance Interconnects, 2010, pp. 75–82.
  32. J. Dean and S. Ghemawat, “MapReduce: simplified data processing on large clusters,” in Proc. of the Sixth Symp. on Operating System Design and Implementation (OSDI), 2004, pp. 137–149.
  33. A. Sugita, A. Kaneko, K. Okamoto, M. Itoh, A. Himeno, and Y. Ohmori, “Very low insertion loss arrayed-waveguide grating with vertically tapered waveguides,” IEEE Photon. Technol. Lett., vol. 12, pp. 1180–1182, 2000. [CrossRef]
  34. N. Ismail, A. C. Baclig, P. J. Caspers, F. Sun, K. Worhoff, R. M. d. Ridder, M. Pollnau, and A. Driessen, “Design of low-loss arrayed waveguide gratings for applications in integrated Raman spectroscopy,” in 2010 Conf. on Lasers and Electro-Optics (CLEO) and Quantum Electronics and Laser Science Conf. (QELS), 2010, pp. 1–2.
  35. R. S. Tucker, “The role of optics and electronics in high-capacity routers,” J. Lightwave Technol., vol. 24, no. 12, pp. 4655–4673, 2006. [CrossRef]
  36. J. Poulton, R. Palmer, A. M. Fuller, T. Greer, J. Eyles, W. J. Dally, and M. Horowitz, “A 14-mW 6.25-Gb/s transceiver in 90-nm CMOS,” IEEE J. Solid-State Circuits, vol. 42, pp. 2745–2757, 2007. [CrossRef]
  37. X. Zheng, J. Lexau, Y. Luo, H. Thacker, T. Pinguet, A. Mekis, G. Li, J. Shi, P. Amberg, N. Pinckney, K. Raj, R. Ho, J. E. Cunningham, and A. V. Krishnamoorth, “Ultra-low-energy all-CMOS modulator integrated with driver,” Opt. Express, vol. 18, no. 3, pp. 3059–3070, 2010. [CrossRef] [PubMed]
  38. W. J. Dally, “From hypercubes to dragonflies: a short history of interconnect,” IAA Workshop, 2008.

Cited By

Alert me when this paper is cited

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.

Next Article »

OSA is a member of CrossRef.

CrossCheck Deposited