Infrastructure and Energy Conservation in Big Data Computing: A Survey
DOI:
https://doi.org/10.26636/jtit.2019.132419Keywords:
Big Data, cloud, cluster, energy-efficient computation, grid, HPC, software platformAbstract
Progress in life, physical sciences and technology depends on efficient data-mining and modern computing technologies. The rapid growth of data-intensive domains requires a continuous development of new solutions for network infrastructure, servers and storage in order to address Big Datarelated problems. Development of software frameworks, include smart calculation, communication management, data decomposition and allocation algorithms is clearly one of the major technological challenges we are faced with. Reduction in energy consumption is another challenge arising in connection with the development of efficient HPC infrastructures. This paper addresses the vital problem of energy-efficient high performance distributed and parallel computing. An overview of recent technologies for Big Data processing is presented. The attention is focused on the most popular middleware and software platforms. Various energy-saving approaches are presented and discussed as well.
Downloads
References
[1] P. D. Healy, T. Lynn, E. Barrett, and J. P. Morrison, “Single system image: A survey”, J. Parallel Distrib. Comput., vol. 90–91, pp. 35–51, 2016. DOI: https://doi.org/10.1016/j.jpdc.2016.01.004
View in Google Scholar
[2] A. Oussous, F. Z. Benjelloun, A. A. Lahcen, and S. Bel ih, “Big data technologies: A survey”, J. of King Saud Univer. – Comp. and Inform. Sci., vol. 30, no. 4, pp. 431–448, 2018. DOI: https://doi.org/10.1016/j.jksuci.2017.06.001
View in Google Scholar
[3] ETP4HPC Strategic Research Agenda achieving HPC leadership in Europe [Online]. Available: www.etp4hpc.eu
View in Google Scholar
[4] IEEE 802.3az-2010 – IEEE standard for information technology.
View in Google Scholar
[5] Mosix home page [Online]. Available: www.mosix.org
View in Google Scholar
[6] OpenSSI home page [Online]. Available: www.openssi.org/cgi-bin/view?page=openssi.html
View in Google Scholar
[7] Kerrighed home page [Online]. Available: www.kerrighed.org
View in Google Scholar
[8] F. Berman, G. Fox, and A. J. G. Hey, Eds., Grid Computing: Making the Global Infrastructure a Reality. Wiley, 2003 (ISBN: 978-0-470-85319-1). DOI: https://doi.org/10.1002/0470867167.ch0
View in Google Scholar
[9] Unicore home page [Online]. Available: www.unicore.eu
View in Google Scholar
[10] Globus toolkit home page [Online]. Available: toolkit.globus.org
View in Google Scholar
[11] M. Cannataro, Handbook of Research on Computational Grid Technologies for Life Sciences, Biomedicine, and Healthcare. Hershey, PA, USA: IGI Global, 2009 (ISBN-13: 978-1605663746). DOI: https://doi.org/10.4018/978-1-60566-374-6
View in Google Scholar
[12] R. J. Walters, S. Crouch, and P. Bennett, “Building computational grids using ubiquitous Web technologies”, in Collaborative Networks in the Internet of Services. 13th IFIP WG 5.5 Working Conference on Virtual Enterprises, PRO-VE 2012, Bournemouth, UK, October 1-3, 2012. Proceedings, L. M. Camarinha-Matos, L. Xu, and H. Afsarmanesh, Eds. IFIPAICT, vol. 380, pp. 254–261. Berlin, Heidelberg: Springer, 2012.
View in Google Scholar
[13] S. Chaudhary, G. Somani, and R. Buyya, Eds., Research Advances in Cloud Computing. Springer, 2017. DOI: https://doi.org/10.1007/978-981-10-5026-8
View in Google Scholar
[14] N. Sehgal and P. Ch. P. Bhatt, Cloud Computing. Concepts and Practices. Springer, 2018 (ISBN: 978-3-319-77839-6). DOI: https://doi.org/10.1007/978-3-319-77839-6
View in Google Scholar
[15] W.-M. Hwu, Ed., GPU Computing Gems Emerald Edition. Morgan Kaufman, 2011 (ISBN: 9780123849885).
View in Google Scholar
[16] A. B. Singh, J. S. Bhat, R. Raju, and R. D’Souza, “Survey on various load balancing techniques in cloud computing”, Adv. in Comput., vol. 7, no. 2, pp. 28–34, 2017.
View in Google Scholar
[17] A. Thakur and M. S. Goraya, “A taxonomic survey on load balancing in cloud”, J. of Netw. and Comp. Appl., vol. 98, pp. 43–57, 2017. DOI: https://doi.org/10.1016/j.jnca.2017.08.020
View in Google Scholar
[18] J. Zhang et al., “Load balancing in data center networks: A survey”, IEEE Commun. Surv. Tutor., vol. 20, no. 3, pp. 2324–2352, 2018. DOI: https://doi.org/10.1109/COMST.2018.2816042
View in Google Scholar
[19] G. Staples, “TORQUE resource manager”, in Proc. of the 2006 ACM/IEEE Conf. on Supercomput. SC’06, Tampa, FL, USA, 2006, Article no. 8. DOI: https://doi.org/10.1145/1188455.1188464
View in Google Scholar
[20] Portable batch system home page [Online]. Available: www.pbspro.org
View in Google Scholar
[21] Slurm workload manager home page [Online]. Available: slurm.schedmd.com
View in Google Scholar
[22] E. Mohamed and Z. Hong, “Hadoop-mapreduce job scheduling algorithms survey”, in Proc. 2016 7th Int. Conf. on Cloud Comput. and Big Data CCBD 2016, Macau, China, 2016, pp. 237–242, 2016. DOI: https://doi.org/10.1109/CCBD.2016.054
View in Google Scholar
[23] T. White, Hadoop: The Definitive Guide. O’Reilly Media, 2015 (ISBN: 9781491901687).
View in Google Scholar
[24] Apache Hadoop home page [Online]. Available: https://hadoop.apache.org
View in Google Scholar
[25] Apache Spark home page [Online]. Available: spark.apache.org
View in Google Scholar
[26] Apache Storm home page [Online]. Available: storm.apache.org
View in Google Scholar
[27] Apache Flink home page [Online]. Available: https://flink.apache.org
View in Google Scholar
[28] Rapidminer studio home page [Online]. Available: https://rapidminer.com/
View in Google Scholar
[29] Orange home page [Online]. Available: orange.biolab.si
View in Google Scholar
[30] E. Frank, M. A. Hall, and I. H. Witten, Data Mining: Practical Machine Learning Tools and Techniques. Morgan Kaufmann, 2016 (ISBN: 9780123748560).
View in Google Scholar
[31] I. Milne, M. Bayer, L. Cardle, P. Shaw, G. Stephen, F. Wright, and D. Marshall, “Tablet – next generation sequence assembly visualization”, Bioinformatics, vol. 26, no. 3, pp. 401–403, 2010. DOI: https://doi.org/10.1093/bioinformatics/btp666
View in Google Scholar
[32] T. Carver, T. D. Bohme, U. Otto, J. Parkhill, and M. Berriman, “Bamview: viewing mapped read alignment data in the context of the reference sequence”, Bioinformatics, vol. 26, no. 5, pp. 676–673, 2010. DOI: https://doi.org/10.1093/bioinformatics/btq010
View in Google Scholar
[33] K. Rutherford et al., “Artemis: sequence visualization and annotation”, Bioinformatics, vol. 16, no. 10, pp. 944–949, 2000. DOI: https://doi.org/10.1093/bioinformatics/16.10.944
View in Google Scholar
[34] T. Carver, S. R. Harris, M. Berriman, J. Parkhill, and J. A. McQuillan, “Artemis: an integrated platform for visualization and analysis of high-throughput sequence-based experimental data”, Bioinformatics, vol. 28, no. 4, pp. 464–469, 2012. DOI: https://doi.org/10.1093/bioinformatics/btr703
View in Google Scholar
[35] D. Desale, Top tools for social network analysis and visualisation, 2018 [Online]. Available: https://www.kdnuggets.com/software/ social-network-analysis.html
View in Google Scholar
[36] A. Sikora and E. Niewiadomska-Szynkiewicz, “A federated approach to parallel and distributed simulation of complex systems. Int. J. of Appl. Mathem. and Comp. Sci., vol. 17, no. 1, pp. 99–106, 2007. DOI: https://doi.org/10.2478/v10006-007-0009-0
View in Google Scholar
[37] A. Inostrosa-Psijas, V. Gil-Costa, M. Marin, and G. Wainer, “Semiasynchronous approximate parallel DEVS simulation of Web search engines”, Concurr. and Comput.: Pract. and Exper., vol. 30, no. 7, 2018. DOI: https://doi.org/10.1002/cpe.4149
View in Google Scholar
[38] J. A. Miller, M. E. Cotterell, and S. J. Buckley, “Supporting a modeling continuum in scalation: from predictive analytics to simulation modeling”, in Proc. of 2013 Winter Simulations Conference WSC 2013, Washington, DC, USA, 2013, pp. 1191–1202. DOI: https://doi.org/10.1109/WSC.2013.6721507
View in Google Scholar
[39] E. Niewiadomska-Szynkiewicz and A. Sikora, “A software tool for federated simulation of Wireless Sensor Networks and mobile ad hoc networks”, in Applied Parallel and Scientific Computing 10th International Conference, PARA 2010, Reykjavík, Iceland, June 6-9, 2010, Revised Selected Papers, Part I, K. Jónasson, Ed. LNCS, vol. 7133, pp. 303–313. Berlin, Heidelberg: Springer, 2012. DOI: https://doi.org/10.1007/978-3-642-28151-8_30
View in Google Scholar
[40] X. Song, Y. Wu, Y. Ma, Y. Ciu, and G. Gong, “Military simulation big data: Background, state of the art, and challenges”, Mathem. Problems in Engin., vol. 2015, Article ID 298356, pp. 1–20, 2015. DOI: https://doi.org/10.1155/2015/298356
View in Google Scholar
[41] A. K. Fidjeland, E. B. Roesch, M. P. Shanahan, and W. Luk, “Nemo: A platform for neural modelling of spiking neurons using GPUS”, in Proc. 2009 20th IEEE Int. Conf. on Appl.-specif. Syst., Architec. and Process., Boston, MA, USA, 2009, pp. 137–144. DOI: https://doi.org/10.1109/ASAP.2009.24
View in Google Scholar
[42] P. Szynkiewicz, “A novel GPU-enabled simulator for large scale spiking neural networks”, J. of Telecommun. and Inform. Technol., no. 2, pp. 34–42, 2016 [Online]. Available: https://www.itl.waw.pl/ czasopisma/JTIT/2016/2/34.pdf DOI: https://doi.org/10.26636/jtit.2016.2.717
View in Google Scholar
[43] M. A. Martinez-del Amor et al., “Accelerated simulation of P systems on the GPU: A survey”, in Bio-Inspired Computing – Theories and Applications. 9th International Conference, BIC-TA 2014, Wuhan, China, October 16-19, 2014. Proceedings, L. Pan, G. P˘ãun, M. J. Perez-Jim ´ enez, ´ and T. Song, Eds. Communications in Computer and Information Science, vol. 472, pp. 308–312. Springer, 2014.
View in Google Scholar
[44] J. Beyer, M. Hadwiger, and H. Pfister, “A survey of GPU-based large-scale volume visualization” in Proc. of the Eurograph. Conf. on Visual. Eurovis 2014, Swansea, UK, 2014, pp. 1–19.
View in Google Scholar
[45] R. K. V. Maeda et al., “JADE: a heterogeneous multiprocessor system simulation platform using recorded and statistical application models”, in Proc. of the 1st Int. Worksh. on Adv. Interconn. Solutions and Technol. for Emerg. Comput. Syst. AISTECS’16, Prague, Czech Republic, 2016. DOI: https://doi.org/10.1145/2857058.2857066
View in Google Scholar
[46] H. Casanova, A. Giersch, A. Legrand, M. Quinson, and F. Suter, “Versatile, scalable, and accurate simulation of distributed applications and platforms”, J. of Parallel and Distrib. Comput., vol. 74, no. 10, pp. 2899–2917, 2014. DOI: https://doi.org/10.1016/j.jpdc.2014.06.008
View in Google Scholar
[47] R. N. Calheiros, R. Ranjan, A. Beloglazov, C. A. F. De Rose, and R. Buyya, “CloudSim: A toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms”, Software: Pract. and Exper. (SPE), vol. 41, no. 1, pp. 23–50, 2011. DOI: https://doi.org/10.1002/spe.995
View in Google Scholar
[48] Multi2sim workload manager home page [Online]. Available: www.multi2sim.org
View in Google Scholar
[49] Energy Efficiency [Online]. Available: https://ec.europa.eu/energy/ en/topics/energy-efficiency
View in Google Scholar
[50] Code of Conduct for Energy Efficiency in Data Centres [Online]. Available: https://ec.europa.eu/jrc/en/energy-efficiency/ code-conduct/datacentres
View in Google Scholar
[51] Carbon Abatement Handbook [Online]. Available: https://gesi.org/report/detail/carbon-abatement-handbook
View in Google Scholar
[52] M. Avgerinou, P. Bertoldi, and L. Castellazzi, “Trends in data centre energy consumption under the European code of conduct for data centre energy efficiency”, Energies, vol. 10, no. 10, pp. 1–18, 2017. DOI: https://doi.org/10.3390/en10101470
View in Google Scholar
[53] B. Subramaniam, W. Saunders, T. Scogland, and W. Feng, “Trends in energy-efficient computing: A perspective from the Green500”, in 2013 Int. Green Comput. Conf. Proc., Arlington, VA, USA 2013, pp. 1–8. DOI: https://doi.org/10.1109/IGCC.2013.6604520
View in Google Scholar
[54] D. L. Beaty, “Internal IT load profile variability”, ASHRAE J., vol. 55, no. 2, pp. 72–74, 2013.
View in Google Scholar
[55] J. S. Vetter, Contemporary High Performance Computing: From Petascale toward Exascale. Chapman and Hall/CRC Computational Science series, CRC Press, 2013 (ISBN: 9781466568341).
View in Google Scholar
[56] M. P. Karpowicz, P. Arabas, and E. Niewiadomska-Szynkiewicz, “Energy-aware multilevel control system for a network of Linux software routers: Design and implementation”, IEEE Systems J., vol. 12, no. 1, pp. 571–582, 2018. DOI: https://doi.org/10.1109/JSYST.2015.2489244
View in Google Scholar
[57] ETSI ES 203 237 v1.1.1 (2014-03) standard. [Online]. Available: www.etsi.org
View in Google Scholar
[58] L. A. Barroso and U. Holzle, “The case for energy-proportional computing”, Computer, vol. 40, no. 12, pp. 33–37, 2007. DOI: https://doi.org/10.1109/MC.2007.443
View in Google Scholar
[59] H. Lim, A. Kansal, and J. Liu, “Power budgeting for virtualized data centers”, in Proc. of the 2011 USENIX Ann. Tech. Conf. USENIX ATC’11, Portland, OR, USA, 2011.
View in Google Scholar
[60] L. Chiaraviglio, M. Mellia, and F. Neri, “Reducing power consumption in backbone networks”, in Proc. of the 2009 IEEE Int. Conf. on Commun. ICC’09, Piscataway, NJ, USA, 2009, pp. 2298–2303. DOI: https://doi.org/10.1109/ICC.2009.5199404
View in Google Scholar
[61] M. Karpowicz, “Energy-efficient CPU frequency control for the Linux system”, Concurr. and Comput.: Pract. and Exper., vol. 28, no. 2, pp. 420–437, 2016. DOI: https://doi.org/10.1002/cpe.3476
View in Google Scholar
[62] R. Bolla et al., “Large-scale validation and benchmarking of a network of power-conservative systems using ETSI’s green abstraction layer”, Trans. on Emerg. Telecommun. Technol., vol. 27, no. 3, pp. 451–468, 2016. DOI: https://doi.org/10.1002/ett.3006
View in Google Scholar
[63] P. Arabas, “Energy aware data centers and networks: a survey”, J. of Telecommun. and Inform. Technol.”, no. 4, pp. 26–36, 2019. DOI: https://doi.org/10.26636/jtit.2018.129818
View in Google Scholar
[64] A. Y. Zomaya and J. Ch. Lee, Eds., Energy-Efficient Distributed Computing Systems. Wiley, 2012 (ISBN: 978-0-470-90875-4).
View in Google Scholar
[65] J. C. McCullough, Y. Agarwal, J. Chandrashekar, S. Kuppuswamy, A. C. Snoeren, and R. K. Gupta, “Evaluating the effectiveness of model-based power characterization”, in in Proc. of the 2011 USENIX Ann. Tech. Conf. USENIX ATC’11, Portland, OR, USA, 2011.
View in Google Scholar
[66] J.-M. Pierson, Large-scale Distributed Systems and Energy Efficiency: A Holistic View. Wiley, 2015 (ISBN: 9781118864630). DOI: https://doi.org/10.1002/9781118981122
View in Google Scholar
[67] S. Mittal, “A survey of architectural techniques for DRAM power management”, Int. J. of High Perform. Syst. Archit., vol. 4, no. 2, pp. 110–119, 2012. DOI: https://doi.org/10.1504/IJHPSA.2012.050990
View in Google Scholar
[68] K. Chalmers et al., Eds., Communicating Process Architectures 2015 & 2016: WoTUG-37 & WoTUG-38. Concurrent Systems Engineering Series, vol. 69. IOS Press, 2018 (ISBN 978-1-61499-885-3).
View in Google Scholar
[69] R. Bolla, R. Bruschi, A. Carrega, and F. Davoli, “Green network technologies and the art of trading-off”, in Proc. 2011 IEEE Conf. on Comp. Commun. Worksh. INFOCOM WKSHPS 2011, Shanghai, China, 2011, pp. 301–306. DOI: https://doi.org/10.1109/INFCOMW.2011.5928827
View in Google Scholar
[70] V. Pallipadi, S. Li, and A. Belay, “cpuidle: Do nothing, efficiently...”, in Proc. Linux Symposium, Ottawa, Ontario, Canada, 2007, vol. 2, pp. 119–125.
View in Google Scholar
[71] V. Pallipadi and A. Starikovskiy, “The ondemand governor”, in Proc. Linux Symposium, Ottawa, Ontario, Canada, 2006, vol. 2, pp. 215–230.
View in Google Scholar
[72] M. Karpowicz, E. Niewiadomska-Szynkiewicz, P. Arabas, and A. Sikora, “Energy and power efficiency in cloud”, in Resource Management for Big Data Platforms: Algorithms, Modelling, and High-Performance Computing Techniques, F. Pop, J. Kołodziej, and B. Di Martino, Eds. Springer, 2016, pp. 97–127. DOI: https://doi.org/10.1007/978-3-319-44881-7_6
View in Google Scholar
[73] I. Manousakis, M. Marazakis, and A. Bilas, “FDIO: A feedback driven controller for minimizing energy in I/O-intensive applications”, in Proc. of the 5th USENIX Conf. on Hot Topics in Storage and File Syst. HotStorage’13, San Jose, CA, USA 2013.
View in Google Scholar
[74] M. Kondo, H. Sasaki, and H. Nakamura, “Improving fairness, throughput and energy-efficiency on a chip multiprocessor through DVFs”, SIGARCH Comp. Archit. News, vol. 35, no. 1, pp. 31–38, 2007. DOI: https://doi.org/10.1145/1241601.1241609
View in Google Scholar
[75] Q. Zhang, L. Cheng, and R. Boutaba, “Cloud computing: state-of-the-art and research challenges”, J. of Internet Serv. and Appl., vol. 1, no. 1, pp. 7–18, DOI: https://doi.org/10.1007/s13174-010-0007-6
View in Google Scholar
[76] H. Jung and M. Pedram, “Supervised learning based power management for multicore processors”, IEEE Trans. on Comp.-Aided Design of Integr. Circ. and Syst., vol. 29, no. 9, pp. 1395–1408, 2010. DOI: https://doi.org/10.1109/TCAD.2010.2059270
View in Google Scholar
[77] J. Howard et al., “A 48-core IA-32 processor in 45 nm CMOS using on-die message-passing and DVFs for performance and power scaling”, IEEE J. of Solid-State Circ., vol. 46, no. 1, pp. 173–183, 2011. DOI: https://doi.org/10.1109/JSSC.2010.2079450
View in Google Scholar
[78] M. E. Salehi et al., “Dynamic voltage and frequency scheduling for embedded processors considering power/performance tradeoffs”, IEEE Trans. on Very Large Scale Integr. (VLSI) Syst., vol. 19, no. 10, pp. 1931–1935, 2011. DOI: https://doi.org/10.1109/TVLSI.2010.2057520
View in Google Scholar
[79] M. Karpowicz, P. Arabas, and E. Niewiadomska-Szynkiewicz, “Design and implementation of energy-aware application-specific CPU frequency governors for the heterogeneous distributed computing systems”, Future Gener. Comp. Syst., vol. 78, pp. 302–315, 2018. DOI: https://doi.org/10.1016/j.future.2016.05.011
View in Google Scholar
[80] L. Wang and S. U. Khan, “Review of performance metrics for green data centers: a taxonomy study”, J. of Supercomput., vol. 63, no. 3, pp. 639–656, 2003. 2010. DOI: https://doi.org/10.1007/s11227-011-0704-3
View in Google Scholar
[81] I. T. Cotes-Ruiz et al., “Dynamic voltage frequency scaling simulator for real workflows energy-aware management in green cloud computing”, PLoS ONE, vol. 12, no. 1, 2017. DOI: https://doi.org/10.1371/journal.pone.0169803
View in Google Scholar
[82] Y. Chiang, Y. Ouyang, and C. Hsu, “An efficient green control algorithm in cloud computing for cost optimization”, IEEE Trans. on Cloud Comput., vol. 3, no. 2, pp. 145–155, 2015. DOI: https://doi.org/10.1109/TCC.2014.2350492
View in Google Scholar
[83] E. Niewiadomska-Szynkiewicz, A. Sikora, P. Arabas, M. Kamola, M. Mincer, and J. Kołodziej, “Dynamic power management in energy-aware computer networks and data intensive systems”, Future Gener. Comp. Syst., vol. 37, pp. 284–296, 2014. DOI: https://doi.org/10.1016/j.future.2013.10.002
View in Google Scholar
[84] J. Kołodziej, S. Khan, L. Wang, and A. Zomaya, “Energy efficient genetic-based schedulers in computational grids”, Concurr. and Comput.: Pract. and Exper., vol. 27, no. 4, pp. 809–829, 2015. DOI: https://doi.org/10.1002/cpe.2839
View in Google Scholar
[85] M. Kamola and P. Arabas, “Shortest path green routing and the importance of traffic matrix knowledge”, in Proc. 2013 24th Tyrrhenian Int. Worksh. on Digit. Commun. – Green ICT TIWDC 2013, Genoa, Italy, 2013. DOI: https://doi.org/10.1109/TIWDC.2013.6664215
View in Google Scholar
[86] K. Govindarajan, V. S. Kumar, and T. S. Somasundaram, “A distributed cloud resource management framework for highperformance computing (HPC) applications”, in Proc. 2016 8th Int. Conf. on Adv. Comput. ICoAC 2017, Chennai, India, 2017, pp. 1–6. DOI: https://doi.org/10.1109/ICoAC.2017.7951735
View in Google Scholar
[87] E. Niewiadomska-Szynkiewicz and P. Arabas, “Resource management system for HPC computing”, in Automation 2018. Advances in Automation, Robotics and Measurement Techniques, R. Szewczyk, C. Zieliński, and M. Kaliczyńska, Eds. Advances in Intelligent Systems and Computing, vol. 743, pp. 52–61. Springer, 2018. DOI: https://doi.org/10.1007/978-3-319-77179-3_5
View in Google Scholar
[88] L.-D. Radu, “Green cloud computing: A literature survey”, Symmetry, vol. 9, no. 12, pp. 1–20, 2017. DOI: https://doi.org/10.3390/sym9120295
View in Google Scholar
[89] N. Akhter and M. Othman, “Energy aware resource allocation of cloud data center: review and open issues”, Cluster Comput., vol. 19, no. 3, pp. 1163–1182, 2016. DOI: https://doi.org/10.1007/s10586-016-0579-4
View in Google Scholar
[90] T. Mastelic, A. Oleksiak, H. Claussen, I. Brandic, J.-M. Pierson, and A. Vasilakos, “Cloud computing: survey on energy efficiency”, ACM Comput. Surv., vol. 47, no. 2, Article no. 33, 2015. DOI: https://doi.org/10.1145/2656204
View in Google Scholar
Downloads
Submitted
Published
Issue
Section
License
Copyright (c) 2019 Journal of Telecommunications and Information Technology
This work is licensed under a Creative Commons Attribution 4.0 International License.
