Ant colony optimization with fuzzy-based ensemble of heuristics for ensemble feature selection

Document Type : Research Paper

Authors

1 Department of Computer Engineering, Faculty of Engineering, Lorestan University, Khorramabad, Iran

2 Department of Computer Engineering, Isfahan University of Technology, Isfahan, Iran

Abstract

One of the crucial stages in machine learning in high-dimensional datasets is feature selection. Unrelated features weaknesses the efficiency of the model. However, merging several feature selection strategies is routine to solve this problem, the way to integrate feature selection methods is problematic. This paper presents a new ensemble of heuristics through fuzzy Type-I based on Ant Colony Optimization (ACO) for ensemble feature selection named Ant-EHFS. At first, three feature selection methods are run; then, the Euclidean Distance between each pair of features is computed as a heuristic (an M×M matrix is constructed), that M is the total of features. After that, a Type-I fuzzy is used individually to address various feature selections' uncertainty and estimate trustworthiness for each feature, as another heuristic. A complete weighted graph based on combining the two heuristics is then built; finally, ACO is applied to the complete graph for finding features that have the highest relevance together in the features space, which in each ant considers the reliability rate and Euclidean Distance of the destination node together for moving between nodes of the graph. Five and eight robust and well-known ensemble feature selection methods and primary feature selection methods, respectively, have been compared with Ant-EHFS on six high-dimensional datasets to show the proposed method's performance. The results have shown that the proposed method outperforms five ensemble feature selection methods and eight primary feature selections in Accuracy, Precision, Recall, and F1-score metrics.

Keywords

References

[1] Laith Mohammad Abualigah and Ahamad Tajudin Khader. “Unsupervised text feature selection technique based on hybrid particle swarm optimization algorithm with genetic operators for the text clustering”. In: The Journal of Supercomputing 73.11 (2017), pp. 4773–4795. issn: 0920-8542.
[2] Laith Mohammad Abualigah, Ahamad Tajudin Khader, and Essam Said Hanandeh. “A new feature selection method to improve the document clustering using particle swarm optimization algorithm”. In: Journal of Computational Science 25 (2018), pp. 456–466. issn: 1877-7503.
[3] Laith Mohammad Qasim Abualigah and Essam S Hanandeh. “Applying genetic algorithms to information retrieval using vector space model”. In: International Journal of Computer Science, Engineering and Applications 5.1 (2015), p. 19.  issn: 2231-0088.
[4] Amir Hossein Atashkar, Nasser Ghadiri, and Mehdi Joodaki. “Linked data partitioning for RDF processing on Apache Spark”. In: 2017 3th International Conference on Web Research (ICWR). IEEE, pp. 73–77. isbn: 1538604205.
[5] Peter N. Belhumeur, Joao P Hespanha, and David J. Kriegman. “Eigenfaces vs. fisherfaces: Recognition using class specific linear projection”.In: IEEE Transactions on pattern analysis and machine intelligence 19.7 (1997), pp. 711–720.
[6] Arindam Bhattacharjee et al. “Classification of human lung carcinomas by mRNA expression profiling reveals distinct adenocarcinoma subclasses”. In: Proceedings of the National Academy of Sciences 98.24 (2001), pp. 13790–13795.
[7] C Alan Boneau. “The effects of violations of assumptions underlying the t test”. In: Psychological bulletin 57.1 (1960), p. 49. issn: 1939-1455.
[8] Paul S Bradley and Olvi L Mangasarian. “Feature selection via concave minimization and support vector machines.” In: ICML. Vol. 98. Citeseer.1998, pp. 82–90.
[9] Elias F Combarro et al. “Introducing a family of linear measures for feature selection in text categorization”. In: IEEE transactions on Knowledge and Data Engineering 17.9 (2005), pp. 1223–1232.
[10] Marco Dorigo, Mauro Birattari, and Thomas Stutzle. “Ant colony optimization”. In: IEEE computational intelligence magazine 1.4 (2006), pp. 28–39. issn: 1556-603X.
[11] Marco Dorigo and Gianni Di Caro. “Ant colony optimization: a new meta-heuristic”. In: Proceedings of the 1999 congress on evolutionary computation-CEC99 (Cat. No. 99TH8406). Vol. 2. IEEE. 1999, pp. 1470–1477.
[12] Mohammad Bagher Dowlatshahi, Vali Derhami, and Hossein Nezamabadipour. “Ensemble of filter-based rankers to guide an epsilon-greedy swarm optimizer for high-dimensional feature subset selection”. In: Information 8.4 (2017), p. 152.
[13] Mohammad Bagher Dowlatshahi, Vali Derhami, and Hossein Nezamabadipour. “A novel three-stage filter-wrapper framework for miRNA subset selection in cancer classification”. In: Informatics. Vol. 5. 1. Multidisciplinary Digital Publishing Institute. 2018, p. 13.[14] Mohammad Bagher Dowlatshahi and Hossein Nezamabadi-Pour. “GGSA:
a grouping gravitational search algorithm for data clustering”. In: Engineering Applications of Artificial Intelligence 36 (2014), pp. 114–121.issn: 0952-1976.
[15] Mohammad Bagher Dowlatshahi, Hossein Nezamabadi-Pour, and Mashaallah Mashinchi. “A discrete gravitational search algorithm for solving combinatorial optimization problems”. In: Information Sciences 258 (2014), pp. 94–107. issn: 0020-0255.
[16] Mohammad Bagher Dowlatshahi, Hossein Nezamabadi-Pour, and Mashaallah Mashinchi. “A discrete gravitational search algorithm for solving combinatorial optimization problems”. In: Information Sciences 258 (2014),
pp. 94–107.
[17] Mohammad Bagher Dowlatshahi and Mehdi Rezaeian. “Training spiking neurons with gravitational search algorithm for data classification”. In: 2016 1st Conference on Swarm Intelligence and Evolutionary Computation (CSIEC). IEEE, pp. 53–58. isbn: 1467387371.
[18] Peter Drot´ar, Matej Gazda, and Liberios Vokorokos. “Ensemble feature selection using election methods and ranker clustering”. In: Information Sciences 480 (2019), pp. 365–380.
[19] Richard O Duda, Peter E Hart, et al. Pattern classification. John Wiley & Sons, 2006.
[20] Mahdi Eftekhari et al. How Fuzzy Concepts Contribute to Machine Learning. 2022.
[21] David E Goldberg. “Genetic algorithms in search”. In: Optimization, and MachineLearning (1989).
[22] Gongde Guo et al. “KNN model-based approach in classification”. In: OTM Confederated International Conferences” On the Move to Meaningful Internet Systems”. Springer. 2003, pp. 986–996.
[23] Isabelle Guyon and Andr´e Elisseeff. “An introduction to variable and feature selection”. In: Journal of machine learning research 3.Mar (2003), pp. 1157–1182.
[24] Amin Hashemi, Mohammad Bagher Dowlatshahi, and Hossein Nezamabadipour. “A Pareto-based ensemble of feature selection algorithms”. In: Expert Systems with Applications (2021), p. 115130. issn: 0957-4174. doi:https://doi.org/10.1016/j.eswa.2021.115130. url: https://www.sciencedirect.com/science/article/pii/S0957417421005716.
[25] Amin Hashemi and Mohammad Bagher Dowlatshahi. “MLCR: a fast multi-label feature selection method based on K-means and L2-norm”. In: 2020 25th International Computer Conference, Computer Society of Iran (CSICC). IEEE, pp. 1–7. isbn: 1728159377.
[26] Amin Hashemi, Mohammad Bagher Dowlatshahi, and Hossein Nezamabadipour. “MFS-MCDM: Multi-label feature selection using multi-criteria decision making”. In: Knowledge-Based Systems 206 (2020), p. 106365. issn: 0950-7051.
[27] Amin Hashemi, Mohammad Bagher Dowlatshahi, and Hossein Nezamabadipour. “MGFS: A multi-label graph-based feature selection algorithm via PageRank centrality”. In: Expert Systems with Applications 142 (2020), p. 113024. issn: 0957-4174.
[28] Amin Hashemi et al. “Ant Colony Optimization equipped with an ensemble of heuristics through Multi-Criteria Decision Making: A case study in ensemble feature selection”. In: Applied Soft Computing (2022), p. 109046.
[29] Nazrul Hoque, Dhruba K Bhattacharyya, and Jugal K Kalita. “MIFSND: A mutual information-based feature selection method”. In: Expert Systems with Applications 41.14 (2014), pp. 6371–6385. issn: 0957-4174.
[30] Mehdi Joodaki, Mohammad Bagher Dowlatshahi, and Nazanin Zahra Joodaki. “An ensemble feature selection algorithm based on PageRank centrality and fuzzy logic”. In: Knowledge-Based Systems (2021), p. 107538. issn: 0950-7051.
[31] Mehdi Joodaki, Nasser Ghadiri, and Amir Hossein Atashkar. “Protein complex detection from PPI networks on Apache Spark”. In: 2017 9th International Conference on Information and Knowledge Technology (IKT). IEEE, pp. 111–115. isbn: 1538625474.
[32] Mehdi Joodaki et al. “A scalable random walk with restart on heterogeneous networks with Apache Spark for ranking disease-related genes through type-II fuzzy data fusion”. In: Journal of Biomedical Informatics (2021), p. 103688. issn: 1532-0464.
[33] Nazanin Zahra Joodaki, Mohammad Bagher Dowlatshahi, and Mehdi Joodaki. “A novel ensemble feature selection method through Type I fuzzy”. In: 2022 9th Iranian Joint Congress on Fuzzy and Intelligent Systems (CFIS). IEEE. 2022, pp. 1–6.
[34] Alan Jovi´c, Karla Brki´c, and Nikola Bogunovi´c. “A review of feature selection methods with applications”. In: 2015 38th international convention on information and communication technology, electronics and microelectronics (MIPRO). Ieee. 2015, pp. 1200–1205.
[35] Hamidreza Rashidy Kanan and Karim Faez. “An improved feature selection method based on ant colony optimization (ACO) evaluated on face recognition system”. In: Applied Mathematics and Computation 205.2(2008), pp. 716–725. issn: 0096-3003.
[36] Igor Kononenko. “Estimating attributes: analysis and extensions of RELIEF”. In: European conference on machine learning. Springer, pp. 171–182.
[37] Cosmin Lazar et al. “A survey on filter techniques for feature selection in gene expression microarray analysis”. In: IEEE/ACM transactions on computational biology and bioinformatics 9.4 (2012), pp. 1106–1119.
[38] Jimin Lee, Nomin Batnyam, and Sejong Oh. “RFS: Efficient feature selection method based on R-value”. In: Computers in biology and medicine 43.2 (2013), pp. 91–99.
[39] Ondrej Linda and Milos Manic. “Interval type-2 fuzzy voter design for fault tolerant systems”. In: Information Sciences 181.14 (2011), pp. 2933–2950.
[40] Michael J Lyons, Julien Budynek, and Shigeru Akamatsu. “Automatic classification of single facial images”. In: IEEE transactions on pattern analysis and machine intelligence 21.12 (1999), pp. 1357–1362.
[41] Rammohan Mallipeddi et al. “Differential evolution algorithm with ensemble of parameters and mutation strategies”. In: Applied soft computing 11.2 (2011), pp. 1679–1696. issn: 1568-4946.
[42] Adele Mehranfar et al. “A Type-2 fuzzy data fusion approach for building reliable weighted protein interaction networks with application in protein complex detection”. In: Computers in biology and medicine 88 (2017), pp. 18–31. issn: 0010-4825.
[43] Majid Mohammadi and Jafar Rezaei. “Ensemble ranking: Aggregation of rankings produced by different multi-criteria decision-making methods”. In: Omega 96 (2020), p. 102254. issn: 0305-0483.
[44] Seyed Vahid Moravvej et al. “A method Based on an Attention Mechanism to Measure the Similarity of two Sentences”. In: 2021 7th International Conference on Web Research (ICWR). 2021, pp. 238–242. doi: 10.1109/ICWR51868.2021.9443135.
[45] SV Moravvej et al. “Efficient GAN-based method for extractive summarization”. In: Journal of Electrical and Computer Engineering Innovations (JECEI) (2021).
[46] Songyot Nakariyakul. “High-dimensional hybrid feature selection using interaction information-guided search”. In: Knowledge-Based Systems 145 (2018), pp. 59–66.
[47] Mohsen Paniri, Mohammad Bagher Dowlatshahi, and Hossein Nezamabadipour. “MLACO: A multi-label feature selection algorithm based on ant colony optimization”. In: Knowledge-Based Systems 192 (2020), p. 105285. issn: 0950-7051.
[48] Marjan Kuchaki Rafsanjani and Mohammad Bagher Dowlatshahi. “Using gravitational search algorithm for finding near-optimal base station location in two-tiered WSNs”. In: International Journal of Machine Learning and Computing 2.4 (2012), p. 377.
[49] Ferdinando S Samaria and Andy C Harter. “Parameterisation of a stochastic model for human face identification”. In: Proceedings of 1994 IEEE workshop on applications of computer vision. IEEE. 1994, pp. 138–142.
[50] Samaneh Samiei, Mehdi Joodaki, and Nasser Ghadiri. “A Scalable Pattern Mining Method Using Apache Spark Platform”. In: 2021 7th International Conference on Web Research (ICWR). IEEE, pp. 114–118. isbn: 1665404264.
[51] Sadia Sharmin et al. “Simultaneous feature selection and discretization based on mutual information”. In: Pattern Recognition 91 (2019), pp. 162–174. issn: 0031-3203.
[52] Sina Tabakhi and Parham Moradi. “Relevance–redundancy feature selection based on ant colony optimization”. In: Pattern recognition 48.9 (2015), pp. 2798–2811. issn: 0031-3203.
[53] Sina Tabakhi, Parham Moradi, and Fardin Akhlaghian. “An unsupervised feature selection algorithm based on ant colony optimization”. In: Engineering Applications of Artificial Intelligence 32 (2014), pp. 112–123. issn: 0952-1976.
[54] Sina Tabakhi, Parham Moradi, and Fardin Akhlaghian. “An unsupervised feature selection algorithm based on ant colony optimization”. In: Engineering Applications of Artificial Intelligence 32 (2014), pp. 112–123. issn: 0952-1976.
[55] El-Ghazali Talbi. Metaheuristics: from design to implementation. Vol. 74. John Wiley Sons, 2009. isbn: 0470496908.
[56] Youchuan Wan et al. “A feature selection method based on modified binary coded ant colony optimization algorithm”. In: Applied Soft Computing 49 (2016), pp. 248–258. issn: 1568-4946.
[57] Guangfen Wei et al. “A novel hybrid feature selection method based on dynamic feature importance”. In: Applied Soft Computing 93 (2020), p. 106337. issn: 1568-4946.
[58] Ian H Witten and Eibe Frank. “Data mining: practical machine learning tools and techniques with Java implementations”. In: Acm Sigmod Record 31.1 (2002), pp. 76–77.
[59] Guohua Wu, Rammohan Mallipeddi, and Ponnuthurai Nagaratnam Suganthan. “Ensemble strategies for population-based optimization algorithms–A survey”. In: Swarm and evolutionary computation 44 (2019), pp. 695–711. issn: 2210-6502.
[60] Xiao Yu, Qing Li, and Jin Liu. “Scalable and parallel sequential pattern mining using spark”. In: World Wide Web 22.1 (2019), pp. 295–324. issn: 1573-1413.
[61] Jixiong Zhang, Yanmei Xiong, and Shungeng Min. “A new hybrid filter/wrapper algorithm for feature selection in classification”. In: Analytica chimica acta 1080 (2019), pp. 43–54.
[62] Wenhao Zhao, Li Yan, and Yunsheng Zhang. “Geometric-constrained multi-view image matching method based on semi-global optimization”. In: Geo-spatial information science 21.2 (2018), pp. 115–126. issn: 1009-5020.

Files

History

  • Receive Date: 18 February 2022
  • Revise Date: 29 April 2022
  • Accept Date: 11 July 2022

Share

How to cite

Statistics