Е-Публикации
Технически университет - София

Abstract: Published by Elsevier B.V.This paper introduces a novel family of activation functions using hypergeometric functions with trainable parameters. Hypergeo-metric functions possess a complex series structure that may create numerical issues. Therefore, we focus on Bessel-type functions of the first kind. The Bessel functions are a subfamily of hypergeometric functions, with characteristics similar to ReLU and sinusoidal activation functions. We investigate the performance of Bessel functions as activation functions within Convolutional Neural Networks (CNNs). Comparative analyses are conducted against established activation functions such as ReLU and sigmoid on benchmark datasets, including MNIST, Happy House, and CIFAR-10. This research aims to explore the potential of Bessel functions as a general class of parametric activation functions adaptable to diverse network architectures.

References

1. Abramowitz M, Stegun IA."Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables." 10th ed. New York: Wiley-Interscience Publication, John Wiley & Sons; 1972.
2. Bilgili E, Göknar CI, Ucan ON."Cellular neural network with trapezoidal activation function." Int J Circuit Theory Appl. 2005;33(5):393–417.
3. Bingham G, Miikkulainen R."Discovering parametric activation functions." ArXiv:2006.03179v4. Available from: https://arxiv.org/abs/2006.03179
4. Clevert DA, Unterthiner T, Hochreiter S."Fast and accurate deep network learning by exponential linear units (ELUs)." In: Proc. 4th Int. Conf. Learning Representations (ICLR); 2016. Article No. 149803.
5. He K, Zhang X, Ren S, Sun J."Delving deep into rectifiers: surpassing human-level performance on ImageNet classification." In: Proc. IEEE Int. Conf. Computer Vision (ICCV); 2015. p. 1026–1034.
6. Jalab HA, Ibrahim RW."New activation functions for complex-valued neural network." Int J Phys Sci. 2011;6(7):1766–1772.
7. Klambauer G, Unterthiner T, Mayr A, Hochreiter S."Self-normalizing neural networks." Adv Neural Inf Process Syst. 2017;2017:972–981.
8. Maas A, Hannum AY, Ng AY."Rectifier nonlinearities improve neural network acoustic models." In: Proc. ICML; 2013. 30(1):Article No. 3.
9. Maguolo G, Nanni L, Ghidoni S."Ensemble of convolutional neural networks trained with different activation functions." Expert Syst Appl. 2021;166:Article No. 114048.
10. Manessi F, Rozza A."Learning combinations of activation functions." In: Proc. 24th Int. Conf. Pattern Recognit. (ICPR); 2018. p. 61–66.
11. Nair V, Hinton GE."Rectified linear units improve restricted Boltzmann machines." In: Proc. 27th Int. Conf. Machine Learning (ICML 2010); 2010. p. 807–814.
12. Özdemir N, Iskender B, Özgür NY."Complex-valued neural network with Möbius activation function." Commun Nonlinear Sci Numer Simul. 2011;16(12):4698–4703.
13. Pearson J."Computation of hypergeometric functions" [MSc thesis]. Oxford (UK): Univ. Oxford; 2009.
14. Ramachandran P, Zoph B, Le QV."Searching for activation functions." ArXiv:1710.05941. Available from: https://arxiv.org/abs/1710.05941
15. Sathasivam S."Boltzmann machine and new activation function comparison." Appl Math Sci. 2011;5(78):3853–3860.
16. Vieira N."Bicomplex neural networks with hypergeometric activation functions." Adv Appl Clifford Algebr. 2023;33(22):Article No. 20.
17. Vieira N."Quaternionic convolutional neural networks with trainable Bessel activation functions." Complex Anal Oper Theory. 2023;17(6):Ar-ticle No. 82.
18. "CIFAR-10 and CIFAR-100 Datasets" [Internet]. Accessed: 2024 Nov 12. Available from: https://www.cs.toronto.edu/~kriz/cifar. html
19. "Happy House Dataset" [Internet]. Accessed: 2024 Mar 12. Available from: https://www.kaggle.com/datasets/iarunava/happy-house-dataset/data
20. "MNIST Database" [Internet]. Accessed: 2024 Nov 12. Available from: http://yann.lecun.com/exdb/mnist/

Issue