Big data has become a part of the industrial revolution 4.0 which is an evolution of the previous technology [1]. The technology is business intelligence (BI), which was developed with the invention of Internet of Things (IoT) technology that allows human and machine interaction through the internet and IoT data visualization which plays an important role in improving decision making [2]. 

The development of the internet and digitalization transformation is the beginning of a breakthrough of financial technology which is used as the main strategy to expand financial inclusion which will support economic expansion as well as improve people's financial welfare [3]. Fintech tends to influence people's current financial patterns and behavior, especially after the Covid-19 lockdown and fintech services have become very competitive [4].

Mobile banking is one of the innovations in the banking industry that is able to eliminate the limitations of time and space in banking transactions [5]. Bank Indonesia is targeting digital transactions in 2022 to reach IDR 51,729 T. Bank Indonesia continues to expand BI-FAST services through mobile banking and to improve communication with the public and other relevant authorities. Bank Indonesia recorded mobile banking transaction volume reaching 3.2 billion in May 2022. This amount increased 67.87% year by year from the same level of 1.9 billion transactions. Based on the survey from the Top Brand Award, mobile banking in Indonesia has changed and had 4 most widely used mobile bankings, namely BCA Mobile, BRImo, Livin' by Mandiri and BNI Mobile. Currently, BCA Mobile still dominates as the Top Brand Award since 2018 until now. When this is viewed based on the rating of mobile banking applications on the Google Play Store, BCA mobile is the mobile banking application with the highest rating of 4.7/5.0. The four mobile banking are owned by the Bank which is included in KBMI 4 according to the OJK, namely the Bank Group based on Core Capital with the highest group that has assets of more than IDR 70 T.

In evaluating and innovating the quality of a company's products or services, user or consumer reviews are very important for the company [6]. In this case, user reviews and ratings can be used as sentiment analysis tools to find the sentiment or reaction of users to assess performance and make improvements that will ultimately influence the decision [7]. This is important to allow application developers to use ratings and reviews in improving their performance [8].           

Sentiment analysis is used to find out an individual's sentiment or feelings about something. So, in this case, Signalling Theory can help to explain the behavior between two parties when they have access to different information. Signalling Theory itself is related to the asymmetry of information between two parties, namely companies and consumers. This is because of information becomes a crucial factor in decision making [9]. 

Several researches have examined how sentiment analysis in a mobile banking uses various methods. Research conducted by Leem and Eum [10], which discussed about Kakao Mobile Bank application in Korea showed a high positive sentiment given by its users with 78.2%. Another study conducted by Brunova and Bidulya which investigated bank reviews in Rusia also showed more negative sentiment than positive sentiment. As claimed by Mittal and Agrawal  sentiment analysis is able to identify the attributes of banking services, positive sentiment, negative sentiment, as well as emotions expressed in online customer reviews at several banks (HSBC, Standard Chartered, Citi, HDFC, ICICI, Kotak Mahindra, IndusInd, IDBI, DCB, Axis, SBI, PNB, BOB, CBI).

In Indonesia, sentiment analysis research on mobile banking was also conducted by some researchers. Research conducted by Nirwandani et al. [11], showed that online Mandiri application containing more negative sentiments. Besides, Wijanarto and Brilianti [12], also conducted research on BNI Mobile which has more negative sentiment than positive sentiment. The results of these research revealed that most of mobile banking applications having negative sentiment. In this regard, the researchers believe that no research have discussed about the way how to make mobile banking applications have positive sentiment. In line with that, Shankar et al. [13], proposed several things that have contribution to the success of mobile banking applications, namely navigation, privacy, convenience, efficiency, security and support to customers. Therefore, the success of mobile banking application can be shown based on these several things by using sentiment analysis. 

Sentiment analysis study can be carried out using several methods. In this case, Onantya et al. investigated the BCA Mobile application by using several research methods namely BM25 and Improved K-Nearest Neighbor which concluded that the accuracy results obtained from the method fluctuates. Meanwhile, research conducted by Sepri discussed sentiment analysis at Bank Muamalat using the Naïve Bayes Algorithm with an accuracy value of 87%. Thus, the previous research before showed how sentiment analysis study carried out by using several methods. Unfortunately, there is no research that compares the performance of mobile banking through sentiment analysis, thus, the researchers believe that it is necessary to be investigated. Therefore, comparing the performance of 4 Indonesian mobile banking applications using sentiment analysis is the purpose of this research. 

This article is consisted of five main discussions, the first part is an introduction that represents the reasons and motivations of writing this article. The second part describes comprehensively the methodology used. The third and fourth parts are presented the empirical results as well as the discussion of those results. Meanwhile, the last part is presented conclusions aa well as the practical and empirical advice for policymakers and subsequent researchers.

This study has the main objective of measuring the performance of mobile banking based on user sentiment analysis. This study used the Knowledge Discovery in Database (KDD) methodology. KDD is the process of extracting information and knowledge from a large amount of data [14]. The dataset used in January-September 2022 totaled 72.027 in the form of review texts or reviews obtained through web scraping on 4 Mobile Banking applications on the Google Play Store which were divided into 2 classifications (positive and negative). In this regard, the review data which has a rating 1 and 2 is classified as negative. Meanwhile, the review data which has a rating 4 and 5 is classified as positive. The dataset reviews were obtained on BCA Mobile as many as 16.375 reviews, on BRImo as many as 20.248 reviews, on Livin' by Mandiri as many as 13.483 reviews and on BNI Mobile as many as 21.921 reviews. Based on the results of data labeling, there was a problem with imbalance in the dataset. This is solved by performing The Synthetic Minority Oversampling Technique (SMOTE). Thus, the data obtained on BCA Mobile for each positive and negative class was 8.074, on BRImo was 8.440, on Livin' by Mandiri was 10.292 and BNI Mobile was 5.543. The modeling process was carried out using the Naïve Bayes Classifier Algorithm using Python programming on Google Colab. The detail about the research process implemented is shown in Figure 1.

Figure 1: Research Process Flow

Data Preprocessing

This preprocessing aims to structure and equalize data by reducing the volume of vocabulary, especially in characters other than letters. In this study, the researchers used 4 stages in carrying out the text processing process. First, Case Folding in text preprocessing was done to equalize the characters in the word, capital letters were changed to lowercase letters and punctuation marks other than letters were omitted [15]. Second, Tokenizing is the stage of truncating a string or input based on the words that compose it or tokenizing can also be said to be the process of breaking sentences into words (tokens) and discarding punctuation characters, hastags, URLs and so on[16]. Third, Filtering is the process of removing meaningless or unimportant words [17]. Filtering was done using the libraries found in Python. Fourth, Stemming is the process of finding the root of a word or base word and removing its affix [15]. The stemming Indonesian used was a literary Library prepared for the Python Programming Language.

Data Clean

The clean data is obtained after examining a series of preprocessing stages as shown in Table 1.

Weighting TF-IDF

Term Frequency Inverse Document Frequency (TF-IDF) is a method used to determine how relevant a word (term) is to a document by giving weight to each word [18].

Splitting Data

This study divided training data and testing data in a ratio of 80:20 of the total data, as much as 80% was used as training data and 20% for testing data.

Table 1: Data Clean

Algorithm Naïve Bayes Classifier

The Naïve Bayes Algorithm is a machine learning technique in text data mining that uses probability and statistical calculations to predict future probabilities based on the past experience [19]. The clean data obtained from a series of pre-processing data processes was modeled using the Naïve Bayes Classifier Algorithm. The formula of the baye theorem is:

Confusion Matrix

The Naïve Bayes Algorithm utilized the Confusion Matrix which can be used as an Accuracy, Precision, Recall and F1-Score assessment parameter  [20]. The confusion matrix results in values shown in the Table 2.

Table 2: TF-IDF Weighting Example 

Accuracy is a reference to assess the accuracy of the model in classifying correctly [21]. In other words, accuracy becomes the ratio in predicting the truth (positive and negative). Accuracy can be calculated using formula in the following:

Precision is a test method used by comparing the amount of relevant information obtained by the system with the amount of all information taken by the system, both relevant and irrelevant [21]. Precision can be calculated using the formula:

Recall is a test method employed by comparing the amount of relevant information obtained by the system with the amount of all relevant information in the information collection [21]. Recall can be calculated using the formula:

F1-score is a single parameter of a retrieval success measure that combines recall and precision [21]. The F1-score can be calculated using the formula:

Sentiment Classification Visualization

Visualization aims to display the majority topics or the most frequently occurring topic in a sentiment class [22]. Therefore, that information can be taken because it is considered important and able to find out indicators that affect the performance of mobile banking. This study used a visualization of the results of sentiment classification analysis using Word Cloud.

The result of the Naïve Bayes Algorithm is a Confusion Matrix. The results of the confusion matrix in mobile banking can be seen in the Table 4.

Table 3: Confusion Matrix

Source: Author, 2022

Table 4: Confusion Matrix Results

Source: Confusion Matrix Results,2022

Based on the Confusion Matrix results in the table above, the highest True Positive (TP) value in Livin' by Mandiri was 2,405 which meant that from 4,385 testing data in machine learning, 2,405 positive data were correct and in fact the data is indeed positive. The highest False Positive (FP) value in Livin' by Mandiri was 181 which meant that from 4,385 testing data in machine learning, 181 data were positive but in reality, the data was negative. The highest False Negative (FN) value in Livin' by Mandiri was 427 which meant that from 4,385 testing data on machine learning, 427 negative data were predicted but in reality, the data was positive. Furthermore, the highest True Negative (TN) value in BRImo is 1693 which meant that from 4050 testing data in machine learning, 1693 negative data are correct and in fact the data is indeed negative. 

The results of this Confusion Matrix are needed to describe the results of modeling that has been carried out using performance parameters in the form of Accuracy, Precision, Recall and F1-Score. These performance parameters can be seen in the Table 5.

Table 5: Naïve Bayes Modeling Results

Source: Naïve Bayes Results, 2022

Based on the Table 5, it shows that the highest accuracy level is 0.90 or 90% in BRImo. Moreover, the lowest accuracy level is on BCA Mobile at 0.85 or 85%. The amount of accuracy value obtained from all data tested on machine learning, the Naïve Bayes Algorithm succeeded in modeling the dataset correctly or the model's ability to correctly classify review data is 85% on BCA Mobile, 90% on BRImo, 86% on Livin' by Mandiri and 97% on BNI Mobile.

Precision (Positive Predictive Value) is used to describe the degree of accuracy between the requested positive true prediction data and the predicted results provided by the model. The precision value of the highest positive class data is 0.95 or 95% on BCA Mobile and BNI Mobile. This means that the model's ability to correctly classify review data when the model predicts positive data is 95% on BCA Mobile, 94% on BRImo, 88% on Livin' by Mandiri, at 95% on BNI Mobile. While the precision value of the most negative class data is 0.87 or 87% in BRImo. This means that the model's ability to correctly classify review data when the model predicts negative data is 76% on BCA mobile, 87% on BRImo, on Livin' by Mandiri at 85% and on BNI Mobile is 82%. The point is that Naïve Bayes has a better ability to find the accuracy of information from data that is positively labeled than data that is labeled negative. 

Furthermore, Recall or Sensitivity (True Positive Rate) is used to describe the success of the model in rediscovering information. The highest Recall value in the positive class is in BRImo with 0.85 or 85%. This means that the model's ability to correctly classify review data for positive class data is 81% on BCA mobile, 85% on BRImo, 76% on Livin' by Mandiri and 79% on BNI Mobile. The highest Recall value in the negative class is on BNI Mobile with 0.96 or 96%. This means that the model's ability to correctly classify review data for negative class data is 93% on BCA mobile, 95% on BRImo, 93% on Livin' by Mandiri and 96% on BNI Mobile.

The highest F1 Score in the positive class data is 0.90 or 90% on BRImo. While the highest F1 Score in the negative class data is 0.91 or 91% on BRImo. This means that Naïve Bayes has a better ability to classify negative class data reviews than positive class data on the BRImo application. The higher the F1-Score, the better the model.

DISCUSSION

Mobile Banking Performance Sentiment Analysis Comparison

Comparison of sentiment analysis results on mobile banking applications in Indonesia can be seen in the Figure 2.

Figure 2: Comparison of Mobile Banking App Sentiment Analysis

Based on the results of data processing of mobile banking applications using the Naive Bayes method, it can be seen the level of accuracy of each application that produces positive and negative sentiment values. The highest positive sentiment was generated by the Livin' by Mandiri application at 54.8%. When viewed based on the rating, the Livin' by Mandiri application has the lowest rating among other applications that is 4.1/5.0. Meanwhile, the highest negative sentiment was generated by the BCA Mobile application at 49%. In fact, in terms of application rating, BCA Mobile has the highest application rating with 4.7/5.0. In general, mobile banking applications in Indonesia have a higher positive sentiment than negative sentiment. The positive sentiment of mobile banking applications in Indonesia is 47%, while the negative sentiment of mobile banking applications in Indonesia is 40%. 

In the BCA Mobile application, which has the highest rating of 4.7 with an accuracy value of 85%, resulting in a correct positive sentiment of 1202 and a negative sentiment of 1605. In the BRImo application, which has the highest rating, which is 4.3 with an accuracy value of 90%, resulting in a correct positive sentiment of 1964 and a negative sentiment of 1693. In the Livin' by Mandiri application, which has the highest rating 4.1 with an accuracy value of 86%, resulting in a correct positive sentiment of 2405 and a negative sentiment of 1372. In the BNI Mobile application, which has the highest rating, which is 4.6 with an accuracy value of 87%, resulting in a correct positive sentiment of 1300 and a negative sentiment of 1061. Therefore, it was concluded that Livin' by Mandiri had the highest positive sentiment of 54.8%. Meanwhile, the highest negative sentiment was on BCA Mobile at 49%.

In fact, mobile banking applications in Indonesia have a good performance in terms of more positive ratings by users of the application compared to negative assessments of mobile banking applications in Indonesia.

Sentiment Analysis of Mobile Banking Application Performance

In assessing the results of sentiment analysis on the mobile banking application, it can be seen from the accuracy value and adjustments made to the rating owned. Mobile banking app rating and mobile banking app sentiment analysis results. Based on the results of the study, it can be seen that the rating, accuracy value, positive and negative sentiment generated by each mobile banking application are different. So that the results of comprehensive testing on the Mobile Banking application have an accuracy rate of 87%, with a true positive sentiment of 6871 and a negative sentiment of 5731. This shows that mobile banking users in Indonesia have a good (positive) assessment of the performance of mobile banking applications in Indonesia. The increasing number of positive sentiments possessed by mobile banking applications illustrates the trust and user satisfaction with the performance provided by the mobile banking application. A study conducted by Permana et al., [23], also found the same thing that application users have a positive rating on mobile banking applications with a positive true value of 93.474%. Moreover, another study conducted on Kakao Mobile Bank in South Korea by Leem and Eum [10],  also had the same assessment that sentiment analysis of 3,359 Cocoa Bank reviews showed 78.2% (2,628 reviews) of users who had positive sentiment. The perception of ease of use is one of the important things in the use of mobile banking [5]. Their study revealed that user experience can be used for long-term evaluation of mobile banking. Mobile banking is an innovative route to provide banking and cost-effective services for users and banks. Identifying the determinants of the success of m-banking applications is necessary to ensure the sustainable growth of these platforms in the competitive banking industry [13]. 

A different view was expressed by Wijanarto and Brilianti [12], that there is a tendency for negative judgment in mobile banking applications. These findings are similar to those put forward by [11]. They revealed that mobile banking applications have more negative sentiments than positive sentiments. Negative sentiment in mobile banking applications can be used as material for evaluating application development by banking developers to improve and improve mobile banking performance in the long term.

Sentiment Classification Visualization

The visualization used was Word Cloud which described each sentiment, the more often a word is used when giving a review, the larger the word size displayed on the Word Cloud visualization. The highest positive sentiment was generated by the Livin' by Mandiri application at 54.8%. Livin' by Mandiri's positive sentiment Word Cloud visualization can be seen in the Figure 3.

Figure 3: Livin' by Mandiri Positive Sentiment Visualization Results

Based on the picture above, it can be seen that in the positive sentiment of Livin' by Mandiri have several prominent words such as "mandiri", "transaction", "livin", "easy", "application", "bantu" and several other words that indicated the users are discussing their satisfaction with the performance of the mobile banking application when transacting. With regards to the application, with the existence of Livin mandiri facilitates transactions, a very helpful application. The positive sentiment’s result shows a good assessment of the performance of mobile banking applications. Showing the advantages of the application according to reviewers so that it needs to be maintained or even developed to continuously improve performance and maintain user satisfaction.

Meanwhile, the highest negative sentiment is on BCA mobile at 49%. The sentiment visualization word Cloud can be seen in the Figure 4.

Figure 4: BCA Mobile Negative Sentiment Visualization Results

Based on the Figure 4, it can be seen that in BCA Mobile's negative sentiment, there are several prominent words such as "ga", "nya", "bca", "update", "balance", "please" and several other words that indicate that users are not satisfied with the performance of the mobile banking application when transacting. This is related to applications that cannot be opened after being updated, cannot be used to transact and frequently loss of balance. The negative sentiment’s result shows an unfavorable assessment of the performance of mobile banking applications. It shows the weakness of the application that is the problem experienced by the user. This problem will have an impact on reducing the assessment of the performance of mobile banking applications which will affect the decrease in application usage and transaction volume which affects the finances of banking companies.

This study analyzed and compared the sentiment of mobile banking application users in Indonesia based on reviews provided on the Google Play Store. The results of this study showed that BRImo had the highest accuracy value of 90% and BCA Mobile had the lowest accuracy value of 85%. The highest positive sentiment was generated by the Livin' by Mandiri application at 54.8%. Meanwhile, the highest negative sentiment was generated by the BCA Mobile application at 49%. This result described that mobile banking applications in Indonesia have good performance, judging from the higher positive sentiment they have compared to negative sentiment.

Limitation and Suggestions

This study has limitation in which only focused on four major mobile banking banks in Indonesia. Therefore, future research can be carried out on other banking companies including small banks in Indonesia which have mobile banking applications. Also, investigating the comparison of mobile banking applications across countries.

This study also suggests banking companies to make the reviews provided by users as material for evaluation and improvement of the mobile banking application system to improve their mobile banking performance. Furthermore, for the public, this finding can be used as a reference regarding the assessment given by old users through reviews on the mobile banking application which can increase the trust in the mobile banking used.

1. Wanasinghe, T.R. et al. “Industry 4.0 based process data analytics platform.” Methods to Assess and Manage Process Safety in Digitalized Process System, 1st ed., vol. 6, Elsevier Inc., 2022. https://doi.org/10.1016/bs.mcps.2022.04.008.

2. Shao, C. et al. “IoT Data Visualization for business intelligence in corporate finance.” Information Processing and Management, vol. 59, no. 1, 2022, p. 102736. https://doi.org/10.1016/j.ipm.2021.102736.

3. Pang, D. et al. “The asymmetric effect of green investment, natural resources and growth on financial inclusion in China.” Resources Policy, vol. 78, July 2022, p. 102885. https://doi.org/10.1016/j.resourpol.2022.102885.

4. Le, M.T.H. “Examining factors that boost intention and loyalty to use fintech Post-COVID-19 lockdown as a New normal behavior.” Heliyon, vol. 7, no. 8, 2021, e07821. https://doi.org/10.1016/j.heliyon.2021.e07821.

5. Kumar, A. et al. “A framework of mobile banking adoption in India.” Journal of Open Innovation: Technology, Market and Complexity, vol. 6, no. 2, 2020. https://doi.org/10.3390/JOITMC6020040.

6. Bertschek, I. and R. Kesler. “Let the user speak: is feedback on facebook a source of firms’ innovation?” Information Economics and Policy, vol. 60, February 2022, p. 100991. https://doi.org/10.1016/j.infoecopol.2022.100991.

7. Dhar, S. and I. Bose. “Walking on air or hopping mad? Understanding the impact of emotions, sentiments and reactions on ratings in online customer reviews of mobile apps.” Decision Support Systems, vol. 162, February 2022, p. 113769. https://doi.org/10.1016/j.dss.2022.113769.

8. Ruslim, K.I., P.P. Adikara and Indriati. “Analisis sentimen pada ulasan aplikasi mobile banking menggunakan metode support vector machine dan lexicon based features.” Jurnal Pengembangan Teknologi Informasi dan Ilmu Komputer, vol. 3, no. 7, 2019, pp. 6694–6702.

9. Fedorova, E. et al. “Impact of MDandA sentiment on corporate investment in developing economies: Chinese evidence.” Asian Review of Accounting, vol. 30, no. 4, 2022, pp. 513–539. https://doi.org/10.1108/ARA-08-2021-0151.

10. Leem, B.H. and S.W. Eum. “Using text mining to measure mobile banking service quality.” Industrial Management and Data Systems, vol. 121, no. 5, 2021, pp. 993–1007. https://doi.org/10.1108/IMDS-09-2020-0545.

11. Nirwandani, E.P., Indriati and R.C. Wihandika. “Analisis Sentimen Pada Ulasan Pengguna Aplikasi Mandiri Online Menggunakan Metode Modified Term Frequency Scheme dan Naïve Bayes.” Jurnal Pengembangan Teknologi Informasi dan Ilmu Komputer, vol. 5, no. 3, 2021, pp. 1039–1047.

12. Wijanarto, W. and S.P. Brilianti. “Peningkatan Performa Analisis Sentimen Dengan Resampling dan Hyperparameter pada Ulasan Aplikasi BNI Mobile.” Jurnal Eksplora Informatika, vol. 9, no. 2, 2020, pp. 140–153. https://doi.org/10.30864/eksplora.v9i2.333.

13. Shankar, A. et al. “Sustainable mobile banking application: A text mining approach to explore critical success factors.” Journal of Enterprise Information Management, vol. 35, no. 2, 2022, pp. 414–428. https://doi.org/10.1108/JEIM-10-2020-0426.

14. Fayyad, U. et al. “Knowledge discovery and data mining: towards a unifying framework.” American Scientist, vol. 87, no. 1, 1996, pp. 54–61. https://doi.org/10.1511/1999.16.807.

15. D’Andrea, E. et al. “Monitoring the public opinion about the vaccination topic from tweets analysis.” Expert Systems with Applications, vol. 116, 2019, pp. 209–226. https://doi.org/10.1016/j.eswa.2018.09.009

16. Kumar, S. and T.D. Singh. “Fake news detection on hindi news dataset.” Global Transitions Proceedings, vol. 3, no. 1, 2022, pp. 289–297. https://doi.org/10.1016/j.gltp.2022.03.014.

17. Rani, R. and D.K. Lobiyal. “Performance evaluation of text-mining models with hindi stopwords lists.” Journal of King Saud University - Computer and Information Sciences, vol. 34, no. 6, 2022, pp. 2771–2786. https://doi.org/10.1016/j.jksuci.2020.03.003.

18. Trstenjak, B. et al. “KNN with TF-IDF based framework for text categorization.” Procedia Engineering, vol. 69, 2014, pp. 1356–1364. https://doi.org/10.1016/j.proeng.2014.03.129.

19. Xu, F., Z. Pan and R. Xia. “E-Commerce product review sentiment classification based on a naïve bayes continuous learning framework.” Information Processing and Management, vol. 57, no. 5, 2020, p. 102221. https://doi.org/10.1016/j.ipm.2020.102221.

20. Saputra, I. et al. “Text mining of pedulilindungi application reviews on google play store.” Faktor Exacta, vol. 15, no. 2, 2022, pp. 101–108.

21. Qorib, M. et al. “Covid-19 vaccine hesitancy: Text mining, sentiment analysis and machine learning on COVID-19 Vaccination Twitter Dataset.” Expert Systems with Applications, vol. 212, August 2022, p. 118715. https://doi.org/10.1016/j.eswa.2022.118715.

22. Koukaras, P. et al. “Mining association rules from COVID-19 related twitter data to discover word patterns, topics and inferences.” Information Systems, vol. 109, 2022, p. 102054. https://doi.org/10.1016/j.is.2022.102054.

23. Eksa Permana, M. et al. “Sentiment analysis and topic detection of mobile banking application review.” 2020 5th International Conference on Informatics and Computing (ICIC), 2020. https://doi.org/10.1109/ICIC50835.2020.9288616.