Predicting payment method from transaction characteristic
Kai Foerster, Krishnamoorty Manohara, Laura Menicacci, Andrew Wells
2022-05-12
Abstract
The introduction of mobile money services is revolutionising the financial lives of low-income households in Kenya by giving them access to formal financial services. While M-Pesa has a potential to change the financial lives of poor Kenyan households, usage of mobile money in everyday financial transactions is less understood.
This study runs explainable machine learning algorithms on the Kenya Financial Diaries, a high-frequency financial transaction dataset. The best algorithm, which is the Random Forest classifier, is able to predict the transaction method for a given transaction with a 98.52% accuracy and an MCC of 93.55%. The main predictors of mobile money transactions were transactions related to: remittances, telecom providers, monthly consumption and income measures as well as mental and physical health characteristics. These findings contribute to a greater understanding of the financial lives of low-income Kenyans.
Background
As of 2015, 35% of the over 50 million people living in Kenya live below the poverty line. Understanding the determinants of poverty in Kenya may help policy makers to create more targeted policies to help the most financially vulnerable population. In particular, the mobile money service M-Pesa has enabled most poor Kenyans to access formal financial services seen its introduction in 2007. While mobile money has been hailed as a beacon that could help many poor Kenyans out of poverty, the mechanism of how M-Pesa helps poor households is less known. Our study uncovers some of these mechanism by investigating which transaction characteristics predict whether it was done/completed by mobile money, cash or another transaction method. Most Kenyans do not have access to (traditional) formal financial services, such as a bank account, and the pre-2007 Kenyan economy was mainly characterized by a transfer economy, operating through cash and informal transfers, such as trade and barter. As of 2013, 74% of the Kenyan population above 15-years-old had a mobile money account. As can demonstrated in the figure below, Kenya has (by far) the highest proportion of adults with a mobile money account. M-Pesa is therefore presented with a certain level of fanfare and promise for its ability to give people traditionally marginalized people access to formal financial services. Research in 2016 predicted that M-Pesa has lifted 2% of Kenyan households out of poverty (Navis 2014).
What is M-Pesa?
Though app-based cash transfer systems such as PayPal and CashApp are becoming more common, M-Pesa (“M” for mobile and “Pesa” for the Kiswahili world for money) pre-dated these systems when it was introduced in Kenya in 2007 by cellphone providers Safaricom and Vodafone. While money transfer systems such as Western Union have operated for many years, fees were debilitatingly high for people making small payments. M-Pesa gives a cheap and convenient way to transfer money via SMS.
Methods and tools
We figured that, by employing some of the usual of machine learning, such as logistic regression, ridge regression, decision trees, support vector machines, and random rorests, we could find a classification algorithm for best predicting modes of transactions, in terms of whether mobile money was used or not. Furthermore, the machine learning algorithms named above are interpretable! That means, we could use them to understand which feature are important to predict the usage of mobile money. Keeping in mind that our classes were strongly imbalanced, we used four evaluation metrics to check on the prediction performance on the differently distributed categories: Accuracy (the classical one), F1 score, the Matthew’s correlation coefficient and the ROC-AUC score. With these four measures, we were able to understand which model was performing better in predicting minority classes as the mobile money one, despite the high numbers given by the accuracy score, that could happen to be misleading. For the baseline, we used a Random Rate Classifier, which is just a fancy name for just guessing the class. Since the classes were unbalanced, we weighted the guessed classes according to their relative frequencies in the training data.
Experiments
The Decision Tree was constructed using sklearn. It had a maximum depth of 50 nodes between the root and leaves. We chose this model over more performance-oriented models since it is not a black box, and the decision-making process that it “learns” can be easily visualized. Since one of our primary goals with this project was to identify which characteristics of the transactions and subjects were more influential in deciding their method of transaction, this was important. By taking a look at the generated tree, we could see clearly which were the best questions to ask to gain the most information on the target.
Decision Tree
This image is just a sample, in reality the generated tree was far deeper and had many more nodes. When training the Decision Tree, a tradeoff had to be made in terms of tree depth; deeper trees had greater space and time complexity, while also being more accurate. However, trees that were too deep would be less accurate, since they would be overfit on the training data. Hence, we calculated metrics for trees of different depths, evaluated on a previously unseen validation set.
Decision Tree performance
A Random Forest was generated with 50 trees. Each tree utilized a different subset of the data, and took their average result. The basic structure of one of the trees in the forest can be seen below. However, simply visualizing the tree is of little benefit without the context of knowing what subset of rows and columns it was trained on. For example, the following tree does not predict cash or mobile money at all, probably because none of the samples it was trained on had those values.
Tree visualization from Random Forest classifier
Our project was able to detect the most important features that predict a mobile money transaction. Apart from administrative features such as “month since the first transaction was recorded” which are not informative, the re-occurring important features across models were related to remittances, telecom providers, monthly consumption and income as well as mental and physical health characteristics.
Analysis
ROC-AUC Curve for the best two models with a selected sample
The figure above represents the final output of this project. In this plot, we recoded the target variable (payment method) using a one-vs-all strategy. We wanted to understand how the model could behave in predicting only one of these classes, mobile money, as such class has always represented the goal of our research question. We plotted the ROCAUC curve for our two best models, namely the Random Forest with 25 trees and the Decision Tree with a depth of 50. We chose the best performing decision tree, and the number of trees in the Random Forest that acted like a cutoff in our GridSearch training. We observed indeed that after 25 trees, the model wasn’t improving significantly as before.
According to the ROC-AUC score, the performance is outperforming for both of the models. These two algorithms indeed were able to gain a high score even with such a strict metric. The ROC-AUC Curve plotted here reflects such performances also on the prediction of a class that is present in only the 2% of the data.
Two things are mostly important to notice when looking at such a graph. Firstly, the AUC (Area Under the Curve): the larger the area, the better the model will be in its true positive prediction rate. We notice that the Random Forest with 25 is slightly better than the Decision Tree, with a smoother curve. This latter one has indeed a strong change of direction at a specific point, meaning that after that threshold the model started to predict only false positives. Another important thing to observe in the ROC-AUC, is the steepness of the curves. We can say that both models reach the top left of the curve in a strikingly similar way.
Conclusions
This project predicted which transaction method Kenyan household use in different circumstance deploying machine learning algorithms on the Kenyan Financial Diaries data. The Random Forest classifier outperformed all other models and achieved a ROC-AUC of 0.937. We were able to find the most important variables that predicted whether someone used M-Pesa or not, these included, remittances, telecom providers, monthly consumptions and income, and health. Future work could extend the analysis to financial diaries of other countries that have experienced a mobile money revolution such as Tanzania or Mozambique.
Acknowledgments
We would like to thank FSD Kenya for providing the data online through its website and Harvard dataverse. We are also grateful to Eric Kolibacz for his advice during labs. We are also grateful for the countless advice we found online on multiple coding problems we ran into.