Airline new customer tier level forecasting for real-time resource allocation of a miles program
© Berengueres and Efimov; licensee Springer. 2014
Received: 30 October 2013
Accepted: 20 February 2014
Published: 24 June 2014
This is a case study on an airline’s miles program resource optimization. The airline had a large miles loyalty program but was not taking advantage of recent data mining techniques. As an example, to predict whether in the coming month(s), a new passenger would become a privileged frequent flyer or not, a linear extrapolation of the miles earned during the past months was used. This information was then used in CRM interactions between the airline and the passenger. The correlation of extrapolation with whether a new user would attain a privileged miles status was 39% when one month of data was used to make a prediction. In contrast, when GBM and other blending techniques were used, a correlation of 70% was achieved. This corresponded to a prediction accuracy of 87% with less than 3% false positives. The accuracy reached 97% if three months of data instead of one were used. An application that ranks users according to their probability to become part of privileged miles-tier was proposed. The application performs real time allocation of limited resources such as available upgrades on a given flight. Moreover, the airline can assign now those resources to the passengers with the highest revenue potential thus increasing the perceived value of the program at no extra cost.
Previously, several works have addressed the issue of optimizing operations of the airline industry. In 2003, a model that predicts no-show ratio with the purpose of optimizing overbooking practice claimed to increase revenues by 0.4% to 3.2% . More recently, in 2013, Alaska Airlines in cooperation with G.E and Kaggle Inc., launched a $250,000 prize competition with the aim of optimizing costs by modifying flight plans . However, there are very little works on miles programs or frequent flier programs that focus on enhancing their value. One example is  from Taiwan, but they focused on segmenting customers by extracting decision-rules from questionnaires. Using a Rough Set Approach they report classification accuracies of 95%.  focused on segmenting users according to return flight and length of stays using a k-means algorithm. A few other works such as , focused on predicting the future value of passengers (Customer Equity). They attempt to predict the top 20% most valuable customers. Using SAS software they report false positive and false negative rate of 13% and 55% respectively. None of them have applied this knowledge to increase the value-proposition of miles programs. All this in spite the fact that some of these programs have become more valuable than the airline that started them. Such is the case of Aeroplan, a spin off of the miles programs started by Toronto based Air Canada. Air Canada is as of today valued at $500 M whereas Aeroplan is valued at $2bn, four times more . Given this it is surprising that so many works focus in ticket or “revenue management”  but few focus on their miles programs. In this case study we report the experience gained when we analyzed the data of an airline’s miles program. We show how by applying modern data mining techniques it is possible to create value for both: (1) the owner of the program and (2) future high-value passengers.
Age and demographics.
Loyalty program purchases etc.
Flight activity, miles earned etc.
After co-examining the opportunities that the data offered with the airline we decided to focus on high-value passengers with the objective to: predict and discriminate which of the new passengers who enroll the miles program will be high-value customers before it is obvious to a human expert.
So that potential high-value customers are identified in order to gain their loyalty sooner.
So that limited resources such as upgrades are optimally allocated to passengers with potential to become high-value.
To enhance customer profiling.
Current method: linear extrapolation
Definition of the target variable
Identifying future high value customers
Miles program dataset
Id, Date_of_birth, Nationality, City, State, Country, Interest_1, Interest_2, Interest_3, Interest_4, Interest_5, Tier
Id, Company, Activity_date, Origin, Destination, Class_code, Flt_number, Definition, Miles, Points,
Activity (not used)
Id, Definition, Issue_date, Miles, Redeposited_points, Flight_date, Class_code, Origin, Destination, Flight_number, ret_flight_date, Ret _flight_number, Ret_class_code, Ret_origin, Ret_destination, Product_code, Cash_before_premium
Cleaning of the data.
Passengers who fly less than one return trip in six years.
Passengers with very high activity (number of flights greater than 500).
Additionally, before the data was handed over to us, the airline anonymized the id field. The id had been replaced by an alphanumeric hash code of length 32 characters long. We changed these cumbersome hash ids into integer numerical values to increase the performance. Big datasets made it very difficult for R Language to operate with hash ids during feature engineering.
The passenger data in the Tables.
Length of D.
Each component of the vector is called a “feature”. The length of the vector is 634 features. The vector is equivalent to a digital fingerprint for each passenger considered in a given period. Some features are straightforward to calculate and others require complex calculations. Following we explain how each feature was calculated. We have divided the features in three groups: metric, categorical and cluster features.
Group 1 - Metric features
A metric feature is data that is already in numerical format. For example, age of a customer.
Group 2 - Categorical features
A categorical feature is a text variable, the content of which can only belong to a finite group of choices. The column “City” is such an example. The main problem with categorical variables is that they must be converted to numbers somehow. Since a computer does not understand “city names” per se, there are different ways to operate with such variables. One way is to encode code each name (aka level or category) of each categorical variable into a binary feature. Therefore, we opted for an interpretation of categorical variables using the dummy variable method as follows:
Let N be the total number of values for a given categorical variable, then we create N new “dummy” features. If a given record has value = i th level, then the i th dummy feature equals 1, otherwise 0.
Unfortunately, this transformation restricts range of algorithms that are effective: algorithms based on metrical approach such as SVM  yield poor results in such cases.
City of Passenger’s address.
State of Passenger’s address.
Country of Passenger’s address.
Passenger’s Company (Employer).
Flight Origin (Airport).
Flight Destination (Airport).
Ticket Class Code (Economy E, F, K…).
Example of conversion from categorical to binary for a given passenger
Group 3. Cluster features
Sum of miles
0 or 1
0 or 1
0 or 1
0 or 1
0 or 1
Example of creation of 60 cluster features by the dummy method
Example of K-means output (categorical)
Label of the new binary features
Example of feature vectors generated per each passenger
(43, 13.54, …, 0, 0, 12, 0, …, 0, 1, 1, 0, 0, 0, …, 0, 0, 0)
(26, 2.9, …, 0, 0, 12, 0, …, 0, 1, 1, 0, 0, 0, …, 0, 0, 0)
Example of feature vectors and target variables
(43, 13.54, …, 0, 0, 12, 0, …, 0, 1, 1, 0, 0, 0, …, 0, 0, 0)
0 (he did not achieve silver status during S period)
(26, 2.9, …, 0, 0, 12, 0, …, 0, 1, 1, 0, 0, 0, …, 0, 0, 0)
1 (yes he did)
Now, we can consider all this vectors as a matrix, where rows are passengers, columns are features, and the last column is the target variable. We will use such a matrix to train a mathematical model with the purpose of predicting the target variable in new passengers. Once the model is trained, to predict if a passenger will attain silver status in a given time frame S (in the future) we only need to generate its feature vector by observing the passenger for a period of time D since their first flight. Once the vector is generated (naturally, without target variable) we can input it into the model and the model will output a number. There are no restrictions on when to ask a model for a prediction as long as the data for the given S period is available.
The high nonlinearity of the features (meaning low correlation between target variable and features) restricts the number of algorithms we can use to predict with high accuracy. We chose to blend two algorithms, which were in our opinion the most appropriate for this dataset: the GBM (Generalized Boosting Machine package) and GLM (Generalized Linear Model, glm in R). Both models are trained with the same target variable: silver_attain, and try minimize the binomial deviance (Log Loss) of prediction error. While, we chose GLM and GBM because they produce different models, GLM can be considered as non-parametric version of GBM.
distribution = “bernoulli”,
n.trees = 2000,
shrinkage = 0.01,
interaction.depth = 0.5,
bag.fraction = 0.5,
train.fraction = 0.1,
n.minobsinnode = 10.
A good description of the GBM implementation we used can be found in . It works as follows: the training matrix is used to train the model with the aforementioned parameters. After, some minutes we have a trained model. Now to make a new prediction on a passenger vector or a number of passenger vectors, we input it in the model and the model will return a number from 0 to 1 for each passenger vector. 0 means that the model predicts 0% chance for that passenger to attain silver tier status within the S period of time. A 1 means that the model is the most confident the silver tier will be attained. Training the model takes about 1 hour, but asking the model to predict what will happen to 10,000 passengers takes just seconds.
The other algorithm used is GLM. GLM stands for General Linear Model. As before we used the implementation in R Language provided by . GLM is just a simple logistic regression where we optimize binomial deviance, where Error = silver_attain – prediction. Default parameters where used except for family which was set to “binomial”.
Blending with grid search
Combining predictions is known to improve accuracy if certain conditions are met . In general, the less correlation between the individual predictors, the higher the gain in accuracy. A way that usually lowers cross-correlation is to combine models of different “nature”. We chose to combine a decision tree based algorithm (GBM) and regression based one (GLM). For a 3/3 model the relative gain in accuracy due to blending was 3%. The final prediction was constructed as a linear combination of the output of the two. Grid search was used to find which linear combination was optimal on the training set. The optimal combination was 90% of the GBM prediction plus 10% of the GLM prediction.
where P is precision, A is accuracy, N0 is a number of true positives or users predicted to become silver and that later indeed became silver, N1 is the number of false positives or users predicted to become silver but who did not indeed become silver after S days and N 2 is the number of false negatives or number of users predicted to not become silver and who indeed became silver. In the next section we will show the calculated error for the normal Precision and Recall and the calculated error according to chosen loss functions (Eq. 1).
Discussion and evaluation
It takes about 18 hours on a laptop to clean the data and construct the features. To build a D/S model takes about 1 hour per each D/S combination. Once a model is pre-calculated, making a prediction for one single passenger takes less than 350 ms (similar to a Google search).
Comparison of prediction power of extrapolation vs. D/S model
Question asked D months after 1st flight: will they become Silver within S months?
Extrapolation of miles Rx1
D/S Model Rx2P3A4
0.60 81% 31%
0.71 87% 53%
0.70 89% 48%
0.89 97% 82%5
0.67 95% 46%
0.83 96% 69%
Precision and accuracy of silver attainment
Will a user become Silver in 1 month?
Predicted (With 3 months of data)
What really happened
Is the prediction correct?
Numbers by case
Percent of total
Confirmation of no data leakage
However, due to the spectacular high accuracy rates obtained, the airline showed a healthy concern that the prediction might be wrong due to data leakage. Data leakage happens when some data from the training set somehow contains information about what wants to be predicted (target_variable). The only way to proof 100% that there is no data leakage is to do predictions in the future (about data that does not exist at the time of the prediction). To address this valid concern the model was used to predict what the passengers would do in the future.
Which of the 49572 customers that had enrolled during the last quarter of 2012 would attain silver in the first 30 days of 2013, this is a D = 3/S = 1 model and
Which of the 7890 customers that had enrolled during the last two weeks of December 2012 would attain silver during the first 90 days of 2013 a D = 0.5/S = 3 model.
By translating passenger data from “Airline” timeline to a timeline relative to each passenger first flight, we have shown that a D/S model yields high accuracies. Furthermore, taking advantage of recently made available data mining libraries [9, 10] we outperformed simple extrapolation models and previous works . False positive rates are less than 3%. The causes of a false positive have not been investigated in the scope of this project, but can be due to either/or a combination of: (1) the predictive nature of the data is not unlimited. (2) The predictive power of the model can be improved. However, the most interesting result is that the perceived value of a miles program can be increased dramatically to the very customers that matter most to the airline: the ones with high likelihood of becoming Silver.
In our experience with previous data mining projects, rather than fine tune models, the most effective way to improve accuracy is to add new features which are as uncorrelated as possible with existing features. A good place to look for potential candidates are features derived from different data sources other than the Airline CRM database, for example publicly available social media data.
Sajat Kamal for the Aeroplan insights. Dr. Barry Green and Roy Kinnear showing themselves to be greater than their prejudices and letting us modelling the data.
- Lawrence RD, Hong SJ, Cherrier J Proceedings of the ninth SIGKDD international conference on Knowledge discovery and data mining. Passenger-based predictive modeling of airline no-show rates 2003, 397–406.Google Scholar
- Alaska Airlines GE Flight Quest data mining challenge. 2013. http://www.gequest.com/c/flight
- Liou JJH, Tzeng G-H: A dominance-based rough set approach to customer behavior in the airline market. Inf Sci 2010,180(11):2230–2238. 10.1016/j.ins.2010.01.025View ArticleGoogle Scholar
- Pritscher L, Feyen H Data Mining for Marketing Applications. In Data mining and strategic marketing in the airline industry. Citeseer; 2001. http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.124.7062 Google Scholar
- Malthouse EC, Blattberg RC Can we predict customer lifetime value? In Journal of Interactive Marketing. Wiley Online Library; 2005. http://www.researchgate.net/publication/227633642_Can_we_predict_customer_lifetime_value/file/60b7d517fd5bb2905e.pdf Google Scholar
- Aeroplan loyalty program. http://en.wikipedia.org/wiki/Aeroplan
- Smith BC, Leimkuhler JF, Darrow RM: Yield management at American airlines. Interfaces 1992,22(1):8–31. 10.1287/inte.22.1.8View ArticleGoogle Scholar
- Burges CJ: A tutorial on support vector machines for pattern recognition. Data Min Knowl Disc 1998,2(2):121–167. 10.1023/A:1009715923555View ArticleGoogle Scholar
- gbm: Generalized Boosted Regression Models. R package version 2.0–8 Greg Ridgeway with contributions from others 2013. http://CRAN.R-project.org/package=gbm
- Ian M: glm2: Fitting Generalized Linear Models. R package version 1.1.1. 2012. http://CRAN.R-project.org/package=glm2 Google Scholar
- Valentini G, Masulli F Neural Nets. In Ensembles of learning machines. Berlin Heidelberg: Springer; 2002:3–20.Google Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.