The still undercurrent

It was a scene from the famous movie ‘Saving Private Ryan’ where Matt Daemon was narrating to Tom Hanks (and they are my all-time favorite actors alongside Al Pacino) that he and two other friends were trying to save the life of the girlfriend of one of those friends as she was being attacked by a very violent mob. The gifted girl had climbed up a tall tree that was also fairly fat when compared to the swank, groggy lasso. Her boyfriend Adam was ruminating because he had left her alone in the middle of the mobile mob that was surrounded solely by soon to be bygone bushes behind the backyard of their huge house. Now afraid to the highest intensity of upon having climbed to the tree and reaching the apex of that tree, she in a still motion swooped down to the ground (not the Gradient Descent method way!) but found her clothes and her inner wear getting shredded to pieces before flying in all directions resembling a gyrating motion, including in the direction from where Matt was watching in a state of pensive yet constant shock. To add to the hysteria, she had touched down all the brown branches of that giant tree (she had not having any time to apply any pruning mechanism or optimized her way to descend) before landing on the ground with a loud thud. There she lay like a still leaf in a rain socked pool of water having little undercurrent.

Sadly, Matt reckoned that it was exactly two years ago that that scene had transpired and when he and those friends were last to be seen together.

Having heard this, Tom replied with ‘Oh, my’ and Matt quickly convoluted to a state of still and then let his laughter aloud. He was laughing from his gut. But before he could gather his wits, his head felt a huge undercurrent and his mind ricocheted in many directions that were not devoid of desolation and despair (we can call these weighted vectors and tensors when speaking in an academic institute of repute).  It was the moment of melancholy which also resembled a lull before the storm, or as I call it the still before the undercurrent, because they were laying stranded in the middle of a fierce battleground with the sound of guns and artillery blaring their ears for fun.

This scene was shown right after the interval of the movie.

Perhaps we can have similar moments in many other movies which decide their fate. For instance, in the movie ‘Sanju, the movie was veering towards a dull biopic of Sanjay Dutt until after the interval, a Gujarati origin man named Kanhaiya from New York city, never married before persona, entered the movie and gave some sane advice to Sanjay Dutt, and stole the limelight from Sanjay due to his demeanor, easy on eyes acting that was often juxtaposed with traces of wit and humor.

We should get videos of such scenes that can potentially change the course of the entire movie, resembling the undercurrents that we can often see in the business world! Can we cull out such patterns spanning say ten minutes from an otherwise two hundred minute long movie, and exclude those movies whose interest and plot tails off in the first half itself?

There can be other patterns in the movies as well, such as the beginner's luck as was evident in 'Dev D', 'Vicky Donor', etc. But we can analyze that pattern in our future analysis.

Definitions of undercurrent:

1.       A flow of water that moves below the surface of the ocean or a river.

2.       A hidden feeling or tendency that is usually different from the one that is easy to see or understand.

3.       There is a strong undercurrent for Mergers and Acquisitions in India right now. Many conversations are happening, even if the deal news has not been announced.

The Popping Pink

The popping pink

I was seated on my couch of the sofa comfortably watching a sports show on my television. There was a fond and vivid memory of the three chocolates that I had munched and then gobbled one after the other in quick succession. There were precisely three chocolate wrappers that were making funny sounds in the left pocket of my pyajama.

Suddenly, the shun shown stupendously and so much was its vigor that it took my eyes of their comfort zone and all the way up to the center of the garden of our house. So much perfect and precise the rays were that I was forced to fixate my eyes on to a very large pink flower. I was so surprised to look at the radius of that flower that it evoked both a moment of happiness and emotion of tears on my face.

I instantly took my phone and switched on its camera to take a snap of the popping pink flower. When I had met my desire to take that snap, I then decided to go to the front of the house and take the snap from there. It was as if the entire flower was pellucid, a feature which was forcing me to take an all-round view of it by forcing me to take a peripatetic path.

When I told my wife about that flower, then she asked me the question “How old is it?”, then swiftly came the reply from me “How is it old?”

The queer syndrome did not stop there. Later I noticed the pocked of my pyjamas and found to my surprise that the chocolate wrappers were brick red and white in color, the colors which when mixed yield a bright pink hue just like the one that was painted on that popping pink flower.

Variable Selection

First, apply simple steps to remove junk variables. For instance, throw those variables that have got too high a proportion of missing values, too low coefficient of variation, etc.

Then apply Weight of Evidence and Information Value (IV).

Then throw those variables with IV either too low or too high.

Then choose those values that have a high WoE.

You can do this for categorical variables. For continuous variables, first apply binning and then apply WoE and IV to those.

Then apply VIF (Variance Inflation Factor) to remove those variables having high muliticollinearity.

Addressing Multicollinearity

First, X and X^2 or for that mater X and XZ can not be considered as co-linear in a mathematical sense. Correlation coefficient as we commonly used to
measure collinearity between two variables bound give erroneous results. So, collinearity measured by this measure is not useful in such a case.

One should use correlation coefficient if you are relatively sure about linear relationship between two variables.

In case of Logistic Regression, All predictors should follow normal distribution and normalization is done to make sure that predictors are unit less (and hence additive in true sense). So if X and Z follows normal distribution then neither X^2, Z^2 nor XZ follows normal distribution. Now one can argue that after applying CLT XZ, X^2 or Z^2 will follow normal distribution, but under limit (n -> inf)
Hence X^2 or Z^2 or XZ in first place does not qualify to be predictors in case of logistic regression.

The idea of any predictive model is to include all those predictor which has predictive value (or has information about the target variable), if X is a predictor and brings some information what more information X^2 can bring is the question one should consider. In fact by nature of function X^2 it conceals or confounds some of the information which X brings. similar is the case for XZ. (I need not explain that there are better ways to handle interaction effect of factors X and Z)

From above three points it is clear that if you decide to include X and/OR Z as your predictors, you should not include X^2, Z^2 and XZ as a predictor into same predictive model.

About p-values, it is clear that CLT is applied before calculating p-values. These are asymptotic p-values. Once I assume X^2 or XZ follows normal distribution it does not matter whether I standardize X^2 or XZ, p-values are bound to be same anyways there is no magic.

How often do you see customers rebuild/refresh models?

I am interested in how frequently you see customers rebuilding/refreshing models they have deployed in production? Are they using our C&DS automation to schedule these, and do they do any refreshing of their models automatically, or do they have analysts perform these manually?

This is what I see customers wanting:

-Champion challenger

-Self improving models, models learns as new data comes in (ala our naive Bayes, 'self learning' model).  

This is what I see customers doing:

‎-use CADS to store models.  

-manually score

-sometime schedule score

-real time deployment.  

-Never: champion challenger. Reason: a model needs thorough checking when re-created.  Interface in CADS is not on par.  

-Never: refresh: R‎eason: a model needs thorough checking.  Refresh works well, but the storing and replacing the model is too complex because it involves scripting and Modeler/CADS interplay.  

This is what I customers want to do:

Analytical reporting:

-being able to set up an experiment in CADS to keep track of model performance over the lifetime of the model. 

-having a comprehensive (=prebuilt) and configurable model evaluation dashboard.  

Operational reporting:

-being able to see what ‎the model predicts (without knowing the outcomes yet, hence the operational reporting).  

-having a comprehensive and configurable model scoring dashboard.  

In addition: having both abilities working if one has many models in production in a convenient way.  On top of this massive model deployment, having a way to quickly get insight in the model trends and being able to alert the worse performing models.  

CLV (Customer Lifetime Modeling) for Retail (supermarket and grocery chain)

First challenge is identifying customer over visits. There's some entity analytics that can be done: you associate loyalty cards with creditcard/bank account numbers and so you can even identify when the same customer changes cards. Don't expect 100% identification here.   Divide your purchases in loose baskets vs identified customers and provide a separate treatment for both.  ‎For the former you can only provide margin details per basket.   

Revenue= easy, sum of purchases -returns. 

Costs = tricky. Top down approach. Get the details from finance and collaborate with finance on this. If they sell say fresh good and electronics, likely, finance has a p&l for them separately. If you find the costs on department level are 35% of the total revenue, you use that number for every product in that category.   Try to get as deep as possible. Likely you cant do product level.   Important distinction is own brand vs foreign brand, so you can have several factors you can account for. Likely you only have cost data for the main effects (own brand vs foreign brand and category A vs B rather‎ then for each of the 4 combination). Use the raking (iterative fitting) for it to get at the 4 combination level. (raking is used in reweighting surveys to make the research group having similar properties to the population). Available in SPSS.   

Now revenue - cost is margin. On product group level very interesting to ‎visualize. (revenue vs margin%, revenue vs costs, spot the outliers, do some segmentation, color the scatterplots with the segmentation and various other characteristics you have).  

Now roll up to customer level. Use 1+ year of purchases, provide numbers on a year basis.   Again, show that customer margin is not just‎ a matter of taking say 15% of the revenue, but differs for everyone. Segment customers on margin and per resulting segment profile the product groups they buy from.  

Next lifetime.  ‎Properly divide your available time line into parts and use the purchase data from say month 1 to 6 to predict the purchase amount for the next six months. Validate this model by back testing it on the data the year before. Depending on the structure in the data, the model can try to predict 1) are you returning (0/1) 2) what will be your revenue class, or 3) what will be your revenue.    You can try to do the same for margin in order to get a sense of customers that change their buying habits.  

Now you can future tell the customer value. You can properly try to account for net present value etc, but I believe the models will never give you enough resolution to justify going through that additional logic.  

Use the future revenue, costs, margin in relation with the current ones and segment to store level (store type, province, area characteristics etc).  

You can use the results as follows:

1) predict overall revenue and margin for next 6 months (interesting for finance in order to determine strategy, specially on store level)

2) spot customers who are going upward or downward (interesting for campaigning purposes).  

‎3) understand the effects of promotions of article categories in the light of those newly obtained kpi's (interesting for category managers).  

Workforce Attrition Predictive Modeling (SPSS)

Recently, I was working on an Analytics / Data Mining project to predict the chances of an employee's attrition. I used SPSS for doing this, although I came to know that there could be more tools like SAS to do this kind of predictive modeling. Here are a few factors that I came up with to predict a Company's employee attrition (employee initiated). If you can suggest some factors, that would be really good to know!

Tenure in system in years
Any department level features, like average/less promotion year, etc.
Last increment % (was it greater than 1%?)
Last annual bonus % (was it greater than 5%?)
Number of perks, e.g., entitlement for a car, work from home, etc.
Last Promotion date? Was the employee ever promoted?
External factors (e.g., geography)