Uplift Modelling Wins Hillstrom E-Mail Analytics Challenge

We won!

Kevin Hillstrom, President of MineThatData, used his blog to issue his E-mail Analytics and Data Mining Challenge.

This, it transpires, is an uplift problem.

Kevin published a dataset describing 64,000 etail customers, one third of whom (randomly chosen) had received one marketing email (the "Men's Email"), another (random) third of whom had received a second email (the "Women's Email") with the final third receiving neither email. (Note that the email refers to the email, not the sex of the customer.)

The challenge was to analyse the data to work out which campaign did better, and then (the real challenge) to analyse which customers responded most positively and least positively to the emails. The first of these, of course, is a simple post-campaign reporting analysis. But the second involves (formally or informally) modelling the incremental impact of the two emails on each customer and to allow selection of the best and worst targets. That's what uplift models do.

The challenge was interesting in all sorts of ways, and in my write-up I tried to reflect the detail of the process I actually had to go through to build a set of useful models. This included some false starts, some serious worries about sample size, and some run-ins with unhelpful anti-correlations.

If you're interested, you can read all about it in the resulting submission/white paper, which won the challenge and which is now available from Stochastic Solutions at http://stochasticsolutions.com/etailPaper.html.

I was particularly pleased to make progress with this problem because whereas, in most cases, the problems I've tackled from the financial services and telecoms industries have almost always yielded to the uplift modelling approach, there seem to be some characteristics of retail problems that sometimes make the problems harder to crack. I think I learned some useful lessons tackling this problem, and I've tried to document most of those in the paper.

Thanks to Kevin for organizing the challenge.

Bastard

Uplift Modelling for Cross-Selling: White Paper Available

I'm pleased to announce the availability of a new white paper entitled Generating Incremental Sales: Maximizing the incremental impact of cross-selling, up-selling and deep-selling through uplift modelling.

Abstract

There is a subtle but important difference between

targeting people who are likely to buy if they are included in a campaign

and

targeting people who are only likely to buy if they are included in a campaign.

It transpires that this single-word distinction is often the difference between a strongly profitable and a severely loss-making campaign. We have seen many cases in which moving to targeting on the second basis (for incremental sales) has more than doubled the extra sales generated by a campaign. Conventional “response” models—despite their name—target on the former basis, and have a marked tendency to concentrate on people who would have bought anyway, thus misallocating marketing resources by increasing costs and failing to maximize sales. This paper discusses the use of a radical new type of predictive modelling—uplift modelling—that allows campaigns to be targeted on the second basis, i.e. so as to maximize incremental sales from cross-sell, up-sell and other sales-generation campaigns.

It's available as a PDF download here (216K, no registration required).

Financial Services Retention White Paper Available

I've recently completed a counterpart to my white paper on retention in financial services—"Identifying who can be saved and who will be driven away by retention activity". It's a counterpart to the previous telecoms-focused white paper on the same theme. The new one focuses on the incremental impact of retention campaigns in financial services. The abstract is below.

Abstract

It has been repeatedly demonstrated that the very act of trying to ‘save’ some customers provokes them to leave. This is not hard to understand, for a key targeting criterion is usually estimated churn probability, and this is highly correlated with customer dissatisfaction. Often, it is mainly lethargy that is preventing a dissatisfied customer from actually leaving. Interventions designed with the express purpose of reducing customer loss can provide an opportunity for such dissatisfaction to crystallise, provoking or bringing forward customer departures that might otherwise have been avoided, or at least delayed. This is especially true when intrusive contact mechanisms, such as outbound calling, are employed. Retention programmes can be made more effective and more profitable by switching the emphasis from customers with a high probability of leaving to those likely to react positively to retention activity. This paper discusses how targeting on the basis of such ‘savability’ can be achieved, illustrating the effectiveness of the approach with case studies. Insofar as a paper can be summarised in a motto, this paper’s is “savability is the key to retention activity”.

It's available as a PDF download here (228K, no registration required).

Do Computers Dream of Electronic Forgiveness?

Jim Novo on Control Groups

There are a couple of good posts on control groups from Jim Novo at his Marketing Productivity Blog. the first is about The Benefits of Using Control Groups. The second is, well, kind of about the same thing, but more specifically discusses how to justify the use of control groups.

Well worth a read, and he's promising more to come.

Kafka

Telecoms Retention White Paper Available

I've recently completed a white paper "Identifying who can be saved and who will be driven away by retention activity". It focuses on the incremental impact of retention campaigns in telecommunications, with a particular focus on the mobile/cellular sector.

Abstract

It has been repeatedly demonstrated that the very act of trying to ‘save’ some customers provokes them to leave. This is not hard to understand, for a key targeting criterion is usually estimated churn probability, and this is highly correlated with customer dissatisfaction. Often, it is mainly lethargy that is preventing a dissatisfied customer from actually leaving. Interventions designed with the express purpose of reducing customer loss can provide an opportunity for such dissatisfaction to crystallise, provoking or bringing forward customer departures that might otherwise have been avoided, or at least delayed. This is especially true when intrusive contact mechanisms, such as outbound calling, are employed. Retention programmes can be made more effective and more profitable by switching the emphasis from customers with a high probability of leaving to those likely to react positively to retention activity. This paper discusses how targeting on the basis of such ‘savability’ can be achieved, illustrating the effectiveness of the approach with case studies. Insofar as a paper can be summarised in a motto, this paper’s is “savability is the key to retention activity”.

It's available as a PDF download here (232K, no registration required).

Another Crazy Dreamer

Uplift Modelling FAQ

[This is a post that will be updated periodically as more FAQs are added, so if you subscribe to the feed, it may keep re-appearing.]

1. Q. What is Uplift Modelling?
   A. Uplift modelling is a way of predicting the difference that an action makes to the behaviour of someone. Typically, it is used to predict the change in purchase probability, attrition probability, spend level or risk that results from a marketing action such as sending a piece of mail, making a call to someone, or changing some aspect of the service that the customer receives.
2. Q. Uplift Modelling sounds like Response Modelling. How is it different?
   A. Ordinary "response" modelling actually doesn't model a change in behaviour (even though it sounds as if it should): it models the behaviour of someone who is subject to some influence. Uplift models instead model the change in behaviour that results when someone is subject to an influence—typically, how much more that person spends, how much less likely (s)he is to leave etc.
   Mathematically, a response model predicts something like

   P (purchase | treatment)

   ("the probability of purchase given some specific treatment", such as a mailing), whereas an uplift model predicts

   P (purchase | treatment) – P (purchase | no treatment)

   ("the difference between the probability of purchase given some specific treatment and the corresponding probability if the customer is not subject to that treatment").
3. Q. Uplift modelling sounds like Voodoo. How can it possibly know the change in behaviour of a single individual?
   A. Uplift modelling can't know the change in behaviour for any individual, any more than a normal model can know the behaviour of an individual in a future. But it can predict it. It does this by looking at two groups of people, one of which was subject to the marketing action in question, and the other of which was not (a control group). Just as it is standard to measure the incrementality of a campaign by looking at the overall difference in purchase rate between the treated group and an otherwise equivalent control group, uplift modelling models the difference in behaviour between these two groups, finding patterns in the variation.
4. Q. Does Uplift Modelling Really Work?
   A. Uplift modelling can work, and has been proven to do so with in-market tests. Uplift models are harder to build than conventional models, because they predict a second-order effect—usually the difference between two probabilities. This means that the error bars tend to be larger than for conventional models, and sometimes there is simply not enough signal for current techniques to model accurately. This is especially true when, as if often the case, the control group is small.
5. Q. When does uplift modelling predict different things from non-uplift models?
   A. It's perhaps easier to say when they predict the same thing. This is usually when there is essentially no behaviour in the control group. For example, if a set of people purchase product X after a mailing, but no one purchases it without the mailing, and uplift model should predict the same thing as a conventional response model. Their predictions are most different when the variation in the change in behaviour opposite from the variation in the underlying behaviour. For example, suppose the background purchase pattern (the one you see if you don't do anything) is that mostly men by product X, but the effect of a marketing action is to make more women buy it, but fewer men, even though still more men than women buy when treated. In this case, uplift models will make radically different different predictions from "response" models. A response model will concentrate on the fact that more men buy (when treated) that women; but an uplift model will recognize that women's purchases are increased by the treatment whereas men's is suppressed.
6. Q. How do you measure the quality of an uplift model?
   A. Standard quality measures for models (such as gini, R-square, classification error etc.) don't work for uplift models as they are all based on comparing an actual, known outcome for an individual with a predicted outcome. However, since a single person can't be simultaneously treated and not-treated, we can't make this comparison.
   There is, however, a generalization of the gini measure called Qini that has some of the characteristics as gini, but which does apply to uplift models. This has been described in the paper referenced as [1].
7. Q. What are the main application so of uplift modelling?
   A. So far the biggest successes with uplift modelling have been in the areas of customer retention and demand generation (cross-sell and up-sell, particularly).
   The state-of-the-art approach to customer retention is to predict which customers are at risk of attrition (or "churn") and then to target those at high risk who are also of high value with some retention activity. Unfortunately, such retention efforts quite often backfire, triggering the very attrition they were intended to save. Uplift models can be used to identify the people who can be saved by the retention activity. There's often a triple win, because you reduce triggered attrition (thus increasing overall retention), reduce the volume targeted (and thus save money) and reduce the dissatisfaction generated by those who don't react well to retention activity.
   The other big successes have come in the area of cross-sell and up-sell, particularly of high-value financial products. Here, purchase rates are often low, and the overall incremental impact of campaigns is often small. Uplift modelling often allows dramatic reduction in the volumes targeted while losing virtually no sales. In some case, where negative effects are present, incremental sales actually increase despite a lower targeting volume.
8. Q. Are there any disadvantages of uplift modelling?
   A. Uplift modelling is harder and requires valid controls groups to be kept, which have to be of reasonable size. Experience shows that it is also easy to misinterpret the results of campaigns when assessing uplift, especially when it is first adopted. Adoption of uplift models usually results in reductions in contact volumes, which is sometimes seen as a negative by marketing departments. An uplift modelling perspective also often reveals that previous targeting has been poor, and sometimes brings to light negative effects that had not previously been identified.
   There is also some evidence that uplift models also seem to need to be refreshed more frequently than conventional models, and there are clearly cases where either data volumes are not adequate to support uplift modelling or where the results of uplift modelling are not significantly different from those of conventional modelling. Anecdotally, this seems to be the case in the retail sector more than in financial services and communications.
9. Q. How does uplift modelling relate to incremental modelling?
   A. It's the same thing. Various people have apparently independently come up with the idea of modelling uplift, and different statistical approaches to it. There is no broad agreement on terminology yet. Names include
   • uplift modelling
   • differential response analysis
   • incremental modelling
   • incremental impact modelling
   • true response modelling
   • true lift modelling
   • proportional hazards modelling
   • net modelling.
   These are all the essentially the same thing.

References

[1]	Using Control Groups to Target on Predicted Lift: Building and Assessing Uplift Models, Nicholas J. Radcliffe, Direct Marketing Journal, Direct Marketing Association Analytics Council, pp. 14–21, 2007.

Rate Plan

41 Timeless Ways to Screw Up Direct Marketing

Screw Up the Control Groups

1. Don't keep a control group
2. Make the control group so small as to be useless
3. Raid the controls to make up campaign numbers
4. Use deadbeats as controls
5. Don't make the controls random
6. Use Treatment Controls but not Targeting Controls

Alienate or Annoy the Customer

7. Trigger defection with retention activity
8. Use intrusive contact mechanisms
9. Use inappropriate or niche creative content
10. Insult or patronize the Customer
11. Mislead or disappint the Customer
12. Fail to Listen to the Customer
    • Bonus marks for refusing to take "no" for an answer
    • Double bonus marks for calling the customer back after she's said "No thanks, I'm not interested" and hung up.
13. Overcommunicate
14. Make it hard for the Customer to do what you want.
    • (Overloaded websites, uncooperative web forms, understaffed call centres, uninformed staff and failure to carry stock to meet demand are but a few of the ways to achieve this).
15. Intrude on the Customer's Privacy
16. (Over)exploit the customer

Misinterpret the Data

17. Confuse "responses" with "incremental responses" (uplift)
18. Confuse revenue with net revenue
19. Take Credit for That Which Would have Happened Anyway
20. Double Count
21. Believe the name of a cluster means something
22. Believe the data
    • The fact that the computer says it's true and prints it prettily, doesn't mean it is.
23. Disbelieve the data
    • The fact that the data doesn't show what you hoped, thought or expected doesn't mean it isn't so.

Screw Up Campaign Execution

24. Discard response information.
25. Revel in unintended discounts and incentives
26. Use dangerous, insulting or disrespectful labels
    • Yes, I really have known an airline label its bottom-tier frequent fliers "scum class", and a retailer label a segment of its customer "vindaloonies".
27. Ship mailings with the internal fields filled in instead of external ones (see also 26)
    • Dear ScumClass, . . .
28. Direct people to slow, hard-to-navigate websites, IVR Systems or overloaded or poorly designed call centres
29. Fail to inform your front-line staff about offers you're sending to customers
30. Fail to ensure your company can fulfill what the marketing promises
31. Use a name that's very visually similar to a better known name that has negative connotations for many people.
    • I frequently get mail from Communisis, but I never read their name correctly.

Mismodel or Misplan the Campaign:

32. Confuse statistical accuracy/excellence with ability to make money
33. Model the wrong thing
34. Use undirected modelling instead of directed modelling
35. Assume equivalence of campaigns when important things have changed.
36. Ignore external factors (seasonality, competitor behaviour etc.)
37. Fail to make dates relative
38. Contaminate the training data with validation data
39. Screw up the Observation Window (Predict the Past, not the Future)
40. Ignore changes in the meaning of data
41. Fail to sanity check anything and everything

Controls

The Two Controls

Uninformation

Target Marketing

Suspicion

Back

The Scientific Marketer has been on vacation.

But no longer.

Demand Suppression