Teasers

How can someone who writes a blog with the title The Scientific Marketer possibly argue against companies doing everything possible to capture the details of the people who view their content. Surely, the first rule of "scientific marketing" has to be to collect data, for without data there is no science. Right?

A lot of companies, particularly those in the direct marketing industry, seem to take it as axiomatic that you should lock up content and make it available only to people who register with you. The (supposed) "benefits" range from the obvious (you get some contact details to use or abuse) to the more subtle (it acts a sort of qualification; only people who are seriously interested will bother to sign up). It might also be regarded as a benefit that you place a small obstacle to your competitors seeing your material, if that's something you wish to prevent.

The clearest disadvantage of the approach is the risk that some genuine prospects will be put off by the process. Indeed, even if they do register, some people will be annoyed at having been made to jump through hoops. Less obviously, there is a different sort of qualification that the company is denied—the self-qualification of people who bother to contact you as opposed to those you call simply because they filled in a form (often against their will) to gain access to some of your content.

But perhaps the most important disadvantage is that your content becomes almost invisible, so you become harder to find. Search engines adore content, and making the full text of papers available on-line makes people who are looking for specific things much more likely to find them even if they don't know your company. Similarly, people are much more willing to link or otherwise refer to content that is publicly available than to content that is accessible only after registration.

So there are significant pros and cons to hiding content, and clearly each case needs to be considered separately on its own merits. Nevertheless, when I started Stochastic Solutions, I took the decision very early that the company would, as a policy, make content available without registration by default. I can't prove that this has been advantageous, but my strong sense is that, over time, people are becoming ever more reluctant to register before there's an obvious reason for being required to do so.

To connect this back to something I do have real and quantifiable experience of, one of the core lessons from looking at incremental response and uplift modelling is that negative effects in marketing are real and much more wide-spread than is commonly believed. I'm always amazed that anyone is surprised by this, given that almost all of us are daily annoyed by various marketing, but most marketers seem to feel that their own messages will be at worst neutral.

Self-delusion notwithstanding, it's hard to believe there's anyone who, when clicking on a link that promises access to some interesting content, doesn't feel some measure of disappointment or irritation when taken instead to a registration form. Should our intrepid surfer submit to the unwelcome interrogation, even the most sanguine is likely to be less than happy if instead of then being taken immediately to the content, she instead gets a message saying that a link will be emailed or (worse) someone will call. People increasingly expect and demand instant gratification, and most are going to feel suckered if even after jumping through hoops there is another level of delay in being granted access to the content they seek, even if it is in the noble cause of filtering out m.mouse@disney.world, competitors or other undesirables.

So think carefully before locking up content. It may be absolutely the right thing for your business in your situation, but on balance I suspect that most demands for early registration—especially for marketing content—actively undermine the very success that they're trying to generate.

The First Cameron Cabinet (Annotated; Final; Venn)

The Guardian's datablog published a breakdown of the cabinet, which has now been updated to include all 29 members and attendees. The Guardian also provided various information about each member, including sex, education, ethnicity etc. I'm hoping the data is now stable, so that this won't need to change again.

Here's an annotated version Miró's view of it on a 5-D nested Venn diagram, with yellow Lib Dems and true blue Tories.

So on the main Venn Diagram we see that 19 (6 +8 + 4 + 1) of the 29 cabinet ministers so far announced are white males, educated at Oxbridge. Of those nineteen

• six are under 45 and went to a private school;
• eight are 45 or over and went to a private school;
• one is under 45 and didn't go to a private school;
• four are 45 and over and didn't go to a private school.

Of the remaining ten, six are white males.

The raw data is available from the Guardian Datastore; the original table I worked from (as presented by Miró) is here:

Although Miró did most of the work, I did annotation, so it's entirely possible that I have mislabelled the discs.

Cabinet Breakdown (improved)

Here's a slightly improved version of that breakdown of the new cabinet, with the natural colour coding (Lib Dems in yellow, Tories in blue).

The New Cabinet (so far): The Nested Venn Diagram

The Guardian's datablog has just published a breakdown of the cabinet, as announced so far, by various attributes such as sex, education, ethnicity etc. Here's Miró's view of it on a 5-D nested Venn diagram.

So on the main Venn Diagram we see that 15 (1 + 5 + 5 + 4) of the 22 cabinet ministers so far announced are white males, educated at Oxbridge. Of those 15, 5 are under 45 and went to a private school, 5 are 45 or over and went to a private school, 1 is under 45 and didn't go to a private school and 4 are 45 and over and didn't go to a private school.

Of the remaining seven, five are male and white but didn't go to Oxbridge (1, privately schooled, all 45 or over).

The two who stand out as different are the two women, Baroness Sayeeda Warsi (lower right red disc), who is not male, white, Oxbridge-educated, privately schooled or 45 or over, and Theresa May (next lowest, next right-most red disc), who wasn't privately educated.

The raw data is available from the Guardian Datastore, but since it may be updated, the original table I worked from (as presented by Miró) is here:

Improving the Journey History on the Oyster Website

If you travel by tube in London any significant amount at all, you probably have an Oyster card. It's a pretty fantastic system: it's easy, it's reliable, and it's much cheaper than paying cash fares. You can register your card so that tops up by direct debit whenever the balance falls below some threshold amount, and if you do that (a) you obviously never find yourself unable to travel, and (b) get to see your journey history online, which can be useful for various reasons.

Unfortunately, the online journey history features some of the worst presentation of information you are ever likely to see. Here is my journey history for March.

If you can be bothered, stare at it till you figure it out. Or don't. Here is how I would present that information (in a table, using text of the same font in the same size).

I think this would be clearer and better for very nearly everyone who looked at the information.

I suspect that what has happened is that whoever designed and implemented the system decided, for whatever reason, to make the table represent the underlying data in the Oyster system as closely as possible. This could have been through laziness or ignorance, but could have been a conscious decision, perhaps based on a belief that this directness would have benefits. But the result is a disaster for comprehensibility and clarity.

I hope it will be immediately clear why I think this is a better layout, but I will list the reasons anyway (partly because I'm hoping someone from Oyster or LRT will read this and make changes).

1. The far of a journey is shown. Incredibly, none of the numbers in the Oyster table is actually a fare in the sense that almost everyone would understand it.
2. The opening and closing balances are shown clearly.
3. There are no confusing negative amounts.
4. Time runs forwards down the page, as is conventional and familiar, e.g. from bank statements. (I accept there is an argument for reverse ordering; I think it is weak, but if you made the other modifications and left the reverse ordering, I still think you would achieve 90% of the benefit.)
5. Journeys occupy a single line.
6. It doesn't use the extraordinary convention of labelling the column 'Fare' but then making negative amounts costs and positive amounts credits. (It's true that accounting conventions usually make debits negative and credits positive, but in these cases the column is usually labelled 'Amount', not Cost (or Fare). A negative fare is a credit or refund.)

Obviously this design requires a litle more width than the current layout, but I don't believe it is so wide as to cause a problem, weighing in at around 600 pixels against the current 520 or so. If width is a major consideration, text could be wrapped in the "From" and "To" sections and even in the time columns, but I don't believe to be necessary.

Clearly, my short transaction history does not represent the full complexity possible. In particular, the current design features a price cap column, there is the possibility of people registering an entry but not an exit, and late-night journeys may span midnight. I'm sure there are other possibilities too.

I contend that whatever such complexities might exist, it is extremely poor design to overcomplicate the presentation of the common case for the sake of accommodating the rare cases uniformly.

In the specific cases identified, I would suggest the following, though better solutions may be available.

Entry without Exit

Show the entry date, time and station; leave the exit time and exit station blank or (better) show "(no exit recorded)" in the To column; show the actual cost applied as the fare (£6.00?)

Capped fares

Just show the fare applied. If there is a pressing need to indicate that it was a cap, add an asterisk to indicate this. If there is a really pressing need to indicate the size of the saving, either add a column for this or make the fare clickable or add hover text to show the uncapped fare. (My feeling is that people don't need to see the capping information. If you are keen to emphasize the saving, perhaps add a note under the table indicating the total saving from fare caps.)

Journeys spanning midnight

Just put the entry date; I think people will figure out that a journey from 23:30 on 16/3/2010 ending att 00:10 probably took 40 minutes rather than the alternative possibility of having spanned a negative time period.

Why blog?

Why am I blogging about this not just mailing Oyster? Well, there are couple of reasons. The first, is that I tried to mail Oyster through their website. After establishing that my feedback related the website, I was taken through a long and complicated form that included mandatory fields for my date of travel, my approximate time of travel (to the nearest minute), my Oyster card number and much more besides. All this I cheerfully provided, meaningless though it was. I was then presented with an input box perhaps 40 characters wide and 3 lines deep in which to share my deepest concerns with Oyster.

Rather amazingly, the painful inadequacy of my box turned out to be a benefit, because it caused me type the message in an editor and then paste it in. This was good, because on finally clicking submit I was met with

500 INTERNAL SERVER ERROR

(almost suggesting a buggy website).

But I confess that I was considering blogging anyway, not to "name and shame" Oyster, who on the whole I think provide an excellent service to Londeners and others; but because while poor information design is all around us, presentation as poor as this is rare, and perhaps does serve as a good illustration of how simple changes can move something from virtually incomprehensible to pretty clear. (Though, as they say, your mileage may vary.)

Venn in Wonderland: Paper on Nested Venn Diagrams & Lewis Carroll Diagrams Available

I did a bit more work on those nested Venn Diagrams, including research that turned up an alternative to Venn Diagrams produced by Lewis Carroll, known naturally enough as Lewis Carroll Diagrams.

You may recognize the two-set version as the same structure used in the more recent "Boston Box".

It transpires that Carroll used nesting of his diagrams as far back as 1896. Here is his diagram for eight sets.

All this and more is described, with including the relationship to the Nested Venn Diagram I described in the previous post, in a paper entitled "Nested Venn Diagrams" from Stochastic Solutions.

The Nested Venn Diagram

Although this will be a long blog post, the essence of it is a single image, which I'm hoping is all you need to know. Here is the Big Idea, the Nested Venn Diagram:

If the picture is immediately self-explanatory, you need read no further; all else is mere elaboration, and I am a happy man. The six sets illustrated relate to the twitter users named (all members of the Guardian Technology's team) and the numbers in the intersections show the number of people they follow in common. At the centre, you will see that Jack Schofield (@jackschofield), Charles Arthur (@charlesarthur), Bobbie Johnson (@bobbiejohnson), Aleks Krotoski (@aleksk), Jemima Kiss (@jemimakiss), and Victor Keegan (@vickeegan) follow five users in common. Similarly, You can see that Aleks and Jemima follow six people who none of the men do, and that the men all follow two who neither of Aleks or Jemima do. (Note, this was as at 10th February 2009; obviously the following relationships may change.)

If you want to find out who they follow in common, tickery.net which lets you look at the intersection of any set of twitter users following relationships. (The links above use Tickery.) (Disclosure: Tickery is built by FluidInfo on its wonderful Fluid DB database; I am a shareholder in and advisor to Fluidinfo Limited.)

The Back Story

A client wanted, among other things, a Venn Diagram to show the which combinations of web sites a set of users visited. This presented two challenges. First, my software packages of choice (Miro and Klee), didn't technically support Venn Diagrams at the time of the request. That, however, was easily solved; after all, it's just code. The second problem was more serious. The number of websites he wanted to illustrate was not two or three, or even four, but six.

Six!

A six-dimensional Venn Diagram is a challenge. I had a vague recollection that no lesser person than Venn himself had come up with a construction that in principle allows an Venn Diagrams to be constructed in an arbitrary number of dimensions. But I also recalled that whenever I looked at such constructions, my head hurt. As Vic Reeves1 might say, well over 99 per cent of all Venn Diagrams in standard use show either two or three sets. I have seen four; I don't believe I have ever seen five used for anything other than explaining how to construct a five-dimensional Venn Diagram. If you're interested, here is Venn's constructions for five dimensions

which compares to my nested venn diagram construction,

and here is his construction for six sets

which compares to the nested Venn Diagram at the top of this article. (The images illustrating Venn's constructions were lifted from the Wikipedia article on Venn Diagrams, and were provided by Kopophex. Thanks, Kopophex.)

My solution, as you have probably gathered, is nesting. The image below shows all sixty-four possible memberships for six sets, which I have imaginatively labelled A through F. The large circles represent sets A, B and C; each of the small Venn Diagrams represents sets D, E and F. By placing a copy of the small Venn Diagram, in each of the eight positions corresponding to the various intersections of A, B, and C, we get a unique position on the diagram for each of the 64 combinations of set memberships for A, B, C, D, E and F. In case this isn't clear, here is a labelled version.

While this solution is far from perfect, so far the reaction from colleagues and others seems to have been positive. Certainly, I find this representation incomparably easier to digest than Venn's clever-but-extremly-difficult-to-read versions. And more significantly, on two occasions I have now gained insights from using these that I had previously failed to elecit from the data by alternative methods. I will follow-up, as time and clients permit, with some examples of their use.

Generalizing Further

It goes without saying that this technique can easily be generalized, to nesting copies of any n-set Venn Diagram in the various intersections of an N-set Venn diagram to yield a nested Venn Diagram in (n + N) dimensions (i.e for n + N sets). In principle, one could obviously go even further, nesting an arbitrary number of levels, but I have severe doubts about the utility of nesting more than once. I had thought that six was probably the largest number of dimensions (sets) the technique handles elegantly, but in fact have now implemented versions for seven and eight sets. Extending the Guardian Tech team to include its new member, Mercedes Bunz (sic; @MrsBunz), we get:

and adding in Kevin Anderson @kevglobal, we get to:

This obviously leaves just one question: who is the one person worthy of being followed by all eight (of these) Guardian Technology writers? You'll have to go to Tickery to find out.

Notes

1 for it was he, you will recall, who made the famous observation that 88.2% of statistics are made up on the spot. ↩

Parlez-vous Français, Miró?

$ miro -f speak-to-me

This is Miro, version 0.2.33.
Copyright (c) Stochastic Solutions 2008-2009.
Seed: 1252839009
Logging to /miro/log/2009/09/13/session018

[1]> echo Hello from Miró
Hello from Miró
[2]> echo I am ☺ ; really ☻ !
I am ☺ ; really ☻ !
[3]> echo I can say  枝豆 too.
I can say 枝豆 too.
[4]> echo Très bon, Miró!   Du er meget sød.
Très bon, Miró! Du er meget sød.
Logs written to /miro/log/2009/09/13/session018

What about numbers?

$ miro
This is Miro, version 0.2.33.
Copyright (c) Stochastic Solutions 2008-2009.
Seed: 1252839325
Logging to /miro/log/2009/09/13/session019

[1]> load hillstrom
hillstrom.miro: 64,000 records; 64,000 (100%) selected.
[2]> count mean spend by segment
      segment     count    mean (spend)
Womens E-Mail    21,387         1.08   
Mens E-Mail      21,307         1.42   
No E-Mail        21,306         0.65   
<null>                0       <null>
[3]> 
Logs written to /miro/log/2009/09/13/session019

But what about Europe?

$ echo $LANG
da_DK.UTF-8

$ miro
This is Miro, version 0.2.33.
Copyright (c) Stochastic Solutions 2008-2009.
Seed: 1252839453
Logging to /miro/log/2009/09/13/session021

[1]> load hillstrom
hillstrom.miro: 64.000 records; 64.000 (100,00%) selected.
[2]> count mean spend by segment
      segment     count    mean (spend)
Womens E-Mail    21.387         1,08   
Mens E-Mail      21.307         1,42   
No E-Mail        21.306         0,65   
<null>                0       <null>  
[3]> 
Logs written to /miro/log/2009/09/13/session021

Miró's Control Group Size Calculator Now Available

Rather surprisngly, the first component of Miró to be released is a control group size calculator. You can use it here. I'll blog a bit about what it does and why it exists over the coming weeks. There are help links for descriptions there too.

There's also slightly more information about Miró itself now online here.

Miró is Coming

It has been pointed out to my by more than one person that the Scientific Marketer has been eerily quiet of late. I can only apologise for that. I'm not going to do anything so rash as promise, but I hope to return to something like normal service.

Anyway, there's news. Miró is coming.

Actually, that's quite big news.

But what is Miró?

It seems to be an anagram of ROMI (Return On Marketing Investment). Coincidence? (Yes; but not an entirely unhappy coincidence.)

Or does it Miraculously Increase Return On, . . ., something? anything? (It can. Well, maybe not miraculously; but measurably.)

Could it be one of those clever-clever recursive acronyms? Miró is ROI Optimizing. Miró Implements Randomized Optimization. Miró is Ridiculously . . . opportune? (No.)

Is it an acronym at all? Has the Scientific Marketer (and Stochastic Solutions, whose product it will be), gone creative and decided to branch out into software for budding artists? (Readers of this blog will appreciate that artistic talents are not top of the list for the Scientific Marketer.)

Is there a coded message? Marketing is Really Optimization, perhaps. (Well it is. But . . .)

All will be revealed over the coming months, when the intention is to make snippets of Miró functionality available (free!) as web applications. For those who dislike teasers, my apologies for engaging in such low behaviour.

In the meantime, I can confirm is that Miró is software that will, in time, be marketed by Stochastic Solutions in some form. I can also confirm that it's real, not vapourware. In fact, it has been used in various forms for over a year by two clients for production work. It's been in the works since January 2008. But it's not finished. (Well, when is software ever finished? What I really mean is that even at version 1.0 yet.) I can also confirm that it's cross-platform (most Unix/Linux, Mac and PC platforms are supported) and tries very hard to be standards-compliant, so the only organizations who won't be able to use it are those whose IT departments mandate Internet Explorer as the one and only web browser. (Actually, even they can use it; it's just that its web functionality will be at best compromised, and at worst fatally non-operational. So if you work for a company with such an IT department, start lobbying them to authorize at least one standards-compliant browser now. The list is long and rich—Firefox, Opera, Safari, and Chrome all fit the bill, and though I haven't tried recently, I suspect Konquerer and friends will as well. Your life will be better with access to one of these anyway.)

Stochastic Solutions will be looking for early adopters and beta testers at some point over the next 12 months or so. I'll put a form up somewhere for people to register if interested. There will be attractive terms for early adopters and testers, but it will be a closed programme, with only companies meeting fairly strict criteria qualifying.

I can also confirm, that one of Miró's interfaces is a command-line interface.

A what? Didn't those go out with the ark? Did you really say a command line interface? Didn't you get the memo??

Yes, it's true. Miró has—among other things—a command line interface. I can reveal that one of Miró's commands is cartoon, which as you may realise, isn't necessarily promising for a command-line interface. But the boffins over at Stochastic Solutions have been working an it, and I can now proudly present the first ever (acknowledged) public output from Miró.

This is Miro, version 0.2.16.
Copyright (c) Stochastic Solutions 2008-2009.
Seed: 1251628773
Logging to log/2009/08/30/session005
Log started at 2009/08/30 11:39:33.

[1]> cartoon

+------------------+------------------+------------------+------------------+
|  HAVE YOU SEEN   |   ACTUALLY, I    |   NONSENSE! I    | BUT I CAN PROVE  |
|    THE SALES     |  THINK YOU'LL    |  SPENT MILLIONS  |  IT. I HAVE A    |
|     FIGURES?     | FIND 93% OF THE  |   ON THOSE ADS.  |  CONTROL GROUP.  |
| THAT'S ALL DOWN  | EXTRA SALES CAME |  YOU CAN'T TAKE  |             |    |
|  TO MY ADS, YOU  |  FROM MY DIRECT  |  CREDIT FOR MY   | BASTARD!    |    |
| KNOW! BRILLIANT! |   MAIL PIECES.   |      SALES!      |     O       |    |
|    /             |            \     |    /             |    o        |    |
|   ___     ____   |   ___     ____   |   ___     ____   |   ___     ____   |
|  /   \   /    \  |  /   \   /    \  |  /   \   /    \  |  /   \   /    \  |
| | O O | | o o  | | | O O | | o o  | | | O O | | o o  | | | O O | | o o  | |
|  \_=_/   \_==_/  |  \_=_/   \_==_/  |  \_=_/   \_==_/  |  \_=_/   \_==_/  |
|    |       ||    |    |       ||    |    |       ||    |    |       ||    |
+------------------+------------------+------------------+------------------+
http://scientificmarketer.com/           Copyright (c) Nicholas J. Radcliffe.
                          All rights reserved.

Command completed in 0.0010 seconds


Job completed after a total of 11.7506 seconds
Logs written to log/2009/08/30/session005
Log closed at 2009/08/30 11:39:33.

So there you have. I'll be blogging about some of Miró's more mainstream and obviously useful capabilities over the coming months. And, as I say, hopefully releasing bits of functionality as web apps.

Off-topic Afterword

I should also mention that although I haven't been blogging here as the Scientific Marketer, I have been involved in a very exciting start-up called Fluidinfo, whose product, FluidDB launched as a "private alpha" last week, and I have been blogging about that at About Tag. Unfortunately, FluidDB is quite hard to describe succinctly, but we think it's going to change the world. Briefly, it's a online database based on tags, giving anyone the ability to put small (or large) amounts of data into the world in a structured and sharable manner. The founder, Terry Jones, has a blog piece called Kaleidescope describing ten different ways of looking at FluidDB. If you're interested in a glimpse of a possible future, you might want to check it out.

Clustering Considered Harmful III: Commensurability

[This is the third in a planned series of posts on clustering that I'll publish over the coming period. The first part is here. When it's finished, I'll collect it all together into a properly printable paper.]

I have always found it surprising that the standard shorthand for a dubious or invalid comparison is apples and oranges. For while, granted, apples and oranges are different in many respects, they're not so different, and a comparison is not completely absurd. I would have thought that fruit flies and symphonies or physicists and personal computers3 or prime numbers and dictionaries might all present more obvious challenges as subjects for meaningful comparison. But perhaps that is the point: people would be unlikely to find themselves inadvertently trying to compare a fruit fly and a symphony4 whereas one can more easily imagine unwittingly comparing, say, a Cox's Orange Pippin with a Seville Orange.

There is a general problem with comparing, and in many cases with combining, so-called non-commensurate variables. The Shorter Oxford English Dictionary defines the adjective commensurate thus:

1. Having the same measure; coextensive. Const. with, to 1641.

2. Of corresponding extent or degree; proportionate, adequate. Const. to, with. 1649.

3. Corresponding in nature (with, to) – 1678.

4. = COMMENSURABLE 1 (rare) 1690.

At the risk of labouring the point, our three spatial dimensions are perfect examples of commensurate quantities: if we lay down an ordinary rectilinear x-y-z coordinate grid, there is a very strong sense in which going one mile parallel to the x-axis is equivalent to travelling one mile parallel to the y-axis or one mile parallel to the z-axis. A mile is a mile is a mile, and we feel very comfortable applying the notion of distance in an arbitrary direction and computing it in the usual Euclidean/Pythagorean manner.

We are on more treacherous ground when our dimensions consist of non-commensurate quantities, as only some of those who study or practise co-called cost-benefit analysis appear to be aware. For the approach of cost-benefit analysis requires us to express all the costs and all the benefits of some proposed course of action (or of some set of alternatives) in terms of a single numéraire—almost always money. Thus, when planning a motorway, we must express not only the financial cost in financial terms, but also such other diverse costs as habitat destruction, contribution to global warming, displacement of families from their homes, increases in pollutive emissions and likely increase in road traffic deaths in pounds, shillings and pence. Similarly, benefits must be expressed in cash terms. To do this, of course, requires suitable equivalences and conversion factors to be put in place (human life lost—£1 million; contestible contribution to possible making of planet uninhabitable by humans at some indeterminate point in the future—£2,452.13). This has led some cynics to suggest that cost-benefit analysis, as practised, is not so much a rational and objective way of collating all relevant evidence and drawing from it the optimal conclusion as a tool for justifying some desired outcome by careful manipulation of the many and infinitely negotiable parameters absolutely required in order for the method to function.

It will not have escaped the reader's attention that we have a similar problem with cluster analysis if we wish to apply it in situations where the factors to be considered as dimensions are not commensurate. Unfortunately, this is almost always the case in marketing, and it this non-commensurability of the inputs, and consequentially necessary setting of many essentially arbitrary conversion factors that forms the first objection to the common application of cluster analysis to customer database segmentation. The clusters that result from a clustering exercise are not merely dependent upon the equivalences and conversion factors chosen for the distance function, but are almost entirely determined by those choices.

As a minimum, this should cause us to treat with extreme scepticism the all-too-common suggestions that cluster analysis impartially and objectively uncovers inherent, underlying or intrinsic structure within the customer data. It might be more accurate to say that it allows the consequences of different choices of equivalence among input variables to be revealed through the mechanism of mapping a distance function to a cluster structure.

David Hand's Example

I'll close this post with a graphic illustration of the effect of scaling based on an example provided by David Hand.5 Consider clustering four points, A(–1, 10), B(1, 10), C(–1, –10) and D(1, –10), as sketched below.

Clearly, points A and B form a cluster, as do points C and D. (The within-cluster distances are small and the between-cluster distances are large; these are defining characteristics of a good clustering.)

Now let us rescale, multiplying the x-coordinate by 10 and dividing the y-coordinate by the same amount. The result would be the following.

Now, clearly, points A and C form a cluster, while points B and D form another.

Or consider multiplying the original x-coordinate by 5 and dividing the original y-coordinate by 2. Now, our plot looks like this.

Our clusters have disappeared entirely!

To be clear, it is not my claim that clustering is always invalid or meaningless: if the variables were commensurate on the original scale, then A and B really do cluster, as do C and D. The point is that scale is crucial, and extremely problematical when for non-commensurate quantities.

Next up: direction (probably).

Notes

3 Though even as I write this I'm thinking Paul Dirac—definitely a Mac kind of a guy; very concerned with elegance and form; happy to do a few things well; a true hero among a narrow subset of scientists. I hesitate slightly at the prospect of nominating a physicist as the natural Wintel PC analogue, but sticking with fathers of the quantum theory, perhaps Nils Bohr, who arguably thought that the key thing was not to spend too long sweating the details like meaning and elegance but simply to get some computational tools into as many people's hands as possible to allow them to make some progress with the job in hand. ↩

4 'How long is Drosophila melanogaster? Around 2.5mm you say. And Beethoven's Ninth? 74 minutes. Hmm. Let's convert those both to natural units using the speed of light, c as the conversion factor . . .' ↩

5 Discrimination and Classification, D. J. Hand, John Wiley (Chichester), 1981. ↩

Clustering Considered Harmful II: Distance and Scaling

[This is the second in a planned series of posts on clustering that I'll publish over the coming period. When it's finished, I'll collect it all together into a properly printable paper.]

I said in the previous post that pretty much all you have to do to apply cluster analysis to a dataset is to pass in the coordinates of each point to be clustered. And this is the case when the points exist in Euclidean space or something similar—anywhere, in fact, where there is a well-defined, unambiguous and reasonable distance function (metric).

But whereas we can ordinarily agree the distance between each pair of objects in our solar system (at a given point in time), or indeed, between each pair of people on the earth ("as the crow flies"), defining the distance between a pair of customers in a customer database requires rather more decisions to be taken.

Numeric Variables with Common Units

First, even for those variables or dimensions that are ordinary numbers, there are potential questions of units and scaling. Even if all they are all expressed as units of a common currency (say, pounds sterling), there might be a question as to whether, for example, a difference of £1,000 in value of house (say between £248,000 and £249,000) is really equivalent to (say) a difference of £1,000 a month in interest payments (perhaps between £200 and £1,200); or indeed, whether it is useful to treat either of these as equivalent to a difference of £1,000 a month in change in savings balance (say between +£500 and –£500).

Numeric Variables with Different Units

There is even greater scope for argument when the variables measure fundamentally different kinds of things. Consider, for example, the variables number of children, and income. Suppose we measure these for four people as follows:

Customer	Number of children	Income
Ann	1	£15,000
Bianca	1	£40,000
Ciara	0	£15,000
Dee	3	£20,000

Which of Bianca, Ciara and Dee is closest to Ann? What is the appropriate conversion rate between number of children and income? There is, of course, no correct answer. It depends what we mean; and what we are trying to achieve.

Z-Scores

It is important to emphasize that the nature of the challenge here is not technical: there is no difficulty at all in devising conversion rates, and even rationales for those conversion rates. One particularly common approach is to compute so-called z-scores, by subtracting the population mean from each variable and dividing the result by the standard deviation, i.e.

z(x) = (x – μ) / σ.

While this approach unarguably provides an unique and unambiguous way of setting the determining a scale among numeric variables, it does rather seem to fly in the face of the very object of our exercise, which is to understand the inhomogeneities.

To illustrate why z-scores are not really a solution, suppose we were trying to cluster stars in our galaxy (space being a domain in which the three spatial dimensions clearly are commensurate). As is well known, the Milky Way is essentially a disc whose thickness is a mere thousand light years but whose diameter is some hundred times greater, i.e. roughly 100,000 light years. Let us suppose we place the origin of a (Euclidean) coordinate system at the centre of the galaxy, with x- and y-axes in the plane of its disc, and a z-axis perpendicular to it. Then, using the z-score approach, we would treat the average distance from the galactic centre perpendicular to the plane of the galaxy (presumably rather less than 250 light years) as being equivalent (in fact, equal) to the average distance from the centre in the plane of the disc, which we may suppose is more like 20,000 light years. Needless to say, this is madness.

Categorical Variables

While it is not necessary to run through every type of data that might occur, categorical variables are common enough to merit some discussion. Whether coded as numbers or strings, the defining characteristic of categorical data is that the values are unordered. Common examples in customer data include gender (typical values male; female) house type (flat; terraced; semi-detatched; detached) or last purchase (Film: Monty Python's Life of Brian (DVD); Book: The Unbearable Lightess of Being, Milan Kundera; Beverage: 70cl bottle of Ardbeg (10 yr) Single Malt Whisky). In some cases, the different values are all simply different—islands unto themselves—while in others they may fall into some (possibly partially ordered) structure such as a hierarchy; or indeed into several hierarchies.

When considering the distance between two members of our customer base, how then should we handle categorical data? And how should we combine categorical and non-categorical variables into a single, meaningful distance measure? Is the difference between a man and a woman zero, £1,000, 2.5 inches, 15lbs, 30 points of emotional intelligence or is the question too silly for words?

It is again important to be clear that the challenge is not technical. We can certainly use the discrete metric and simply say that if the two genders or house types or most recent purchases are the same, then the distance between them is zero, and if they are different it is one. But as before, it is hard not to feel that this solution is more satisfactory from the perspective of one who seeks to hammer the square peg of customer reality into the round hole of clustering than to a either a passionate or dispassionate inquisitor seeking truth and insight into her customers; it is the tidy-but-unhelpful bureaucrat's solution.

In the next part of this series, we will focus on the question of commensurability.

Clustering Considered Harmful: I. Introduction

[This is the first in a planned series of posts on clustering that I'll publish over the coming period. When it's finished, I'll collect it all together into a properly printable paper.]

Lumpiness

Wherever we look we see inhomogeneity. The universe is lumpy at all scales. Stars clump together as galaxies, galaxies form clusters, and clusters themselves cohere into superclusters. More locally, almost all the matter in our solar system is in the Sun; most of that which isn't is in Jupiter; and almost all the rest is tightly packed into other planets. Even towards the bottom, matter is lumpy, each atom being made up of a tiny sun-like nucleus surrounded mostly by empty space and a very diffuse cloud of point-like electrons.

People clump too. Rather than spreading ourselves out uniformly over the planet's landmass, we form dense populations in some areas (Monaco; Singapore; Bangladesh) and inhabit other places more sparsely (Greenland; Australia; Canada). Similarly, most countries have their high-density cities (Mumbai; Karachi; Lagos) and their sparse regions (the Gobi; Wyoming; the Rannoch Moor).

Truly, this is one lumpy universe.

Cluster Analysis

Cluster analysis is the name given to a collection of techniques for finding the clumps (or clusters). The idea is simply to make a list of the coordinates of the items of interest (the stars, the people or the atoms), possibly to set some parameters (like a target number of clusters), then to feed the list into the algorithm which divides it into groups—or "clusters". Typically, the clusters will be characterized by a cluster centre (the coordinate of some centre of gravity of its members), and perhaps some measures of compactness.

There are two main families of techniques for clustering: "bottom up" techniques start by assigning each point to its own cluster and then merge clusters in close proximity (so-called agglommerative or accretive methods), while "top-down" approaches begin by creating a single cluster and then look for ways to divide it and its children into ever smaller subclusters (so-called divisive approaches). There are interesting and bizarre variations, particularly so-called self-organizing maps, of which the Kohonen Network is a popular example with appealing properties.

Numerous algorithms exist, and many software packages, free and commercial, implement one or more. I will mention David Wishart's Clustan as a fine example, and David has applied his software and his approach widely, not least to the important problem of classifying single malt whiskies1 (though I personally favour an approach based more on repeated sampling without replacement . . .)

From Matter and Space to Customers and Marketing

Though I do not know, I imagine that somewhere in the not-too-distant mists of time, a marketer wanting to understand his (or her) customers must have read about cluster analysis and had the idea of applying it to his customer database. "After all," he perhaps reasoned, "we often think of the columns in our database as dimensions. Maybe, just as stars naturally clump together in space, customers naturally cluster in the virtual hyperspace of my database; and if so, perhaps cluster analysis could help me to uncover the natural, intrinsic, underlying structure within my customer base. Separating my customers into their natural groups," he may have continued to reason, "will surely help me to design marketing programmes that are well tuned to their different needs, attitudes, aspirations and behaviours."

Whether it happened quite like this or not, certainly it has become a practice among some marketers to do just this. The rest of this series will critically examine this approach and (ultimately) argue that there are some fairly major challenges and problems with the application of clustering methods to customer databases, and that companies' interests would often be better served by alternative, and more direct approaches.

In the next part of this series (now published!), we will focus on the questions of distance and scaling.

Notes

1 Whisky Classified. David Wishart, Pavilion Books, (London) 2006. ↩

IVR Master Control Panel

Some Don'ts for IVR Systems

Printable Version (PDF)

1. Exist

How much do you hate your customers?

The purpose of this article is to try to reduce the pain that interactive voice response (IVR) systems inflict upon the humans. IVR systems, or Customer Alienation Systems, as Herb Edelstein, of Two Crows Consulting1 calls them, are those automated telephone answering systems beloved of banks, utilities, and other companies, that lead us through a tree of menus "in order to help [them] to assist [us] more quickly".

There are many stated justifications for using IVR systems, the bulk of which focus on the (legitimate) desire for businesses to reduce costs, and the more tendentious of which sometimes go as far as to claim actual benefits for customers.

There is no disguising the fact that the starting point for this author is that IVR systems are the problem, and that ideal solution is to replace them with a system based on a more sophisticated components known as human beings.2

So the first "don't" for IVR systems is exist; get this one right and you can ignore the rest of the article.

2. Service the Wrong Number

"I'm afraid you've come through to the wrong number, Sir."

Some problems with IVR start before the customer arrives. One of these is to sit behind so-called "non-geographic" telephone numbers (0845, 0870 etc. in the UK), as increasingly these are some of the most expensive numbers for people to call; obviously, the pain of languishing in IVR hell is heightened by knowing that the experience is costing you an arm and a leg.

The other sense in which an IVR system can sit behind the wrong phone number is for its number to be given out in inappropriate contexts. The most egregious such error arises when calling the most prominent number on a letter, email or website takes you through to a system that isn't suitable for calls directly related to the matter on said letter, email or website.

3. Waste Time

Welcome to The Scientific Marketer ("the world's most cantankerous marketing website") and THANK YOU for calling. The Scientific Marketer always remembers that YOU have a CHOICE and could just as easily get your marketing advice elsewhere. Please listen carefully to ALL of the following options before making your selection and please note that menus have changed.

4. Lie

"Your call is important to us."

Yes, we've installed a customer alienation system, which might unwittingly give the impression that we care nothing about your time and aren't prepared to offer you the access to a human being you crave without first putting you through confusing and labrynthine menus and indeterminate waits, and generally trying to get you off the phone before an operator becomes available, but—contrary to all evidence—your call is important to us.

"We are currently experiencing unusually high call volumes."

Unusual. a. 1582. Not often occurring or observed, different from what is usual; out of the common, remarkable, exceptional.
   —The Shorter Oxford English Dictionary.

Tornados in England are unusual; several customers calling a call centre at lunchtime is not unusual.

"Did you know that we have a dedicated easy-to-use website at double-u double-u double-u dot sci-en-ti-fic-mar-ket-er dot com that allows you to do nearly everything that you can do at the call centre with no queuing and no outrageous non-geographic telephone charges? At the Scientific Marketer website you can read articles, leave comments, read other users' comments, report problems, learn how administer control groups, book an expert one-to-one consultation with our highly trained expert marketing professional—in fact, almost all the things you'd expect to be able to do at a website (Microsoft Internet Explorer 6.0 or Netscape Navigator 4.1 required)."

People who have current web access and like using the web have already tried that; people braving IVR hell either don't use the web, don't have access to the web at this time, don't want to do this particular transaction on the web or have already tried and failed on the web. One of the few things more annoying than failing to achieve something on the web is sitting in IVR hell for a while, giving up and going back and trying it on the web one more time, discovering that (as you thought) it doesn't work on the web, and then going back into IVR hell at the back of the queue, only to be lectured on how much better life could be on the web.

5. Run Security Checks too Early

"I just need to know whether your call centre will be open on Sunday."

It's alright. You can tell me that. Even if I don't know my PIN.

6. Forget

"Now, could I just begin by taking a few security details from you."

"But I just spent the first half of my morning telling your computer those exact same things."

"Ah yes, Sir, but we don't have access to that system here. Now, if you could just tell me the name of your great grandmother's first boyfriend . . ."

7. Deafen Your Customers

"Don't Keep Me Hanging on the Telephone."
   —Blondie.

There is a problem with cavernous silence on a telephone line; customers tend to think that they've been cut off. It used to be that the solution to this was to present comfort tones—a soft beep, every few seconds, just to let the customer know that (s)he hasn't been cut off. Unfortuately, comfort tones become annoying fairly quickly and suggest to a minority of customers that they have been cut off. So it is now more common to play music at them.

There are several problems with this. One is volume, which commonly seems signficantly above "comfortable". Another is the paradox of selection: you want neither something people dislike (obviously) nor something they like (less obviously) because there is no such thing as music that sounds reasonable over a tinny, distorted, noisy phone line, and people especially don't want music they actually like murdered in this way. So the music is wrong for most people most of the time.

There is no solution based on making a better choice.3

8. Keep Customers in the Dark

"There are 5 customers ahead of you in the queue."

"Most calls at your current position in the queue are answered within 5 minutes."

There is no excuse for leaving customers in the dark about how long they are likely to have to wait. Allowing them to see their progression along the queue, and giving them the best possible information about how much longer they are likely to have to wait is extremely helpful.

The goal, of course, is to underpromise and over-deliver: people rarely complain about a call being answered sooner than an estimate. I've worded the second piece of information quite carefully: "most calls from your current position in the queue are answered within 5 minutes". The trick here is that you can be accurate (which is important) but also conservative. Interpret "most" not optimistically (at just over 50 per cent) but pessimistically (say at 90% or 95%). You don't want to be so pessimistic that you drive people away needlessly4, but you do want to deliver on your promise in the vast bulk of cases.

9. Put Your Customers on Hold

"I'm just going to have to put you on hold for a second, Sir"

No doubt there are times when customers absolutely have to be put on hold. Try to design these out. Customers hate it. Calls get mysteriously dropped by the system. Customer's give up. They get subjected to more muzak. They get more frustrated.

Recognize that having to put a customer on hold is a failure.

10. Drop Your Customers Calls

[disconnected tone, usually after long wait]

I've personally never been sure whether random disconnection is an approach to call volume management or simply a sign of low-quality systems; however, calls to call centres do seem to get dropped more than almost any other kind of (non-mobile) call.

11. Exploit Queuing Users

"Did you know about the range of great marketing advice freely available from the Scientific Marketer? For substantially less than the cost of a mailshot we could advise you not to bother. Or why not check out our great range of white papers on retention, freely available at double-u double-u double-u dot sci-en-ti-fic-mar-ket-er dot com forward-slash retentionWhitePapers dot html (remember to use a capital W in White and capital P in Papers; registration required.5) Do you really understand . . .?

12. Use Voice Recognition

"I'm now going to ask you to answer a few simple questions. Please answer by speaking in your normal voice. For example, if your name is Nick Radcliffe, please say `Nick Radcliffe', in your normal voice, at your normal speed, without touching your telephone keypad. Now, please state your join date in the form day, month, year. [pause] Please speak after the tone. [pause] [beep]

"Thirty March Two Thousand and Four,"

"I'm sorry, I didn't quite get that. Please answer by speaking in your normal voice at your normal speed, without touching your telephone keypad. Now, please try again. State your join date in the form day, month, year. [pause] Please speak after the tone. [pause] [beep]

"Thirty March Two Thousand and Four."

"I think you said [pause] Thirteenth March Two Thousand and Four. [pause] If that is correct, please say `yes'. If that is incorrect, please say `no'. [pause] Please speak after the tone. [pause] [beep]

[Sound of customer pulling out fingernails.]

One day, voice recognition systems might be able to understand human speech significantly better than other human beings.6

Today is not that day.

13. Use Hierarchies or Dead Ends

If you are calling to congratulate the Scientific Marketer on its new IVR system, press one. If you are calling to tell the Scientific Marketer that you love its new IVR system more than life itself, press two. Please make your selection now.

Not all calls fit into your predefined menu structure. Given a choice between A, B, C and D, some customers will call about A and B, and some will call about E. (Unless D is "anything else", E always exists.) So the first labrynth problem is dead ends: you have to cater for the user who doesn't fit your categories neatly.

The usual fallback is to add for all other enquiries, press 5. In the context of IVR systems, this is good practice, provided that 5 directs the caller to a human. The bad dream turns into a nightmare when pressing 5 takes the user to another menu.

IVR menus bad. Hierarchical IVR menus, doubleplusbad.

Footnotes

1 http://www.twocrows.com/ ↩

2 with IVR system acting possibly as a backup to deal with rare cases of genuinely "exceptional call volumes". ↩

3 or worse, putting the customer into a new IVR menu to select music} The only solution is to avoid making customers dangle on phone lines. ↩

4 No, you really don't . . . ↩

5 In reality, of course, registration is not required for papers from The Scientific Marketer. ↩

6 They will have to be much better than human beings to compensate for all the ways in which they are worse. ↩

Murphy's Laws for Data

I've had the privilege of digging through some of Murphy's papers and it transpires that there is a whole collection of lesser-known variants of the Murphy's Law specifically for data.

Murphy's handwriting leaves a little to be desired, and my access was fairly limited, but from what I can gather the following laws are inviolate

Murphy's Laws for Data (ML4D)

1. If data can be wrong, it will be.
2. If data can be misinterpreted, it will be.
3. If data can be biased, it will be.
4. If data can be misformatted, it will be.
5. If data can be incomplete, it will be.
6. If errors in data can pass silently, some will.
7. If data formats are ambiguous, all interpretations will be used.
8. If data formats are unambiguous, they will be ignored.
9. If summarization can destroy meaning, it will.
10. If patterns can be non-linear, they will be.
11. If data items can contain separators, they will.
12. If data can be destroyed it will be (except when the goal is data destruction).
13. The life expectancy of any datum is inversely proportional to its utility and correctness.
14. The likelihood of data being correct is inversely propotional to the importance of the decisions it will be used to inform.
15. Some of the data will be case sensitive.
16. If input and output encodings can be different, they will be.
17. Representative samples aren't.
18. If data can be encoded in EBCDIC, it will be.
19. If escape conventions can differ, they will.
20. If the data is correct, then the checksum will be incorrect; and vice versa.
21. Encryption will render the data unreadable by the encryptor and transparent to others.
22. Dates are subject to their own special versions of Murphy's Laws for Data.
23. Passwords are also subject to their own special versions of Murphy's Laws for Data.
24. Data that demands to be graphed won't be.
25. Excel will obscure all meaning in data with a combination of chart-junk and inappropriate defaults.
26. Causal relationships change immediately after detection.
27. The likelihood that a confidence test on data has been applied correctly is less than the stated confidence level.
28. Backups become corrupted/missing at exactly the same time as their corresponding master.
29. The obvious interpretation is incorrect.
30. The correct interpretation is implausible.

Summary

Data will at best be incorrect, misinterpreted, misformatted, biased, incomplete, non-linear, misgraphed and quickly lost.

Footnote (Data and Plurals)

I am aware that there is a school of thought that maintains that the word data is plural and that on this basis we should say things like "the data are wrong". Neither Murphy nor I attended that school, but it is our opinion that that data supporting this view is questionable and that such usage, in this twenty-first century, is at best archaic and possibly even affected. Those of a different and more delicate sensibility are respectively requested to pass over these laws quickly to avoid undue distress.