tag:blogger.com,1999:blog-38857904.post1821790642840895753..comments2009-03-04T07:10:26.533Znjrhttp://www.blogger.com/profile/08980758986023344486noreply@blogger.comBlogger2125tag:blogger.com,1999:blog-38857904.post-8990347336334915072009-03-04T07:10:00.000Z2009-03-04T07:10:00.000ZHi Ed<BR/><BR/>I'd agree with at least your 1 and 2. Indeed, one of the articles in this series will be about the so-called curse of dimensionality, which speaks to your first point.<BR/><BR/>As for the normal assumption/requirement/transformation: well, sure. But as with scaling, we need to be a bit careful that we don't wash out the pattern we're trying to find. I guess that's covered by your "carefully chosen" comment: if you're comfortable that any transformations you're applying are reasonable (i.e. make the variables more meaningful, or at least, not less meaningful) then all's well and good. Where I start to feel uncomfortable is when we just map the distribution to what we want to make the technique work and then claim that the result reflects an innate structure in the data.<BR/><BR/>Thanks for dropping by...njrhttp://www.blogger.com/profile/08980758986023344486noreply@blogger.comtag:blogger.com,1999:blog-38857904.post-23135115682012366512009-03-04T01:44:00.000Z2009-03-04T01:44:00.000ZClustering is supposed to be the Great Data Mining Garbage Disposal. I've found that it works a whole lot better if I've got 1) a few variables, carefully chosen that 2) are qualitatively similar and 3) are normally distributed, roughly. <BR/><BR/>For instance, if I'm starting with the assets people have in different investment types (bank account, CD, home, stocks, etc.) which are probably going to be roughly normal after a log transform, then all my dimensions of clustering are fairly similar.<BR/><BR/>==EdEdmund Freemanhttp://www.blogger.com/profile/13646131637425938810noreply@blogger.com